You are viewing a plain text version of this content. The canonical link for it is here.
Posted to pr@cassandra.apache.org by "iamaleksey (via GitHub)" <gi...@apache.org> on 2026/06/29 15:43:09 UTC
[PR] CEP-45: Topology Change Support for Mutation Tracking [cassandra]
iamaleksey opened a new pull request, #4910:
URL: https://github.com/apache/cassandra/pull/4910
patch by Aleksey Yeschenko; reviewed by Blake Eggleston for CASSANDRA-20386
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CEP-45: Topology Change Support for Mutation Tracking [cassandra]
Posted by "iamaleksey (via GitHub)" <gi...@apache.org>.
iamaleksey commented on code in PR #4910:
URL: https://github.com/apache/cassandra/pull/4910#discussion_r3796907292
##########
src/java/org/apache/cassandra/tcm/sequences/UnbootstrapAndLeave.java:
##########
@@ -211,16 +213,23 @@ public SequenceState executeNext()
}
break;
case FINISH_LEAVE:
+ ClusterMetadata postFinish;
try
{
- ClusterMetadataService.instance().commit(finishLeave);
+ postFinish = ClusterMetadataService.instance().commit(finishLeave);
StorageService.instance.clearTransientMode();
}
catch (Throwable t)
{
JVMStabilityInspector.inspectThrowable(t);
return continuable();
}
+
+ // Seal the shards obsoleted by FINISH_LEAVE: the departed node's over-replicated generations created
+ // during START_LEAVE, plus the merge-half folded into the departed node's range by the merge.
+ if (MutationTrackingService.isEnabled())
+ if (streams.kind() == LeaveStreams.Kind.UNBOOTSTRAP || streams.kind() == LeaveStreams.Kind.REMOVENODE)
+ SealingCoordinator.sealShardsAtFinishLeave(postFinish, finishLeave.delta(), finishLeave.nodeId(), streams.kind());
Review Comment:
Same response applies (so let's continue the discussion in the first thread).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CEP-45: Topology Change Support for Mutation Tracking [cassandra]
Posted by "iamaleksey (via GitHub)" <gi...@apache.org>.
iamaleksey commented on code in PR #4910:
URL: https://github.com/apache/cassandra/pull/4910#discussion_r3797164111
##########
src/java/org/apache/cassandra/tcm/sequences/BootstrapAndReplace.java:
##########
@@ -290,6 +292,9 @@ public SequenceState executeNext()
}
ClusterMetadataService.instance().ensureCMSPlacement(metadata);
+ if (MutationTrackingService.isEnabled())
+ SealingCoordinator.sealShardsAtFinishReplace(metadata, finishReplace.delta(), startReplace.replaced());
Review Comment:
Relatedly, `cancel()` will also need to be implemented, which wasn't on my radar at all (and is something that Claude reviewer hasn't flagged either). This might end up being as simple as naively implementing shard unsealing, but it could be challenging as well - haven't yet fully thought it through.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CEP-45: Topology Change Support for Mutation Tracking [cassandra]
Posted by "iamaleksey (via GitHub)" <gi...@apache.org>.
iamaleksey commented on code in PR #4910:
URL: https://github.com/apache/cassandra/pull/4910#discussion_r3797155838
##########
src/java/org/apache/cassandra/tcm/sequences/BootstrapAndReplace.java:
##########
@@ -290,6 +292,9 @@ public SequenceState executeNext()
}
ClusterMetadataService.instance().ensureCMSPlacement(metadata);
+ if (MutationTrackingService.isEnabled())
+ SealingCoordinator.sealShardsAtFinishReplace(metadata, finishReplace.delta(), startReplace.replaced());
Review Comment:
I'm working on adding the additional states right now (having had a long chat with Sam to confirm the correct way).
That said, I'm not sure what you mean by making them implicitly sealing. Elaborate?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CEP-45: Topology Change Support for Mutation Tracking [cassandra]
Posted by "iamaleksey (via GitHub)" <gi...@apache.org>.
iamaleksey commented on code in PR #4910:
URL: https://github.com/apache/cassandra/pull/4910#discussion_r3796907110
##########
src/java/org/apache/cassandra/tcm/sequences/BootstrapAndJoin.java:
##########
@@ -300,10 +302,15 @@ public SequenceState executeNext()
return continuable();
}
ClusterMetadataService.instance().ensureCMSPlacement(metadata);
+
+ if (MutationTrackingService.isEnabled())
+ SealingCoordinator.sealShardsAtFinishJoin(metadata, finishJoin.delta());
Review Comment:
Same response applies (so let's continue the discussion in the first thread).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CEP-45: Topology Change Support for Mutation Tracking [cassandra]
Posted by "bdeggleston (via GitHub)" <gi...@apache.org>.
bdeggleston commented on PR #4910:
URL: https://github.com/apache/cassandra/pull/4910#issuecomment-6372372413
I pushed up a branch addressing the remaining PR items and fixing some other bugs/gaps found during re-review here:
https://github.com/bdeggleston/cassandra/tree/C20386-suggest
Rebased on cep-45-mutation tracking as of ~Wednesday with some tests fixed and no regressions in CI here:
https://github.com/bdeggleston/cassandra/tree/C20386-suggest-rebased
If you’re +1, I’m +1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CEP-45: Topology Change Support for Mutation Tracking [cassandra]
Posted by "iamaleksey (via GitHub)" <gi...@apache.org>.
iamaleksey commented on code in PR #4910:
URL: https://github.com/apache/cassandra/pull/4910#discussion_r4072701463
##########
src/java/org/apache/cassandra/tcm/sequences/UnbootstrapAndLeave.java:
##########
@@ -211,16 +213,23 @@ public SequenceState executeNext()
}
break;
case FINISH_LEAVE:
+ ClusterMetadata postFinish;
try
{
- ClusterMetadataService.instance().commit(finishLeave);
+ postFinish = ClusterMetadataService.instance().commit(finishLeave);
StorageService.instance.clearTransientMode();
}
catch (Throwable t)
{
JVMStabilityInspector.inspectThrowable(t);
return continuable();
}
+
+ // Seal the shards obsoleted by FINISH_LEAVE: the departed node's over-replicated generations created
+ // during START_LEAVE, plus the merge-half folded into the departed node's range by the merge.
+ if (MutationTrackingService.isEnabled())
+ if (streams.kind() == LeaveStreams.Kind.UNBOOTSTRAP || streams.kind() == LeaveStreams.Kind.REMOVENODE)
+ SealingCoordinator.sealShardsAtFinishLeave(postFinish, finishLeave.delta(), finishLeave.nodeId(), streams.kind());
Review Comment:
Solved by adding the additional step to the sequence, too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CEP-45: Topology Change Support for Mutation Tracking [cassandra]
Posted by "iamaleksey (via GitHub)" <gi...@apache.org>.
iamaleksey commented on code in PR #4910:
URL: https://github.com/apache/cassandra/pull/4910#discussion_r4135432611
##########
src/java/org/apache/cassandra/replication/SealingCoordinator.java:
##########
@@ -0,0 +1,1401 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.cassandra.replication;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.TimeoutException;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.concurrent.locks.LockSupport;
+
+import javax.annotation.Nonnull;
+import javax.annotation.Nullable;
+
+import com.google.common.base.Preconditions;
+
+import org.apache.cassandra.config.DatabaseDescriptor;
+import org.apache.cassandra.db.ColumnFamilyStore;
+import org.apache.cassandra.db.Keyspace;
+import org.apache.cassandra.db.TypeSizes;
+import org.apache.cassandra.dht.AbstractBounds;
+import org.apache.cassandra.dht.IPartitioner;
+import org.apache.cassandra.dht.Range;
+import org.apache.cassandra.dht.Token;
+import org.apache.cassandra.exceptions.RequestFailure;
+import org.apache.cassandra.io.UnversionedSerializer;
+import org.apache.cassandra.io.util.DataInputPlus;
+import org.apache.cassandra.io.util.DataOutputPlus;
+import org.apache.cassandra.locator.EndpointsByReplica;
+import org.apache.cassandra.locator.EndpointsForRange;
+import org.apache.cassandra.locator.InetAddressAndPort;
+import org.apache.cassandra.locator.RangesAtEndpoint;
+import org.apache.cassandra.locator.Replica;
+import org.apache.cassandra.net.IVerbHandler;
+import org.apache.cassandra.net.Message;
+import org.apache.cassandra.net.MessagingService;
+import org.apache.cassandra.net.RequestCallback;
+import org.apache.cassandra.net.Verb;
+import org.apache.cassandra.schema.KeyspaceMetadata;
+import org.apache.cassandra.tcm.ClusterMetadata;
+import org.apache.cassandra.tcm.membership.NodeId;
+import org.apache.cassandra.tcm.ownership.DataPlacement;
+import org.apache.cassandra.tcm.ownership.MovementMap;
+import org.apache.cassandra.tcm.ownership.PlacementDeltas;
+import org.apache.cassandra.tcm.ownership.VersionedEndpoints;
+import org.apache.cassandra.tcm.sequences.LeaveStreams;
+import org.apache.cassandra.utils.CollectionSerializers;
+import org.apache.cassandra.utils.FBUtilities;
+import org.apache.cassandra.utils.concurrent.AsyncPromise;
+import org.apache.cassandra.utils.concurrent.Future;
+import org.apache.cassandra.utils.concurrent.FutureCombiner;
+
+import static java.lang.String.format;
+import static java.util.concurrent.TimeUnit.MILLISECONDS;
+import static java.util.concurrent.TimeUnit.NANOSECONDS;
+import static java.util.concurrent.TimeUnit.SECONDS;
+import static org.apache.cassandra.utils.Clock.Global.nanoTime;
+
+/*
+ * Sealing {@link Shard}s once a topology change obsoletes them.
+ *
+ * Happy Path Sequence:
+ * 1. Bring every participant up to (at least) this seal's epoch and fence allocation, so that everyone
+ * stops writing to the old shards; then wait until all in-progress mutations have been done.
+ * 2. Same for tracked transfers
+ * 3. Wait until all participating replicas have reconciled the union of their witnessed ids
+ * 4. Mark the shard as fully sealed
+ * 5. Stop including the shard in mutation summaries for read requests.
+ *
+ * Unhappy Path Sequence (not yet implemented):
+ * 1. Wait until everyone who's up stops writing to the old shards and all in-progress mutations have been done
+ * 2. Same for tracked transfers
+ * 3. Wait until all *live* participating replicas have reconciled the union of their witnessed ids
+ * 4. Mark the shard as partially sealed, label it with the set of participants that were able to seal
+ * 5. When a down node comes up, have it talk to other nodes and learn about sealing, and what mutations/logs
+ * they have exchanged. Diff with what we have - in system tables and in the journal (though system table
+ * plus local replay should give us an up-to-date metadata state for the shard)
+ */
+public final class SealingCoordinator
+{
+ /*
+ * BOOTSTRAP sequence illustrated. Stages in which write placements
+ * remain unchanged, and only read placements change, are skipped
+ * here as irrelevant to sharding / sealing.
+ *
+ * There is a tracked keyspace with RF=3.
+ * A node n35 with token=35 is added in the middle of the ring.
+ *
+ * 1. Initial state at e0:
+ *
+ * (10,20] | {n20,n30,n40} | e0 | ACTIVE
+ * (20,30] | {n30,n40,n50} | e0 | ACTIVE
+ * (30,40] | {n40,n50,n60} | e0 | ACTIVE
+ *
+ * 2. PREPARE_JOIN at e1:
+ *
+ * (10,20] | {n20,n30,n40} | e0 | ACTIVE
+ * (20,30] | {n30,n40,n50} | e0 | ACTIVE
+ * (30,40] | {n40,n50,n60} | e0 | ACTIVE
+ * + (30,35] | {n40,n50,n60} | e1 | ACTIVE
+ * + (35,40] | {n40,n50,n60} | e1 | ACTIVE
+ *
+ * 3. START_JOIN at e2:
+ *
+ * (10,20] | {n20,n30,n40} | e0 | ACTIVE
+ * + (10,20] | {n20,n30,n35,n40} | e2 | ACTIVE
+ * (20,30] | {n30,n40,n50} | e0 | ACTIVE
+ * + (20,30] | {n30,n35,n40,n50} | e2 | ACTIVE
+ * (30,40] | {n40,n50,n60} | e0 | ACTIVE
+ * (30,35] | {n40,n50,n60} | e1 | ACTIVE
+ * + (30,35] | {n35,n40,n50,n60} | e2 | ACTIVE
+ * (35,40] | {n40,n50,n60} | e1 | ACTIVE
+ *
+ * 4. MID_JOIN at e3 [*]:
+ *
+ * S (10,20] | {n20,n30,n40} | e0 | SEALED
+ * (10,20] | {n20,n30,n35,n40} | e2 | ACTIVE
+ * S (20,30] | {n30,n40,n50} | e0 | SEALED
+ * (20,30] | {n30,n35,n40,n50} | e2 | ACTIVE
+ * S (30,40] | {n40,n50,n60} | e0 | SEALED
+ * S (30,35] | {n40,n50,n60} | e1 | SEALED
+ * (30,35] | {n35,n40,n50,n60} | e2 | ACTIVE
+ * (35,40] | {n40,n50,n60} | e1 | ACTIVE
+ *
+ * 5. FINISH_JOIN at e4:
+ *
+ * (10,20] | {n20,n30,n40} | e0 | SEALED
+ * S (10,20] | {n20,n30,n35,n40} | e2 | SEALED
+ * + (10,20] | {n20,n30,n35} | e4 | ACTIVE
+ * (20,30] | {n30,n40,n50} | e0 | SEALED
+ * S (20,30] | {n30,n35,n40,n50} | e2 | SEALED
+ * + (20,30] | {n30,n35,n40} | e4 | ACTIVE
+ * (30,40] | {n40,n50,n60} | e0 | SEALED
+ * (30,35] | {n40,n50,n60} | e1 | SEALED
+ * S (30,35] | {n35,n40,n50,n60} | e2 | SEALED
+ * + (30,35] | {n35,n40,n50} | e4 | ACTIVE
+ * (35,40] | {n40,n50,n60} | e1 | ACTIVE
+ *
+ * [*] At this stage, after sealing the shards, we stream the sstables.
+ * All the offsets belonging to these sealed shards get stripped from
+ * stats component of streamed sstables. Otherwise, the new node
+ * won't be able to compact them or mark them as reconciled/repaired -
+ * it is rightfully unaware of the sealed shards.
+ */
+
+ /**
+ * Seal the shards obsoleted by bootstrap before the new node starts streaming.
+ */
+ public static void sealShardsAtMidJoin(ClusterMetadata metadata, MovementMap movements)
+ {
+ seal(discoverShardsAtMidJoinOrReplace(metadata, movements));
+ }
+
+ /**
+ * Seal the intermediate shards that were created during START_JOIN.
+ * TODO (required): this MUST eventually run while the affected ranges are still locked - between the
+ * removeFromWrites placement change and the range unlock. FinishJoin/FinishReplace currently apply
+ * removeFromWrites and the unlock atomically in a single transformation, so there is no committed
+ * dropped-but-locked epoch to hook. Until then, this runs just after FINISH_JOIN/FINISH_REPLACE,
+ * when the ranges are already unlocked. Left to implement for the unhappy path (AY).
+ */
+ public static void sealShardsAtFinishJoin(ClusterMetadata metadata, PlacementDeltas finishDelta)
+ {
+ seal(discoverShardsAtFinishJoinOrReplace(metadata, finishDelta, null));
+ }
+
+ /*
+ * REPLACE sequence illustrated. Stages in which write placements
+ * remain unchanged, and only read placements change, are skipped
+ * here as irrelevant to sharding / sealing.
+ *
+ * There is a tracked keyspace with RF=3.
+ * Node n40 is dead, and is being replaced with $n40.
+ *
+ * 1. Initial state at e0:
+ *
+ * (10,20] | {n20,n30,n40} | e0 | ACTIVE
+ * (20,30] | {n30,n40,n50} | e0 | ACTIVE
+ * (30,40] | {n40,n50,n60} | e0 | ACTIVE
+ *
+ * 2. START_REPLACE at e1:
+ *
+ * (10,20] | {n20,n30,n40} | e0 | ACTIVE
+ * + (10,20] | {n20,n30,n40,$n40}| e1 | ACTIVE
+ * (20,30] | {n30,n40,n50} | e0 | ACTIVE
+ * + (20,30] | {n30,n40,$n40,n50}| e1 | ACTIVE
+ * (30,40] | {n40,n50,n60} | e0 | ACTIVE
+ * + (30,40] | {n40,$n40,n50,n60}| e1 | ACTIVE
+ *
+ * 3. MID_REPLACE at e2 [*]:
+ *
+ * S (10,20] | {n20,n30,n40} | e0 | SEALED
+ * (10,20] | {n20,n30,n40,$n40}| e1 | ACTIVE
+ * S (20,30] | {n30,n40,n50} | e0 | SEALED
+ * (20,30] | {n30,n40,$n40,n50}| e1 | ACTIVE
+ * S (30,40] | {n40,n50,n60} | e0 | SEALED
+ * (30,40] | {n40,$n40,n50,n60}| e1 | ACTIVE
+ *
+ * 4. FINISH_REPLACE at e3:
+ *
+ * (10,20] | {n20,n30,n40} | e0 | SEALED
+ * S (10,20] | {n20,n30,n40,$n40}| e1 | SEALED
+ * + (10,20] | {n20,n30,$n40} | e3 | ACTIVE
+ * (20,30] | {n30,n40,n50} | e0 | SEALED
+ * S (20,30] | {n30,n40,$n40,n50}| e1 | SEALED
+ * + (20,30] | {n30,$n40,n50} | e3 | ACTIVE
+ * (30,40] | {n40,n50,n60} | e0 | SEALED
+ * S (30,40] | {n40,$n40,n50,n60}| e1 | SEALED
+ * + (30,40] | {$n40,n50,n60} | e3 | ACTIVE
+ *
+ * [*] At this stage, after sealing the shards, we stream the sstables.
+ * All the offsets belonging to these sealed shards get stripped from
+ * stats component of streamed sstables. Otherwise, the new node
+ * won't be able to compact them or mark them as reconciled/repaired -
+ * it is rightfully unaware of the sealed shards.
+ */
+
+ /**
+ * Seal the shards obsoleted by host replacement before the replacement node starts streaming.
+ * The {@code deadNode} node cannot participate in the sealing.
+ */
+ public static void sealShardsAtMidReplace(ClusterMetadata metadata, MovementMap movements, @Nonnull NodeId deadNode)
+ {
+ seal(discoverShardsAtMidJoinOrReplace(metadata, movements), deadNode);
+ }
+
+ public static void sealShardsAtFinishReplace(ClusterMetadata metadata, PlacementDeltas finishDelta, @Nonnull NodeId deadNode)
+ {
+ seal(discoverShardsAtFinishJoinOrReplace(metadata, finishDelta, deadNode), deadNode);
+ }
+
+ /**
+ * Discover all shards that need to be sealed at MID_JOIN or MID_REPLACE.
+ */
+ private static Set<ShardMetadata> discoverShardsAtMidJoinOrReplace(ClusterMetadata metadata, MovementMap movements)
+ {
+ List<Future<Set<ShardMetadata>>> futures = new ArrayList<>();
+
+ for (KeyspaceMetadata ksm : metadata.schema.getKeyspaces())
+ {
+ if (!ksm.params.replicationType.isTracked())
+ continue;
+
+ EndpointsByReplica acquired = movements.get(ksm.params.replication);
+ if (acquired.isEmpty()) // when the keyspace is not replicated to this DC
+ continue;
+
+ DataPlacement placement = metadata.placements.get(ksm.params.replication);
+
+ for (Map.Entry<Replica, EndpointsForRange> entry : acquired.entrySet())
+ {
+ Range<Token> range = entry.getKey().range();
+ Set<InetAddressAndPort> endpoints = entry.getValue().endpoints();
+ // cutoff: the epoch this node was added to the write placement for the range;
+ // shards over the range with sinceEpoch < cutoff are obsoleted by this bootstrap or replace.
+ long beforeEpoch = placement.writes.forRange(range).lastModified().getEpoch();
+ futures.add(FetchShards.fetch(ksm.name, beforeEpoch, range, endpoints));
+ }
+ }
+
+ Set<ShardMetadata> shards = new HashSet<>();
+ try
+ {
+ for (Set<ShardMetadata> fetched : FutureCombiner.allOf(futures).get(DatabaseDescriptor.getRpcTimeout(MILLISECONDS), MILLISECONDS))
+ shards.addAll(fetched);
+ }
+ catch (InterruptedException | ExecutionException | TimeoutException e)
+ {
+ throw new RuntimeException("Failed to fetch shards to seal", e);
+ }
+ return shards;
+ }
+
+ /**
+ * Discover all shards that need to be sealed at FINISH_JOIN or FINISH_REPLACE.
+ */
+ private static Set<ShardMetadata> discoverShardsAtFinishJoinOrReplace(ClusterMetadata metadata, PlacementDeltas finishDelta, @Nullable NodeId deadNode)
+ {
+ long finishEpoch = metadata.epoch.getEpoch();
+ Set<ShardMetadata> shards = new HashSet<>();
+ for (KeyspaceMetadata ksm : metadata.schema.getKeyspaces())
+ {
+ if (!ksm.params.replicationType.isTracked())
+ continue;
+
+ DataPlacement placement = metadata.placements.get(ksm.params.replication);
+ finishDelta.get(ksm.params.replication).writes.removals.flattenValues().forEach(removed -> {
+ Set<Integer> nodeIDs = new HashSet<>();
+ for (InetAddressAndPort ep : placement.writes.forRange(removed.range()).endpoints())
+ nodeIDs.add(metadata.directory.peerId(ep).id());
+ nodeIDs.add(deadNode != null ? deadNode.id() : metadata.directory.peerId(removed.endpoint()).id());
+ Participants participants = new Participants(nodeIDs);
+
+ MutationTrackingService.instance().forEachShardInKeyspace(ksm.name, shard -> {
+ if (shard.sinceEpoch < finishEpoch
+ && shard.range.equals(removed.range())
+ && shard.participants.equals(participants)
+ && !shard.isSealed())
+ shards.add(new ShardMetadata(shard.keyspace, shard.sinceEpoch, shard.range, shard.participants));
+ });
+ });
+ }
+ return shards;
+ }
+
+ /*
+ * UNBOOTSTRAP and REMOVENODE sequences illustrated (same sequence as far as sharding is concerned).
+ * Stages in which write placements remain unchanged, and only read placements change, are skipped
+ * here as irrelevant to sharding / sealing.
+ *
+ * There is a tracked keyspace with RF=3.
+ * A node n40 with token=40 is removed in the middle of the ring.
+ *
+ * 1. Initial state at e0:
+ *
+ * (10,20] | {n20,n30,n40} | e0 | ACTIVE
+ * (20,30] | {n30,n40,n50} | e0 | ACTIVE
+ * (30,40] | {n40,n50,n60} | e0 | ACTIVE
+ * (40,50] | {n50,n60,n70} | e0 | ACTIVE
+ *
+ * 2. START_LEAVE at e1:
+ *
+ * (10,20] | {n20,n30,n40} | e0 | ACTIVE
+ * + (10,20] | {n20,n30,n40,n50} | e1 | ACTIVE
+ * (20,30] | {n30,n40,n50} | e0 | ACTIVE
+ * + (20,30] | {n30,n40,n50,n60} | e1 | ACTIVE
+ * (30,40] | {n40,n50,n60} | e0 | ACTIVE
+ * + (30,40] | {n40,n50,n60,n70} | e1 | ACTIVE
+ * (40,50] | {n50,n60,n70} | e0 | ACTIVE
+ *
+ * 3. MID_LEAVE at e2 [*]:
+ *
+ * S (10,20] | {n20,n30,n40} | e0 | SEALED
+ * (10,20] | {n20,n30,n40,n50} | e1 | ACTIVE
+ * S (20,30] | {n30,n40,n50} | e0 | SEALED
+ * (20,30] | {n30,n40,n50,n60} | e1 | ACTIVE
+ * S (30,40] | {n40,n50,n60} | e0 | SEALED
+ * (30,40] | {n40,n50,n60,n70} | e1 | ACTIVE
+ * (40,50] | {n50,n60,n70} | e0 | ACTIVE
+ *
+ * 4. FINISH_LEAVE at e3:
+ *
+ * (10,20] | {n20,n30,n40} | e0 | SEALED
+ * S (10,20] | {n20,n30,n40,n50} | e1 | SEALED
+ * + (10,20] | {n20,n30,n50} | e3 | ACTIVE
+ * (20,30] | {n30,n40,n50} | e0 | SEALED
+ * S (20,30] | {n30,n40,n50,n60} | e1 | SEALED
+ * + (20,30] | {n30,n50,n60} | e3 | ACTIVE
+ * (30,40] | {n40,n50,n60} | e0 | SEALED
+ * S (30,40] | {n40,n50,n60,n70} | e1 | SEALED
+ * S (40,50] | {n50,n60,n70} | e0 | SEALED
+ * + (30,50] | {n50,n60,n70} | e3 | ACTIVE
+ *
+ * [*] At this stage, after sealing the shards, we stream the sstables.
+ * All the offsets belonging to these sealed shards get stripped from
+ * stats component of streamed sstables. Otherwise, the new node
+ * won't be able to compact them or mark them as reconciled/repaired -
+ * it is rightfully unaware of the sealed shards.
+ */
+
+ /**
+ * Seal the shards obsoleted by START_LEAVE before streaming, for both UNBOOTSTRAP and REMOVENODE.
+ */
+ public static void sealShardsAtMidLeave(ClusterMetadata metadata, PlacementDeltas startDelta, NodeId leavingOrDeparted, LeaveStreams.Kind kind)
+ {
+ InetAddressAndPort leavingOrDepartedEndpoint = metadata.directory.endpoint(leavingOrDeparted);
+
+ List<Future<Set<ShardMetadata>>> futures = new ArrayList<>();
+ for (KeyspaceMetadata ksm : metadata.schema.getKeyspaces())
+ {
+ if (!ksm.params.replicationType.isTracked())
+ continue;
+
+ DataPlacement placement = metadata.placements.get(ksm.params.replication);
+ startDelta.get(ksm.params.replication).writes.additions.flattenValues().forEach(added -> {
+ Range<Token> range = added.range();
+ long beforeEpoch = placement.writes.forRange(range).lastModified().getEpoch();
+ Set<InetAddressAndPort> survivors = new HashSet<>(placement.writes.forRange(range).endpoints());
+ survivors.remove(leavingOrDepartedEndpoint);
+ futures.add(FetchShards.fetch(ksm.name, beforeEpoch, range, survivors));
+ });
+ }
+
+ Set<ShardMetadata> shards = new HashSet<>();
+ try
+ {
+ for (Set<ShardMetadata> fetched : FutureCombiner.allOf(futures).get(DatabaseDescriptor.getRpcTimeout(MILLISECONDS), MILLISECONDS))
+ for (ShardMetadata shard : fetched)
+ if (shard.participants.contains(leavingOrDeparted.id()))
+ shards.add(shard);
+ }
+ catch (InterruptedException | ExecutionException | TimeoutException e)
+ {
+ throw new RuntimeException("Failed to fetch shards to seal for leave", e);
+ }
+
+ seal(shards, kind == LeaveStreams.Kind.UNBOOTSTRAP ? null : leavingOrDeparted);
+ }
+
+ /**
+ * Seal the shards that need to be sealed at LEAVE_FINISH for both UNBOOTSTRAP and REMOVENODE.
+ */
+ public static void sealShardsAtFinishLeave(ClusterMetadata postFinish, PlacementDeltas finishDelta, NodeId leavingOrDeparted, LeaveStreams.Kind kind)
+ {
+ long finishEpoch = postFinish.epoch.getEpoch();
+ InetAddressAndPort leavingOrDepartedEndpoint = postFinish.directory.endpoint(leavingOrDeparted);
+
+ List<Future<Set<ShardMetadata>>> futures = new ArrayList<>();
+ for (KeyspaceMetadata ksm : postFinish.schema.getKeyspaces())
+ {
+ if (!ksm.params.replicationType.isTracked())
+ continue;
+
+ DataPlacement placement = postFinish.placements.get(ksm.params.replication);
+ RangesAtEndpoint removals = finishDelta.get(ksm.params.replication).writes.removals.get(leavingOrDepartedEndpoint);
+ for (Replica removedReplica : removals)
+ {
+ // FINISH_LEAVE merges the leavingOrDeparted node's (pre-merge) range with an adjacent one. Query the *merged*
+ // range (not the pre-merge range) so discovery covers both the leavingOrDeparted node's own obsoleted
+ // generations and the merge-half - the adjacent pre-merge range folded into the merged range, whose
+ // shard never listed the leavingOrDeparted node. matchRange() yields the containing merged group + its
+ // current (survivor) write endpoints.
+ VersionedEndpoints.ForRange group = placement.writes.matchRange(removedReplica.range());
+ Range<Token> mergedRange = group.range();
+ Set<InetAddressAndPort> survivors = new HashSet<>(group.endpoints());
+ survivors.remove(leavingOrDepartedEndpoint);
+ futures.add(FetchShards.fetch(ksm.name, finishEpoch, mergedRange, survivors));
+ }
+ }
+
+ Set<ShardMetadata> shards = new HashSet<>();
+ try
+ {
+ for (Set<ShardMetadata> fetched : FutureCombiner.allOf(futures).get(DatabaseDescriptor.getRpcTimeout(MILLISECONDS), MILLISECONDS))
+ shards.addAll(fetched);
+ }
+ catch (InterruptedException | ExecutionException | TimeoutException e)
+ {
+ throw new RuntimeException("Failed to fetch shards to seal for leave finish", e);
+ }
+
+ seal(shards, kind == LeaveStreams.Kind.UNBOOTSTRAP ? null : leavingOrDeparted);
+ }
+
+ /*
+ * Sealing logic.
+ */
+
+ private static void seal(Set<ShardMetadata> shards)
+ {
+ seal(shards, null);
+ }
+
+ private static void seal(Set<ShardMetadata> shards, @Nullable NodeId withoutNode)
+ {
+ initiate(shards, withoutNode); // ACTIVE -> SEALING for each Shard
+ drain(shards, withoutNode); // drain in-flight local writes
+ reconcile(shards, withoutNode); // wait for logs to reconcile
+ complete(shards, withoutNode); // SEALING-> SEALED for each Shard + journal flush
+ }
+
+ private static void initiate(Set<ShardMetadata> shards, @Nullable NodeId withoutNode)
+ {
+ List<Future<Void>> futures = new ArrayList<>(shards.size());
+ for (ShardMetadata shard : shards)
+ futures.add(initiate(shard, withoutNode));
+ try
+ {
+ FutureCombiner.allOf(futures).get(DatabaseDescriptor.getRpcTimeout(MILLISECONDS), MILLISECONDS);
+ }
+ catch (InterruptedException | ExecutionException | TimeoutException e)
+ {
+ throw new RuntimeException("Failed to initiate sealing", e);
+ }
+ }
+
+ /**
+ * Fence allocation on every participant of the obsoleted shard (ACTIVE -> SEALING) by marking the shard
+ * SEALING on each.
+ * TODO (expected): this assumes every participant has already enacted the topology change that obsoleted the shard (so
+ * it no longer allocates new ids on it). The ProgressBarrier gating bootstrap()/MID_JOIN only waits for
+ * EACH_QUORUM of the affected replicas, NOT all live replicas, so a live-but-lagging participant could still
+ * be on the old shard. Bringing each participant up to the seal epoch must be done per-replica (not
+ * per-shard) and will be added separately. TLDR - add a step to guaranteed up-to-speed rest of replicas epochs.
+ */
+ private static AsyncPromise<Void> initiate(ShardMetadata shard, @Nullable NodeId withoutNode)
+ {
+ return InitSealing.initiate(shard.keyspace, shard.sinceEpoch, shard.range, toEndpoints(shard.participants, withoutNode));
+ }
+
+ private static void drain(Set<ShardMetadata> shards, @Nullable NodeId withoutNode)
+ {
+ long deadlineNanos = nanoTime() + DatabaseDescriptor.getRpcTimeout(NANOSECONDS);
+ Set<ShardMetadata> pending = new HashSet<>(shards);
+ while (true)
+ {
+ // non-blocking poll: each participant reports its drain status immediately; retry until all drained
+ pending.removeIf(shard -> drain(shard, withoutNode));
+ if (pending.isEmpty())
+ return;
+ if (nanoTime() >= deadlineNanos)
+ throw new RuntimeException("Timed out draining shards; still pending: " + pending);
+ LockSupport.parkNanos(SECONDS.toNanos(1));
+ }
+ }
+
+ /**
+ * Poll every participant of the SEALING shard once for drain status; true iff all report that no mutation
+ * id allocated before the SEALING fence is still applying locally.
+ */
+ private static boolean drain(ShardMetadata shard, @Nullable NodeId withoutNode)
+ {
+ try
+ {
+ return Drain.poll(shard.keyspace, shard.sinceEpoch, shard.range, toEndpoints(shard.participants, withoutNode))
+ .get(DatabaseDescriptor.getRpcTimeout(MILLISECONDS), MILLISECONDS);
+ }
+ catch (InterruptedException | ExecutionException | TimeoutException e)
+ {
+ throw new RuntimeException(format("Failed to poll drain for shard %s", shard), e);
+ }
+ }
+
+ /**
+ * For each of the shards, capture the witnessed union then poll each participant until it has
+ * caught up and witnessed the entire offset union itself.
+ */
+ private static void reconcile(Set<ShardMetadata> shards, @Nullable NodeId withoutNode)
+ {
+ for (ShardMetadata shard : shards)
+ reconcile(shard, withoutNode);
+ }
+
+ private static void reconcile(ShardMetadata shard, @Nullable NodeId withoutNode)
+ {
+ // capture shard's witnessed offsets once
+ Log2OffsetsMap.Mutable offsets = captureWitnessedOffsets(shard, withoutNode);
+ // poll every participant until they've each witnessed the union of offsets
+ for (InetAddressAndPort endpoint : toEndpoints(shard.participants, withoutNode))
+ pollUntilWitnesses(shard, offsets, endpoint);
+ }
+
+ private static Log2OffsetsMap.Mutable captureWitnessedOffsets(ShardMetadata shard, @Nullable NodeId withoutNode)
+ {
+ List<InetAddressAndPort> endpoints = toEndpoints(shard.participants, withoutNode);
+ try
+ {
+ return ReconcileCapture.capture(shard.keyspace, shard.sinceEpoch, shard.range, endpoints)
+ .get(DatabaseDescriptor.getRpcTimeout(MILLISECONDS), MILLISECONDS);
+ }
+ catch (InterruptedException | ExecutionException | TimeoutException e)
+ {
+ throw new RuntimeException(format("Failed to capture witnessed offsets for shard %s", shard), e);
+ }
+ }
+
+ private static void pollUntilWitnesses(ShardMetadata shard, Log2OffsetsMap.Mutable offsets, InetAddressAndPort endpoint)
+ {
+ // TODO (expected): use a longer timeout
+ long deadlineNanos = nanoTime() + DatabaseDescriptor.getRpcTimeout(NANOSECONDS);
+
+ while (true)
+ {
+ if (nanoTime() >= deadlineNanos)
+ throw new RuntimeException("Timed out reconciling shard: " + shard);
+ try
+ {
+ boolean witnessed =
+ ReconcilePoll.poll(shard.keyspace, shard.sinceEpoch, shard.range, offsets, endpoint)
+ .get(DatabaseDescriptor.getRpcTimeout(MILLISECONDS), MILLISECONDS);
+ if (witnessed)
+ return;
+ }
+ catch (InterruptedException | ExecutionException | TimeoutException e)
+ {
+ throw new RuntimeException(format("Failed to poll reconcile for shard %s", shard), e);
+ }
+
+ LockSupport.parkNanos(SECONDS.toNanos(1));
+ }
+ }
+
+ private static void complete(Set<ShardMetadata> shards, @Nullable NodeId withoutNode)
+ {
+ List<Future<Void>> futures = new ArrayList<>(shards.size());
+ for (ShardMetadata shard : shards)
+ futures.add(complete(shard, withoutNode));
+
+ try
+ {
+ FutureCombiner.allOf(futures).get(DatabaseDescriptor.getRpcTimeout(MILLISECONDS), MILLISECONDS);
+ }
+ catch (InterruptedException | ExecutionException | TimeoutException e)
+ {
+ throw new RuntimeException("Failed to complete sealing", e);
+ }
+ }
+
+ /**
+ * Promote every participant of the (drained and reconciled) shard from SEALING to SEALED.
+ */
+ private static AsyncPromise<Void> complete(ShardMetadata shard, @Nullable NodeId withoutNode)
+ {
+ return CompleteSealing.complete(shard.keyspace, shard.sinceEpoch, shard.range, toEndpoints(shard.participants, withoutNode));
+ }
+
+ /**
+ * A bootstrapping node knows nothing about the shards that currently exist,
+ * so it must collect the list of shards from the existing replicas before
+ * it can proceed with the rest of the sealing steps.
+ */
+ public static final class FetchShards
+ {
+ public static final class Request
+ {
+ final String keyspace;
+ final long beforeEpoch;
+ final Range<Token> range;
+
+ Request(String keyspace, long beforeEpoch, Range<Token> range)
+ {
+ this.keyspace = keyspace;
+ this.beforeEpoch = beforeEpoch;
+ this.range = range;
+ }
+ }
+
+ public static final class Response
+ {
+ final Set<ShardMetadata> shards;
+
+ Response(Set<ShardMetadata> shards)
+ {
+ this.shards = shards;
+ }
+ }
+
+ /**
+ * Query all replicas in a group in parallel for the shards over {@code ranges} obsoleted by an
+ * in-flight topology change (those with {@code sinceEpoch < beforeEpoch}). Resembles
+ * {@link ShardMetadataRequest#queryPeers}, but waits for every (live) replica to respond and
+ * merges their deduplicated shards rather than taking the first response.
+ */
+ public static AsyncPromise<Set<ShardMetadata>> fetch(
+ String keyspace, long beforeEpoch, Range<Token> range, Set<InetAddressAndPort> endpoints)
+ {
+ Preconditions.checkArgument(!endpoints.isEmpty());
+ AsyncPromise<Set<ShardMetadata>> promise = new AsyncPromise<>();
+ Set<ShardMetadata> merged = ConcurrentHashMap.newKeySet();
+
+ RequestCallback<Response> callback = new RequestCallback<>()
+ {
+ private final AtomicInteger remaining = new AtomicInteger(endpoints.size());
+
+ @Override
+ public void onResponse(Message<Response> msg)
+ {
+ merged.addAll(msg.payload.shards);
+ if (remaining.decrementAndGet() == 0)
+ promise.trySuccess(merged);
+ }
+
+ @Override
+ public void onFailure(InetAddressAndPort from, RequestFailure failure)
+ {
+ // happy path assumes all replicas are up; fail the whole fetch if any can't respond
+ promise.tryFailure(new RuntimeException(format("Failed to fetch shards to seal from %s for keyspace %s: %s", from, keyspace, failure)));
+ }
+
+ @Override
+ public boolean invokeOnFailure()
+ {
+ return true;
+ }
+ };
+
+ Message<Request> message = Message.out(Verb.MT_FETCH_SHARDS_REQ, new Request(keyspace, beforeEpoch, range));
+ for (InetAddressAndPort peer : endpoints)
+ MessagingService.instance().sendWithCallback(message, peer, callback);
+
+ return promise;
+ }
+
+ public static final IVerbHandler<Request> verbHandler = message ->
+ {
+ MutationTrackingService.ensureEnabled();
+ Request request = message.payload;
+ Set<ShardMetadata> shards = new HashSet<>();
+ MutationTrackingService.instance().forEachShardInKeyspace(request.keyspace, shard -> {
+ if (shard.sinceEpoch < request.beforeEpoch
+ && (request.range.contains(shard.range) || shard.range.contains(request.range))
+ && !shard.isSealed())
+ shards.add(new ShardMetadata(shard.keyspace, shard.sinceEpoch, shard.range, shard.participants));
+ });
+ MessagingService.instance().send(message.responseWith(new Response(shards)), message.from());
+ };
+
+ public static final VersionedSerializer<Request> requestSerializer = new VersionedSerializer<>()
+ {
+ @Override
+ public void serialize(Request r, DataOutputPlus out, Version version) throws IOException
+ {
+ out.writeUTF(r.keyspace);
+ out.writeLong(r.beforeEpoch);
+ AbstractBounds.tokenSerializer.serialize(r.range, out, version.messagingVersion());
+ }
+
+ @Override
+ public Request deserialize(DataInputPlus in, Version version) throws IOException
+ {
+ String keyspace = in.readUTF();
+ long beforeEpoch = in.readLong();
+ Range<Token> range = (Range<Token>) AbstractBounds.tokenSerializer.deserialize(in, IPartitioner.global(), version.messagingVersion());
+ return new Request(keyspace, beforeEpoch, range);
+ }
+
+ @Override
+ public long serializedSize(Request r, Version version)
+ {
+ long size = TypeSizes.sizeof(r.keyspace);
+ size += TypeSizes.sizeof(r.beforeEpoch);
+ size += AbstractBounds.tokenSerializer.serializedSize(r.range, version.messagingVersion());
+ return size;
+ }
+ };
+
+ public static final VersionedSerializer<Response> responseSerializer = new VersionedSerializer<>()
+ {
+ @Override
+ public void serialize(Response r, DataOutputPlus out, Version version) throws IOException
+ {
+ CollectionSerializers.serializeCollection(r.shards, out, version, ShardMetadata.serializer);
+ }
+
+ @Override
+ public Response deserialize(DataInputPlus in, Version version) throws IOException
+ {
+ return new Response(CollectionSerializers.deserializeSet(in, version, ShardMetadata.serializer));
+ }
+
+ @Override
+ public long serializedSize(Response r, Version version)
+ {
+ return CollectionSerializers.serializedCollectionSize(r.shards, version, ShardMetadata.serializer);
+ }
+ };
+ }
+
+ public static final class InitSealing
+ {
+ public static final class Request
+ {
+ final String keyspace;
+ final long sinceEpoch;
+ final Range<Token> range;
+
+ Request(String keyspace, long sinceEpoch, Range<Token> range)
+ {
+ this.keyspace = keyspace;
+ this.sinceEpoch = sinceEpoch;
+ this.range = range;
+ }
+ }
+
+ // cannot use NoPayload, because its serializer cannot be wrapped inside mtEmbedded()
+ public static final class Response
+ {
+ private static final Response instance = new Response();
+ }
+
+ /**
+ * Tell every participant of an obsoleted shard to fence id allocation (mark it SEALING)
+ */
+ public static AsyncPromise<Void> initiate(
+ String keyspace, long sinceEpoch, Range<Token> range, List<InetAddressAndPort> endpoints)
+ {
+ AsyncPromise<Void> promise = new AsyncPromise<>();
+
+ RequestCallback<Response> callback = new RequestCallback<>()
+ {
+ private final AtomicInteger remaining = new AtomicInteger(endpoints.size());
+
+ @Override
+ public void onResponse(Message<Response> msg)
+ {
+ if (remaining.decrementAndGet() == 0)
+ promise.trySuccess(null);
+ }
+
+ @Override
+ public void onFailure(InetAddressAndPort from, RequestFailure failure)
+ {
+ promise.tryFailure(new RuntimeException(format("Failed to initiate sealing on %s for keyspace %s: %s", from, keyspace, failure)));
+ }
+
+ @Override
+ public boolean invokeOnFailure()
+ {
+ return true;
+ }
+ };
+
+ Message<Request> message = Message.out(Verb.MT_INIT_SEALING_REQ, new Request(keyspace, sinceEpoch, range));
+ for (InetAddressAndPort peer : endpoints)
+ MessagingService.instance().sendWithCallback(message, peer, callback);
+
+ return promise;
+ }
+
+ public static final IVerbHandler<Request> verbHandler = message -> {
+ MutationTrackingService.ensureEnabled();
+ Request request = message.payload;
+ MutationTrackingService.instance().markShardSealing(request.keyspace, request.sinceEpoch, request.range);
+ MessagingService.instance().send(message.responseWith(Response.instance), message.from());
+ };
+
+ public static final VersionedSerializer<Request> requestSerializer = new VersionedSerializer<>()
+ {
+ @Override
+ public void serialize(Request r, DataOutputPlus out, Version version) throws IOException
+ {
+ out.writeUTF(r.keyspace);
+ out.writeLong(r.sinceEpoch);
+ AbstractBounds.tokenSerializer.serialize(r.range, out, version.messagingVersion());
+ }
+
+ @Override
+ public Request deserialize(DataInputPlus in, Version version) throws IOException
+ {
+ String keyspace = in.readUTF();
+ long sinceEpoch = in.readLong();
+ Range<Token> range = (Range<Token>) AbstractBounds.tokenSerializer.deserialize(in, IPartitioner.global(), version.messagingVersion());
+ return new Request(keyspace, sinceEpoch, range);
+ }
+
+ @Override
+ public long serializedSize(Request r, Version version)
+ {
+ long size = TypeSizes.sizeof(r.keyspace);
+ size += TypeSizes.sizeof(r.sinceEpoch);
+ size += AbstractBounds.tokenSerializer.serializedSize(r.range, version.messagingVersion());
+ return size;
+ }
+ };
+
+ public static final UnversionedSerializer<Response> responseSerializer = new UnversionedSerializer<>()
+ {
+ @Override
+ public void serialize(Response r, DataOutputPlus out)
+ {
+ }
+
+ @Override
+ public Response deserialize(DataInputPlus in)
+ {
+ return Response.instance;
+ }
+
+ @Override
+ public long serializedSize(Response r)
+ {
+ return 0;
+ }
+ };
+ }
+
+ public static final class Drain
+ {
+ public static final class Request
+ {
+ final String keyspace;
+ final long sinceEpoch;
+ final Range<Token> range;
+
+ Request(String keyspace, long sinceEpoch, Range<Token> range)
+ {
+ this.keyspace = keyspace;
+ this.sinceEpoch = sinceEpoch;
+ this.range = range;
+ }
+ }
+
+ public static final class Response
+ {
+ final boolean drained;
+
+ Response(boolean drained)
+ {
+ this.drained = drained;
+ }
+ }
+
+ /**
+ * Poll every participant of a SEALING shard for whether it has drained its in-flight local
+ * writes. Succeeds with {@code true} iff every participant reports drained.
+ */
+ public static AsyncPromise<Boolean> poll(
+ String keyspace, long sinceEpoch, Range<Token> range, List<InetAddressAndPort> endpoints)
+ {
+ AsyncPromise<Boolean> promise = new AsyncPromise<>();
+
+ RequestCallback<Response> callback = new RequestCallback<>()
+ {
+ private final AtomicInteger remaining = new AtomicInteger(endpoints.size());
+ private volatile boolean allDrained = true;
+
+ @Override
+ public void onResponse(Message<Response> msg)
+ {
+ if (!msg.payload.drained)
+ allDrained = false;
+ if (remaining.decrementAndGet() == 0)
+ promise.trySuccess(allDrained);
+ }
+
+ @Override
+ public void onFailure(InetAddressAndPort from, RequestFailure failure)
+ {
+ promise.tryFailure(new RuntimeException(format("Failed to poll drain on %s for keyspace %s: %s", from, keyspace, failure)));
+ }
+
+ @Override
+ public boolean invokeOnFailure()
+ {
+ return true;
+ }
+ };
+
+ Message<Request> message = Message.out(Verb.MT_DRAIN_REQ, new Request(keyspace, sinceEpoch, range));
+ for (InetAddressAndPort peer : endpoints)
+ MessagingService.instance().sendWithCallback(message, peer, callback);
+
+ return promise;
+ }
+
+ public static final IVerbHandler<Request> verbHandler = message -> {
+ MutationTrackingService.ensureEnabled();
+ Request request = message.payload;
+ boolean drained = MutationTrackingService.instance().isShardDrained(request.keyspace, request.sinceEpoch, request.range);
+ MessagingService.instance().send(message.responseWith(new Response(drained)), message.from());
+ };
+
+ public static final VersionedSerializer<Request> requestSerializer = new VersionedSerializer<>()
+ {
+ @Override
+ public void serialize(Request r, DataOutputPlus out, Version version) throws IOException
+ {
+ out.writeUTF(r.keyspace);
+ out.writeLong(r.sinceEpoch);
+ AbstractBounds.tokenSerializer.serialize(r.range, out, version.messagingVersion());
+ }
+
+ @Override
+ public Request deserialize(DataInputPlus in, Version version) throws IOException
+ {
+ String keyspace = in.readUTF();
+ long sinceEpoch = in.readLong();
+ Range<Token> range = (Range<Token>) AbstractBounds.tokenSerializer.deserialize(in, IPartitioner.global(), version.messagingVersion());
+ return new Request(keyspace, sinceEpoch, range);
+ }
+
+ @Override
+ public long serializedSize(Request r, Version version)
+ {
+ long size = TypeSizes.sizeof(r.keyspace);
+ size += TypeSizes.sizeof(r.sinceEpoch);
+ size += AbstractBounds.tokenSerializer.serializedSize(r.range, version.messagingVersion());
+ return size;
+ }
+ };
+
+ public static final UnversionedSerializer<Response> responseSerializer = new UnversionedSerializer<>()
+ {
+ @Override
+ public void serialize(Response r, DataOutputPlus out) throws IOException
+ {
+ out.writeBoolean(r.drained);
+ }
+
+ @Override
+ public Response deserialize(DataInputPlus in) throws IOException
+ {
+ return new Response(in.readBoolean());
+ }
+
+ @Override
+ public long serializedSize(Response r)
+ {
+ return TypeSizes.sizeof(r.drained);
+ }
+ };
+ }
+
+ public static final class ReconcileCapture
+ {
+ public static final class Request
+ {
+ final String keyspace;
+ final long sinceEpoch;
+ final Range<Token> range;
+
+ Request(String keyspace, long sinceEpoch, Range<Token> range)
+ {
+ this.keyspace = keyspace;
+ this.sinceEpoch = sinceEpoch;
+ this.range = range;
+ }
+ }
+
+ public static final class Response
+ {
+ final Log2OffsetsMap<?> witnessed;
+
+ Response(Log2OffsetsMap<?> witnessed)
+ {
+ this.witnessed = witnessed;
+ }
+ }
+
+ public static AsyncPromise<Log2OffsetsMap.Mutable> capture(
+ String keyspace, long sinceEpoch, Range<Token> range, List<InetAddressAndPort> endpoints)
+ {
+ AsyncPromise<Log2OffsetsMap.Mutable> promise = new AsyncPromise<>();
+ Log2OffsetsMap.Mutable union = new Log2OffsetsMap.Mutable();
+
+ RequestCallback<Response> callback = new RequestCallback<>()
+ {
+ private final AtomicInteger remaining = new AtomicInteger(endpoints.size());
+
+ @Override
+ public void onResponse(Message<Response> msg)
+ {
+ synchronized (union)
+ {
+ union.addAll(msg.payload.witnessed);
+ }
+ if (remaining.decrementAndGet() == 0)
+ promise.trySuccess(union);
+ }
+
+ @Override
+ public void onFailure(InetAddressAndPort from, RequestFailure failure)
+ {
+ promise.tryFailure(new RuntimeException(format("Failed to capture witnessed offsets on %s for keyspace %s: %s", from, keyspace, failure)));
+ }
+
+ @Override
+ public boolean invokeOnFailure()
+ {
+ return true;
+ }
+ };
+
+ Message<Request> message = Message.out(Verb.MT_RECONCILE_CAPTURE_REQ, new Request(keyspace, sinceEpoch, range));
+ for (InetAddressAndPort peer : endpoints)
+ MessagingService.instance().sendWithCallback(message, peer, callback);
+
+ return promise;
+ }
+
+ public static final IVerbHandler<Request> verbHandler = message -> {
+ MutationTrackingService.ensureEnabled();
+ Request request = message.payload;
+ Log2OffsetsMap<?> witnessed =
+ MutationTrackingService.instance()
+ .collectLocallyWitnessedOffsets(request.keyspace, request.sinceEpoch, request.range);
Review Comment:
We are intentionally capturing offsets of a specific shard identified by `{keyspace, sinceEpoch, range}` tuple, and we do it for a set of shards that we define precisely for every transformation invoking `seal()`. I have audited the callers of `seal()` individually and the all seem to be correct (and Opus is happy).
Leaving it to you to resolve this conversation in case you have further questions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CEP-45: Topology Change Support for Mutation Tracking [cassandra]
Posted by "beobal (via GitHub)" <gi...@apache.org>.
beobal commented on code in PR #4910:
URL: https://github.com/apache/cassandra/pull/4910#discussion_r3951185335
##########
src/java/org/apache/cassandra/tcm/transformations/UnlockSequence.java:
##########
@@ -0,0 +1,126 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.cassandra.tcm.transformations;
+
+import java.io.IOException;
+
+import org.apache.cassandra.config.CassandraRelevantProperties;
+import org.apache.cassandra.exceptions.ExceptionCode;
+import org.apache.cassandra.io.util.DataInputPlus;
+import org.apache.cassandra.io.util.DataOutputPlus;
+import org.apache.cassandra.tcm.ClusterMetadata;
+import org.apache.cassandra.tcm.Transformation;
+import org.apache.cassandra.tcm.membership.NodeId;
+import org.apache.cassandra.tcm.sequences.LockedRanges;
+import org.apache.cassandra.tcm.serialization.AsymmetricMetadataSerializer;
+import org.apache.cassandra.tcm.serialization.Version;
+
+public class UnlockSequence implements Transformation
Review Comment:
I have a slight unease over the naming of this transformation. On one hand, it is concise and mostly communicates the intent, but
* it requires a `NodeId` specifically, not a `SequenceKey` and can't be easily refactored to change that. So really it is only applicable to those MSO impls collected by `SingleNodeSequences`.
* As well as unlocking, it's also responsible for cleaning up the MSO from `ClusterMetadata::inProgressSequences`
I don't have any great alternatives, only clunky ones, so I wouldn't object if you really want to keep this name, but do you have any suggestions?
##########
src/java/org/apache/cassandra/tcm/sequences/BootstrapAndJoin.java:
##########
@@ -300,10 +312,31 @@ public SequenceState executeNext()
return continuable();
}
ClusterMetadataService.instance().ensureCMSPlacement(metadata);
+
+ break;
+ case UNLOCK_SEQUENCE:
+ try
+ {
+ if (MutationTrackingService.isEnabled())
+ {
+ SealingCoordinator.sealShardsAtFinishJoin(ClusterMetadata.current(),
Review Comment:
Should a sealing failure occur during `SealingCoordinator::sealShardsAtMidJoin` the MSO be left in an state which will require operator intervention, similar to (for e.g.) a streaming failure during bootstrap
An exception thrown from `sealShardsAtFinishJoin` would return a `continuable` state without committing the `UnlockSequence` leading to it being automatically retried in `InProgessSequences::finishInProgressSequences`.
A permanent failure at this point would be somewhat difficult for operators to identify as the node would be be fully joined (`UN` in `nodetool status`), but the ranges would remain locked and the shards unsealed.
The MSO state would be visible in the `system_views.cluster_metadata_directory` table and an operator would be able to cancel it, using the `cancelInProgressSequences` method on the `org.apache.cassandra.tcm:type=CMSOperations` mbean.
This does mean the cluster state remains correct under failure but the pre-existing ergonomics are not great. Some of the steps in `SealingCoordinator::seal` do look like they could be a little fragile, so I'm just trying to consider potential operational impact.
##########
test/unit/org/apache/cassandra/tcm/transformations/EventsMetadataTest.java:
##########
@@ -136,6 +136,7 @@ public void joinTest() throws ExecutionException, InterruptedException
ClusterMetadataService.instance().commit(plan.midJoin);
ClusterMetadataService.instance().commit(plan.finishJoin);
+ ClusterMetadataService.instance().commit(new org.apache.cassandra.tcm.transformations.UnlockSequence(nodeId, plan.lockKey));
Review Comment:
nit: unnecessary qualification
##########
src/java/org/apache/cassandra/tcm/sequences/BootstrapAndReplace.java:
##########
@@ -290,6 +304,26 @@ public SequenceState executeNext()
}
ClusterMetadataService.instance().ensureCMSPlacement(metadata);
Review Comment:
Same as with `BootstrapAndJoin`, this needs to be guarded by finishReplace.unlocks() and an equivalent call added to the UNLOCK_SEQUENCE branch so that when the MSO is extended, any required CMS reconfiguration is done after the final step in the MSO.
##########
src/java/org/apache/cassandra/tcm/serialization/Version.java:
##########
@@ -90,7 +90,9 @@ public enum Version
* The version that Accord was added to TCM.
*/
public static final Version MIN_ACCORD_VERSION = V7;
+
public static final Version MIN_MUTATION_TRACKING_VERSION = V8;
+ public static final Version TBD_UNLOCK_STEP = V9;
Review Comment:
There are ways to mitigate/work around this, involving taking metadata snapshots prior to upgrade.
##########
src/java/org/apache/cassandra/tcm/sequences/BootstrapAndJoin.java:
##########
@@ -300,10 +312,31 @@ public SequenceState executeNext()
return continuable();
}
ClusterMetadataService.instance().ensureCMSPlacement(metadata);
Review Comment:
This needs to be guarded by `finishJoin.unlocks()` and an equivalent call added to the `UNLOCK_SEQUENCE` branch so that when the MSO is extended, any required CMS reconfiguration is done after the final step in the MSO.
##########
src/java/org/apache/cassandra/tcm/serialization/Version.java:
##########
@@ -90,7 +90,9 @@ public enum Version
* The version that Accord was added to TCM.
*/
public static final Version MIN_ACCORD_VERSION = V7;
+
public static final Version MIN_MUTATION_TRACKING_VERSION = V8;
+ public static final Version TBD_UNLOCK_STEP = V9;
Review Comment:
I know this is a temporary placeholder, but I think there are a couple of things worth stating explicitly.
The unlock step is being added without bumping the current version. When the parent CEP-45 branch is integrated to trunk, this will be taken care of by virtue of all the CEP-45 functionality landing in a single Version. But, if anyone has long-lived deployed clusters out there running on `cep-45-mutation-tracking` or some build derived from it, then `FINISH_(JOIN|LEAVE|REPLACE|MOVE)` transforms previously serialized to the metadata log will become un-replayable.
##########
src/java/org/apache/cassandra/replication/SealingCoordinator.java:
##########
@@ -0,0 +1,1445 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.cassandra.replication;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.TimeoutException;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.concurrent.locks.LockSupport;
+
+import javax.annotation.Nonnull;
+import javax.annotation.Nullable;
+
+import com.google.common.base.Preconditions;
+
+import org.apache.cassandra.config.DatabaseDescriptor;
+import org.apache.cassandra.db.ColumnFamilyStore;
+import org.apache.cassandra.db.Keyspace;
+import org.apache.cassandra.db.TypeSizes;
+import org.apache.cassandra.dht.AbstractBounds;
+import org.apache.cassandra.dht.IPartitioner;
+import org.apache.cassandra.dht.Range;
+import org.apache.cassandra.dht.Token;
+import org.apache.cassandra.exceptions.RequestFailure;
+import org.apache.cassandra.io.UnversionedSerializer;
+import org.apache.cassandra.io.util.DataInputPlus;
+import org.apache.cassandra.io.util.DataOutputPlus;
+import org.apache.cassandra.locator.EndpointsByReplica;
+import org.apache.cassandra.locator.EndpointsForRange;
+import org.apache.cassandra.locator.InetAddressAndPort;
+import org.apache.cassandra.locator.RangesAtEndpoint;
+import org.apache.cassandra.locator.Replica;
+import org.apache.cassandra.net.IVerbHandler;
+import org.apache.cassandra.net.Message;
+import org.apache.cassandra.net.MessagingService;
+import org.apache.cassandra.net.RequestCallback;
+import org.apache.cassandra.net.Verb;
+import org.apache.cassandra.schema.KeyspaceMetadata;
+import org.apache.cassandra.tcm.ClusterMetadata;
+import org.apache.cassandra.tcm.membership.NodeId;
+import org.apache.cassandra.tcm.ownership.DataPlacement;
+import org.apache.cassandra.tcm.ownership.MovementMap;
+import org.apache.cassandra.tcm.ownership.PlacementDeltas;
+import org.apache.cassandra.tcm.ownership.VersionedEndpoints;
+import org.apache.cassandra.tcm.sequences.LeaveStreams;
+import org.apache.cassandra.utils.CollectionSerializers;
+import org.apache.cassandra.utils.FBUtilities;
+import org.apache.cassandra.utils.concurrent.AsyncPromise;
+import org.apache.cassandra.utils.concurrent.Future;
+import org.apache.cassandra.utils.concurrent.FutureCombiner;
+
+import static java.lang.String.format;
+import static java.util.concurrent.TimeUnit.MILLISECONDS;
+import static java.util.concurrent.TimeUnit.NANOSECONDS;
+import static java.util.concurrent.TimeUnit.SECONDS;
+import static org.apache.cassandra.utils.Clock.Global.nanoTime;
+
+/*
+ * Sealing {@link Shard}s once a topology change obsoletes them.
+ *
+ * Happy Path Sequence:
+ * 1. Bring every participant up to (at least) this seal's epoch and fence allocation, so that everyone
+ * stops writing to the old shards; then wait until all in-progress mutations have been done.
+ * 2. Same for tracked transfers
+ * 3. Wait until all participating replicas have reconciled the union of their witnessed ids
+ * 4. Mark the shard as fully sealed
+ * 5. Stop including the shard in mutation summaries for read requests.
+ *
+ * Unhappy Path Sequence (not yet implemented):
+ * 1. Wait until everyone who's up stops writing to the old shards and all in-progress mutations have been done
+ * 2. Same for tracked transfers
+ * 3. Wait until all *live* participating replicas have reconciled the union of their witnessed ids
+ * 4. Mark the shard as partially sealed, label it with the set of participants that were able to seal
+ * 5. When a down node comes up, have it talk to other nodes and learn about sealing, and what mutations/logs
+ * they have exchanged. Diff with what we have - in system tables and in the journal (though system table
+ * plus local replay should give us an up-to-date metadata state for the shard)
+ */
+public final class SealingCoordinator
+{
+ /*
+ * BOOTSTRAP sequence illustrated. Stages in which write placements
+ * remain unchanged, and only read placements change, are skipped
+ * here as irrelevant to sharding / sealing.
+ *
+ * There is a tracked keyspace with RF=3.
+ * A node n35 with token=35 is added in the middle of the ring.
+ *
+ * 1. Initial state at e0:
+ *
+ * (10,20] | {n20,n30,n40} | e0 | ACTIVE
+ * (20,30] | {n30,n40,n50} | e0 | ACTIVE
+ * (30,40] | {n40,n50,n60} | e0 | ACTIVE
+ *
+ * 2. PREPARE_JOIN at e1:
+ *
+ * (10,20] | {n20,n30,n40} | e0 | ACTIVE
+ * (20,30] | {n30,n40,n50} | e0 | ACTIVE
+ * (30,40] | {n40,n50,n60} | e0 | ACTIVE
+ * + (30,35] | {n40,n50,n60} | e1 | ACTIVE
+ * + (35,40] | {n40,n50,n60} | e1 | ACTIVE
+ *
+ * 3. START_JOIN at e2:
+ *
+ * (10,20] | {n20,n30,n40} | e0 | ACTIVE
+ * + (10,20] | {n20,n30,n35,n40} | e2 | ACTIVE
+ * (20,30] | {n30,n40,n50} | e0 | ACTIVE
+ * + (20,30] | {n30,n35,n40,n50} | e2 | ACTIVE
+ * (30,40] | {n40,n50,n60} | e0 | ACTIVE
+ * (30,35] | {n40,n50,n60} | e1 | ACTIVE
+ * + (30,35] | {n35,n40,n50,n60} | e2 | ACTIVE
+ * (35,40] | {n40,n50,n60} | e1 | ACTIVE
+ *
+ * 4. MID_JOIN at e3 [*]:
+ *
+ * S (10,20] | {n20,n30,n40} | e0 | SEALED
+ * (10,20] | {n20,n30,n35,n40} | e2 | ACTIVE
+ * S (20,30] | {n30,n40,n50} | e0 | SEALED
+ * (20,30] | {n30,n35,n40,n50} | e2 | ACTIVE
+ * S (30,40] | {n40,n50,n60} | e0 | SEALED
+ * S (30,35] | {n40,n50,n60} | e1 | SEALED
+ * (30,35] | {n35,n40,n50,n60} | e2 | ACTIVE
+ * (35,40] | {n40,n50,n60} | e1 | ACTIVE
+ *
+ * 5. FINISH_JOIN at e4:
+ *
+ * (10,20] | {n20,n30,n40} | e0 | SEALED
+ * (10,20] | {n20,n30,n35,n40} | e2 | ACTIVE
+ * + (10,20] | {n20,n30,n35} | e4 | ACTIVE
+ * (20,30] | {n30,n40,n50} | e0 | SEALED
+ * (20,30] | {n30,n35,n40,n50} | e2 | ACTIVE
+ * + (20,30] | {n30,n35,n40} | e4 | ACTIVE
+ * (30,40] | {n40,n50,n60} | e0 | SEALED
+ * (30,35] | {n40,n50,n60} | e1 | SEALED
+ * (30,35] | {n35,n40,n50,n60} | e2 | ACTIVE
+ * + (30,35] | {n35,n40,n50} | e4 | ACTIVE
+ * (35,40] | {n40,n50,n60} | e1 | ACTIVE
+ *
+ * 6. UNLOCK_SEQUENCE:
+ *
+ * (10,20] | {n20,n30,n40} | e0 | SEALED
+ * S (10,20] | {n20,n30,n35,n40} | e2 | SEALED
+ * (10,20] | {n20,n30,n35} | e4 | ACTIVE
+ * (20,30] | {n30,n40,n50} | e0 | SEALED
+ * S (20,30] | {n30,n35,n40,n50} | e2 | SEALED
+ * (20,30] | {n30,n35,n40} | e4 | ACTIVE
+ * (30,40] | {n40,n50,n60} | e0 | SEALED
+ * (30,35] | {n40,n50,n60} | e1 | SEALED
+ * S (30,35] | {n35,n40,n50,n60} | e2 | SEALED
+ * (30,35] | {n35,n40,n50} | e4 | ACTIVE
+ * (35,40] | {n40,n50,n60} | e1 | ACTIVE
+ *
+ * [*] At this stage, after sealing the shards, we stream the sstables.
+ * All the offsets belonging to these sealed shards get stripped from
+ * stats component of streamed sstables. Otherwise, the new node
+ * won't be able to compact them or mark them as reconciled/repaired -
+ * it is rightfully unaware of the sealed shards.
+ */
+
+ /**
+ * Seal the shards obsoleted by bootstrap before the new node starts streaming.
+ */
+ public static void sealShardsAtMidJoin(ClusterMetadata metadata, MovementMap movements)
+ {
+ seal(discoverShardsAtMidJoinOrReplace(metadata, movements));
+ }
+
+ /**
+ * Seal the intermediate, over-replicated shards created during START_JOIN and obsoleted by FINISH_JOIN.
+ */
+ public static void sealShardsAtFinishJoin(ClusterMetadata metadata, long finishEpoch, PlacementDeltas finishDelta)
+ {
+ seal(discoverShardsAtFinishJoinOrReplace(metadata, finishEpoch, finishDelta, null));
+ }
+
+ /*
+ * REPLACE sequence illustrated. Stages in which write placements
+ * remain unchanged, and only read placements change, are skipped
+ * here as irrelevant to sharding / sealing.
+ *
+ * There is a tracked keyspace with RF=3.
+ * Node n40 is dead, and is being replaced with $n40.
+ *
+ * 1. Initial state at e0:
+ *
+ * (10,20] | {n20,n30,n40} | e0 | ACTIVE
+ * (20,30] | {n30,n40,n50} | e0 | ACTIVE
+ * (30,40] | {n40,n50,n60} | e0 | ACTIVE
+ *
+ * 2. START_REPLACE at e1:
+ *
+ * (10,20] | {n20,n30,n40} | e0 | ACTIVE
+ * + (10,20] | {n20,n30,n40,$n40}| e1 | ACTIVE
+ * (20,30] | {n30,n40,n50} | e0 | ACTIVE
+ * + (20,30] | {n30,n40,$n40,n50}| e1 | ACTIVE
+ * (30,40] | {n40,n50,n60} | e0 | ACTIVE
+ * + (30,40] | {n40,$n40,n50,n60}| e1 | ACTIVE
+ *
+ * 3. MID_REPLACE at e2 [*]:
+ *
+ * S (10,20] | {n20,n30,n40} | e0 | SEALED
+ * (10,20] | {n20,n30,n40,$n40}| e1 | ACTIVE
+ * S (20,30] | {n30,n40,n50} | e0 | SEALED
+ * (20,30] | {n30,n40,$n40,n50}| e1 | ACTIVE
+ * S (30,40] | {n40,n50,n60} | e0 | SEALED
+ * (30,40] | {n40,$n40,n50,n60}| e1 | ACTIVE
+ *
+ * 4. FINISH_REPLACE at e3:
+ *
+ * (10,20] | {n20,n30,n40} | e0 | SEALED
+ * (10,20] | {n20,n30,n40,$n40}| e1 | ACTIVE
+ * + (10,20] | {n20,n30,$n40} | e3 | ACTIVE
+ * (20,30] | {n30,n40,n50} | e0 | SEALED
+ * (20,30] | {n30,n40,$n40,n50}| e1 | ACTIVE
+ * + (20,30] | {n30,$n40,n50} | e3 | ACTIVE
+ * (30,40] | {n40,n50,n60} | e0 | SEALED
+ * (30,40] | {n40,$n40,n50,n60}| e1 | ACTIVE
+ * + (30,40] | {$n40,n50,n60} | e3 | ACTIVE
+ *
+ * 5. UNLOCK_SEQUENCE:
+ *
+ * (10,20] | {n20,n30,n40} | e0 | SEALED
+ * S (10,20] | {n20,n30,n40,$n40}| e1 | SEALED
+ * (10,20] | {n20,n30,$n40} | e3 | ACTIVE
+ * (20,30] | {n30,n40,n50} | e0 | SEALED
+ * S (20,30] | {n30,n40,$n40,n50}| e1 | SEALED
+ * (20,30] | {n30,$n40,n50} | e3 | ACTIVE
+ * (30,40] | {n40,n50,n60} | e0 | SEALED
+ * S (30,40] | {n40,$n40,n50,n60}| e1 | SEALED
+ * (30,40] | {$n40,n50,n60} | e3 | ACTIVE
+ *
+ * [*] At this stage, after sealing the shards, we stream the sstables.
+ * Non-repaired (not fully reconciled) SSTables must be split into
+ * fully reconciled part and non-reconciled part. When streaming the
+ * former, we will strip all the log offsets from their metadata.
+ * Otherwise, the new node won't be able to compact them or mark them
+ * as reconciled/repaired - it is rightfully unaware of the sealed shards.
+ * If we don't split them into two parts, then we risk being unable to
+ * rebuild and filter any of the streamed SSTables on the receiving node.
+ */
+
+ /**
+ * Seal the shards obsoleted by host replacement before the replacement node starts streaming.
+ * The {@code deadNode} node cannot participate in the sealing.
+ */
+ public static void sealShardsAtMidReplace(ClusterMetadata metadata, MovementMap movements, @Nonnull NodeId deadNode)
+ {
+ seal(discoverShardsAtMidJoinOrReplace(metadata, movements), deadNode);
+ }
+
+ /**
+ * Seal the intermediate, over-replicated shards created during START_REPLACE and obsoleted by FINISH_REPLACE.
+ * The {@code deadNode} node cannot participate in the sealing.
+ */
+ public static void sealShardsAtFinishReplace(ClusterMetadata metadata, long finishEpoch, PlacementDeltas finishDelta, @Nonnull NodeId deadNode)
+ {
+ seal(discoverShardsAtFinishJoinOrReplace(metadata, finishEpoch, finishDelta, deadNode), deadNode);
+ }
+
+ /**
+ * Discover all shards that need to be sealed at MID_JOIN or MID_REPLACE.
+ */
+ private static Set<ShardMetadata> discoverShardsAtMidJoinOrReplace(ClusterMetadata metadata, MovementMap movements)
+ {
+ List<Future<Set<ShardMetadata>>> futures = new ArrayList<>();
+
+ for (KeyspaceMetadata ksm : metadata.schema.getKeyspaces())
+ {
+ if (!ksm.params.replicationType.isTracked())
+ continue;
+
+ EndpointsByReplica acquired = movements.get(ksm.params.replication);
+ if (acquired.isEmpty()) // when the keyspace is not replicated to this DC
+ continue;
+
+ DataPlacement placement = metadata.placements.get(ksm.params.replication);
+
+ for (Map.Entry<Replica, EndpointsForRange> entry : acquired.entrySet())
+ {
+ Range<Token> range = entry.getKey().range();
+ Set<InetAddressAndPort> endpoints = entry.getValue().endpoints();
+ // cutoff: the epoch this node was added to the write placement for the range;
+ // shards over the range with sinceEpoch < cutoff are obsoleted by this bootstrap or replace.
+ long beforeEpoch = placement.writes.forRange(range).lastModified().getEpoch();
+ futures.add(FetchShards.fetch(ksm.name, beforeEpoch, range, endpoints));
+ }
+ }
+
+ Set<ShardMetadata> shards = new HashSet<>();
+ try
+ {
+ for (Set<ShardMetadata> fetched : FutureCombiner.allOf(futures).get(DatabaseDescriptor.getRpcTimeout(MILLISECONDS), MILLISECONDS))
+ shards.addAll(fetched);
+ }
+ catch (InterruptedException | ExecutionException | TimeoutException e)
+ {
+ throw new RuntimeException("Failed to fetch shards to seal", e);
+ }
+ return shards;
+ }
+
+ /**
+ * Discover all shards that need to be sealed at FINISH_JOIN or FINISH_REPLACE.
+ */
+ private static Set<ShardMetadata> discoverShardsAtFinishJoinOrReplace(
+ ClusterMetadata metadata, long finishEpoch, PlacementDeltas finishDelta, @Nullable NodeId deadNode)
+ {
+ Set<ShardMetadata> shards = new HashSet<>();
+ for (KeyspaceMetadata ksm : metadata.schema.getKeyspaces())
+ {
+ if (!ksm.params.replicationType.isTracked())
+ continue;
+
+ DataPlacement placement = metadata.placements.get(ksm.params.replication);
+ finishDelta.get(ksm.params.replication).writes.removals.flattenValues().forEach(removed -> {
Review Comment:
When the parent branch is next rebased/merged with trunk this will need a minor adjustment due to changes from CASSANDRA-21582. Nothing major, just something like:
```
finishDelta.get(ksm.params.replication).writes.removals(metadata.directory).flattenValues().forEach(...
```
##########
src/java/org/apache/cassandra/tcm/serialization/Version.java:
##########
@@ -90,7 +90,9 @@ public enum Version
* The version that Accord was added to TCM.
*/
public static final Version MIN_ACCORD_VERSION = V7;
+
public static final Version MIN_MUTATION_TRACKING_VERSION = V8;
+ public static final Version TBD_UNLOCK_STEP = V9;
Review Comment:
The other scenario to consider is a cluster wide upgrade to whichever version this lands in being interleaved with a bootstrap (or decommission/replacement/etc).
* Bootstrap of new node begins while the current cluster version is X (which precedes this patch).
* The `PREPARE_JOIN` is committed and encoded into the metadata log.
* Before the bootstrap proceeds any further, the cluster is upgraded to version X+1, including CEP-45 and this patch.
* As the bootstrapping node restarts, it will replay the `PREPARE_JOIN` from the log before any `STARTUP` transforms which ultimately bump `metadata.directory.commonSerializationVersion`. This means that the constituent transforms of the inflight MSO will be created and reified in `ClusterMetadata` before the version bump, so `FinishJoin.unlock` will be set to `true`.
* Having fully replayed the log to date, the bootstrapping node should then proceed to complete the MSO. With `unlock == true` it will not enter the `UNLOCK_SEQUENCE` branch of `BootstrapAndJoin::executeNext` and the sealing after `FINISH_JOIN` won't be done.
This is definitely a bit of an edge case, and could probably be mitigated in a similar way to how CASSANDRA-21582 handles the change in delta format for MSOs. i.e. don't permit last node to upgrade, and so bump the `commonSerializationVersion`, if there are in flight MSOs.
##########
src/java/org/apache/cassandra/replication/SealingCoordinator.java:
##########
@@ -0,0 +1,1401 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.cassandra.replication;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.TimeoutException;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.concurrent.locks.LockSupport;
+
+import javax.annotation.Nonnull;
+import javax.annotation.Nullable;
+
+import com.google.common.base.Preconditions;
+
+import org.apache.cassandra.config.DatabaseDescriptor;
+import org.apache.cassandra.db.ColumnFamilyStore;
+import org.apache.cassandra.db.Keyspace;
+import org.apache.cassandra.db.TypeSizes;
+import org.apache.cassandra.dht.AbstractBounds;
+import org.apache.cassandra.dht.IPartitioner;
+import org.apache.cassandra.dht.Range;
+import org.apache.cassandra.dht.Token;
+import org.apache.cassandra.exceptions.RequestFailure;
+import org.apache.cassandra.io.UnversionedSerializer;
+import org.apache.cassandra.io.util.DataInputPlus;
+import org.apache.cassandra.io.util.DataOutputPlus;
+import org.apache.cassandra.locator.EndpointsByReplica;
+import org.apache.cassandra.locator.EndpointsForRange;
+import org.apache.cassandra.locator.InetAddressAndPort;
+import org.apache.cassandra.locator.RangesAtEndpoint;
+import org.apache.cassandra.locator.Replica;
+import org.apache.cassandra.net.IVerbHandler;
+import org.apache.cassandra.net.Message;
+import org.apache.cassandra.net.MessagingService;
+import org.apache.cassandra.net.RequestCallback;
+import org.apache.cassandra.net.Verb;
+import org.apache.cassandra.schema.KeyspaceMetadata;
+import org.apache.cassandra.tcm.ClusterMetadata;
+import org.apache.cassandra.tcm.membership.NodeId;
+import org.apache.cassandra.tcm.ownership.DataPlacement;
+import org.apache.cassandra.tcm.ownership.MovementMap;
+import org.apache.cassandra.tcm.ownership.PlacementDeltas;
+import org.apache.cassandra.tcm.ownership.VersionedEndpoints;
+import org.apache.cassandra.tcm.sequences.LeaveStreams;
+import org.apache.cassandra.utils.CollectionSerializers;
+import org.apache.cassandra.utils.FBUtilities;
+import org.apache.cassandra.utils.concurrent.AsyncPromise;
+import org.apache.cassandra.utils.concurrent.Future;
+import org.apache.cassandra.utils.concurrent.FutureCombiner;
+
+import static java.lang.String.format;
+import static java.util.concurrent.TimeUnit.MILLISECONDS;
+import static java.util.concurrent.TimeUnit.NANOSECONDS;
+import static java.util.concurrent.TimeUnit.SECONDS;
+import static org.apache.cassandra.utils.Clock.Global.nanoTime;
+
+/*
+ * Sealing {@link Shard}s once a topology change obsoletes them.
+ *
+ * Happy Path Sequence:
+ * 1. Bring every participant up to (at least) this seal's epoch and fence allocation, so that everyone
+ * stops writing to the old shards; then wait until all in-progress mutations have been done.
+ * 2. Same for tracked transfers
+ * 3. Wait until all participating replicas have reconciled the union of their witnessed ids
+ * 4. Mark the shard as fully sealed
+ * 5. Stop including the shard in mutation summaries for read requests.
+ *
+ * Unhappy Path Sequence (not yet implemented):
+ * 1. Wait until everyone who's up stops writing to the old shards and all in-progress mutations have been done
+ * 2. Same for tracked transfers
+ * 3. Wait until all *live* participating replicas have reconciled the union of their witnessed ids
+ * 4. Mark the shard as partially sealed, label it with the set of participants that were able to seal
+ * 5. When a down node comes up, have it talk to other nodes and learn about sealing, and what mutations/logs
+ * they have exchanged. Diff with what we have - in system tables and in the journal (though system table
+ * plus local replay should give us an up-to-date metadata state for the shard)
+ */
+public final class SealingCoordinator
+{
+ /*
+ * BOOTSTRAP sequence illustrated. Stages in which write placements
+ * remain unchanged, and only read placements change, are skipped
+ * here as irrelevant to sharding / sealing.
+ *
+ * There is a tracked keyspace with RF=3.
+ * A node n35 with token=35 is added in the middle of the ring.
+ *
+ * 1. Initial state at e0:
+ *
+ * (10,20] | {n20,n30,n40} | e0 | ACTIVE
+ * (20,30] | {n30,n40,n50} | e0 | ACTIVE
+ * (30,40] | {n40,n50,n60} | e0 | ACTIVE
+ *
+ * 2. PREPARE_JOIN at e1:
+ *
+ * (10,20] | {n20,n30,n40} | e0 | ACTIVE
+ * (20,30] | {n30,n40,n50} | e0 | ACTIVE
+ * (30,40] | {n40,n50,n60} | e0 | ACTIVE
+ * + (30,35] | {n40,n50,n60} | e1 | ACTIVE
+ * + (35,40] | {n40,n50,n60} | e1 | ACTIVE
+ *
+ * 3. START_JOIN at e2:
+ *
+ * (10,20] | {n20,n30,n40} | e0 | ACTIVE
+ * + (10,20] | {n20,n30,n35,n40} | e2 | ACTIVE
+ * (20,30] | {n30,n40,n50} | e0 | ACTIVE
+ * + (20,30] | {n30,n35,n40,n50} | e2 | ACTIVE
+ * (30,40] | {n40,n50,n60} | e0 | ACTIVE
+ * (30,35] | {n40,n50,n60} | e1 | ACTIVE
+ * + (30,35] | {n35,n40,n50,n60} | e2 | ACTIVE
+ * (35,40] | {n40,n50,n60} | e1 | ACTIVE
+ *
+ * 4. MID_JOIN at e3 [*]:
+ *
+ * S (10,20] | {n20,n30,n40} | e0 | SEALED
+ * (10,20] | {n20,n30,n35,n40} | e2 | ACTIVE
+ * S (20,30] | {n30,n40,n50} | e0 | SEALED
+ * (20,30] | {n30,n35,n40,n50} | e2 | ACTIVE
+ * S (30,40] | {n40,n50,n60} | e0 | SEALED
+ * S (30,35] | {n40,n50,n60} | e1 | SEALED
+ * (30,35] | {n35,n40,n50,n60} | e2 | ACTIVE
+ * (35,40] | {n40,n50,n60} | e1 | ACTIVE
+ *
+ * 5. FINISH_JOIN at e4:
+ *
+ * (10,20] | {n20,n30,n40} | e0 | SEALED
+ * S (10,20] | {n20,n30,n35,n40} | e2 | SEALED
+ * + (10,20] | {n20,n30,n35} | e4 | ACTIVE
+ * (20,30] | {n30,n40,n50} | e0 | SEALED
+ * S (20,30] | {n30,n35,n40,n50} | e2 | SEALED
+ * + (20,30] | {n30,n35,n40} | e4 | ACTIVE
+ * (30,40] | {n40,n50,n60} | e0 | SEALED
+ * (30,35] | {n40,n50,n60} | e1 | SEALED
+ * S (30,35] | {n35,n40,n50,n60} | e2 | SEALED
+ * + (30,35] | {n35,n40,n50} | e4 | ACTIVE
+ * (35,40] | {n40,n50,n60} | e1 | ACTIVE
+ *
+ * [*] At this stage, after sealing the shards, we stream the sstables.
+ * All the offsets belonging to these sealed shards get stripped from
+ * stats component of streamed sstables. Otherwise, the new node
+ * won't be able to compact them or mark them as reconciled/repaired -
+ * it is rightfully unaware of the sealed shards.
+ */
+
+ /**
+ * Seal the shards obsoleted by bootstrap before the new node starts streaming.
+ */
+ public static void sealShardsAtMidJoin(ClusterMetadata metadata, MovementMap movements)
+ {
+ seal(discoverShardsAtMidJoinOrReplace(metadata, movements));
+ }
+
+ /**
+ * Seal the intermediate shards that were created during START_JOIN.
+ * TODO (required): this MUST eventually run while the affected ranges are still locked - between the
+ * removeFromWrites placement change and the range unlock. FinishJoin/FinishReplace currently apply
+ * removeFromWrites and the unlock atomically in a single transformation, so there is no committed
+ * dropped-but-locked epoch to hook. Until then, this runs just after FINISH_JOIN/FINISH_REPLACE,
+ * when the ranges are already unlocked. Left to implement for the unhappy path (AY).
+ */
+ public static void sealShardsAtFinishJoin(ClusterMetadata metadata, PlacementDeltas finishDelta)
+ {
+ seal(discoverShardsAtFinishJoinOrReplace(metadata, finishDelta, null));
+ }
+
+ /*
+ * REPLACE sequence illustrated. Stages in which write placements
+ * remain unchanged, and only read placements change, are skipped
+ * here as irrelevant to sharding / sealing.
+ *
+ * There is a tracked keyspace with RF=3.
+ * Node n40 is dead, and is being replaced with $n40.
+ *
+ * 1. Initial state at e0:
+ *
+ * (10,20] | {n20,n30,n40} | e0 | ACTIVE
+ * (20,30] | {n30,n40,n50} | e0 | ACTIVE
+ * (30,40] | {n40,n50,n60} | e0 | ACTIVE
+ *
+ * 2. START_REPLACE at e1:
+ *
+ * (10,20] | {n20,n30,n40} | e0 | ACTIVE
+ * + (10,20] | {n20,n30,n40,$n40}| e1 | ACTIVE
+ * (20,30] | {n30,n40,n50} | e0 | ACTIVE
+ * + (20,30] | {n30,n40,$n40,n50}| e1 | ACTIVE
+ * (30,40] | {n40,n50,n60} | e0 | ACTIVE
+ * + (30,40] | {n40,$n40,n50,n60}| e1 | ACTIVE
+ *
+ * 3. MID_REPLACE at e2 [*]:
+ *
+ * S (10,20] | {n20,n30,n40} | e0 | SEALED
+ * (10,20] | {n20,n30,n40,$n40}| e1 | ACTIVE
+ * S (20,30] | {n30,n40,n50} | e0 | SEALED
+ * (20,30] | {n30,n40,$n40,n50}| e1 | ACTIVE
+ * S (30,40] | {n40,n50,n60} | e0 | SEALED
+ * (30,40] | {n40,$n40,n50,n60}| e1 | ACTIVE
+ *
+ * 4. FINISH_REPLACE at e3:
+ *
+ * (10,20] | {n20,n30,n40} | e0 | SEALED
+ * S (10,20] | {n20,n30,n40,$n40}| e1 | SEALED
+ * + (10,20] | {n20,n30,$n40} | e3 | ACTIVE
+ * (20,30] | {n30,n40,n50} | e0 | SEALED
+ * S (20,30] | {n30,n40,$n40,n50}| e1 | SEALED
+ * + (20,30] | {n30,$n40,n50} | e3 | ACTIVE
+ * (30,40] | {n40,n50,n60} | e0 | SEALED
+ * S (30,40] | {n40,$n40,n50,n60}| e1 | SEALED
+ * + (30,40] | {$n40,n50,n60} | e3 | ACTIVE
+ *
+ * [*] At this stage, after sealing the shards, we stream the sstables.
+ * All the offsets belonging to these sealed shards get stripped from
+ * stats component of streamed sstables. Otherwise, the new node
+ * won't be able to compact them or mark them as reconciled/repaired -
+ * it is rightfully unaware of the sealed shards.
+ */
+
+ /**
+ * Seal the shards obsoleted by host replacement before the replacement node starts streaming.
+ * The {@code deadNode} node cannot participate in the sealing.
+ */
+ public static void sealShardsAtMidReplace(ClusterMetadata metadata, MovementMap movements, @Nonnull NodeId deadNode)
+ {
+ seal(discoverShardsAtMidJoinOrReplace(metadata, movements), deadNode);
+ }
+
+ public static void sealShardsAtFinishReplace(ClusterMetadata metadata, PlacementDeltas finishDelta, @Nonnull NodeId deadNode)
+ {
+ seal(discoverShardsAtFinishJoinOrReplace(metadata, finishDelta, deadNode), deadNode);
+ }
+
+ /**
+ * Discover all shards that need to be sealed at MID_JOIN or MID_REPLACE.
+ */
+ private static Set<ShardMetadata> discoverShardsAtMidJoinOrReplace(ClusterMetadata metadata, MovementMap movements)
+ {
+ List<Future<Set<ShardMetadata>>> futures = new ArrayList<>();
+
+ for (KeyspaceMetadata ksm : metadata.schema.getKeyspaces())
+ {
+ if (!ksm.params.replicationType.isTracked())
+ continue;
+
+ EndpointsByReplica acquired = movements.get(ksm.params.replication);
+ if (acquired.isEmpty()) // when the keyspace is not replicated to this DC
+ continue;
+
+ DataPlacement placement = metadata.placements.get(ksm.params.replication);
+
+ for (Map.Entry<Replica, EndpointsForRange> entry : acquired.entrySet())
+ {
+ Range<Token> range = entry.getKey().range();
+ Set<InetAddressAndPort> endpoints = entry.getValue().endpoints();
+ // cutoff: the epoch this node was added to the write placement for the range;
+ // shards over the range with sinceEpoch < cutoff are obsoleted by this bootstrap or replace.
+ long beforeEpoch = placement.writes.forRange(range).lastModified().getEpoch();
+ futures.add(FetchShards.fetch(ksm.name, beforeEpoch, range, endpoints));
+ }
+ }
+
+ Set<ShardMetadata> shards = new HashSet<>();
+ try
+ {
+ for (Set<ShardMetadata> fetched : FutureCombiner.allOf(futures).get(DatabaseDescriptor.getRpcTimeout(MILLISECONDS), MILLISECONDS))
+ shards.addAll(fetched);
+ }
+ catch (InterruptedException | ExecutionException | TimeoutException e)
+ {
+ throw new RuntimeException("Failed to fetch shards to seal", e);
+ }
+ return shards;
+ }
+
+ /**
+ * Discover all shards that need to be sealed at FINISH_JOIN or FINISH_REPLACE.
+ */
+ private static Set<ShardMetadata> discoverShardsAtFinishJoinOrReplace(ClusterMetadata metadata, PlacementDeltas finishDelta, @Nullable NodeId deadNode)
+ {
+ long finishEpoch = metadata.epoch.getEpoch();
+ Set<ShardMetadata> shards = new HashSet<>();
+ for (KeyspaceMetadata ksm : metadata.schema.getKeyspaces())
+ {
+ if (!ksm.params.replicationType.isTracked())
+ continue;
+
+ DataPlacement placement = metadata.placements.get(ksm.params.replication);
+ finishDelta.get(ksm.params.replication).writes.removals.flattenValues().forEach(removed -> {
+ Set<Integer> nodeIDs = new HashSet<>();
+ for (InetAddressAndPort ep : placement.writes.forRange(removed.range()).endpoints())
+ nodeIDs.add(metadata.directory.peerId(ep).id());
+ nodeIDs.add(deadNode != null ? deadNode.id() : metadata.directory.peerId(removed.endpoint()).id());
+ Participants participants = new Participants(nodeIDs);
+
+ MutationTrackingService.instance().forEachShardInKeyspace(ksm.name, shard -> {
+ if (shard.sinceEpoch < finishEpoch
+ && shard.range.equals(removed.range())
+ && shard.participants.equals(participants)
+ && !shard.isSealed())
+ shards.add(new ShardMetadata(shard.keyspace, shard.sinceEpoch, shard.range, shard.participants));
+ });
+ });
+ }
+ return shards;
+ }
+
+ /*
+ * UNBOOTSTRAP and REMOVENODE sequences illustrated (same sequence as far as sharding is concerned).
+ * Stages in which write placements remain unchanged, and only read placements change, are skipped
+ * here as irrelevant to sharding / sealing.
+ *
+ * There is a tracked keyspace with RF=3.
+ * A node n40 with token=40 is removed in the middle of the ring.
+ *
+ * 1. Initial state at e0:
+ *
+ * (10,20] | {n20,n30,n40} | e0 | ACTIVE
+ * (20,30] | {n30,n40,n50} | e0 | ACTIVE
+ * (30,40] | {n40,n50,n60} | e0 | ACTIVE
+ * (40,50] | {n50,n60,n70} | e0 | ACTIVE
+ *
+ * 2. START_LEAVE at e1:
+ *
+ * (10,20] | {n20,n30,n40} | e0 | ACTIVE
+ * + (10,20] | {n20,n30,n40,n50} | e1 | ACTIVE
+ * (20,30] | {n30,n40,n50} | e0 | ACTIVE
+ * + (20,30] | {n30,n40,n50,n60} | e1 | ACTIVE
+ * (30,40] | {n40,n50,n60} | e0 | ACTIVE
+ * + (30,40] | {n40,n50,n60,n70} | e1 | ACTIVE
+ * (40,50] | {n50,n60,n70} | e0 | ACTIVE
+ *
+ * 3. MID_LEAVE at e2 [*]:
+ *
+ * S (10,20] | {n20,n30,n40} | e0 | SEALED
+ * (10,20] | {n20,n30,n40,n50} | e1 | ACTIVE
+ * S (20,30] | {n30,n40,n50} | e0 | SEALED
+ * (20,30] | {n30,n40,n50,n60} | e1 | ACTIVE
+ * S (30,40] | {n40,n50,n60} | e0 | SEALED
+ * (30,40] | {n40,n50,n60,n70} | e1 | ACTIVE
+ * (40,50] | {n50,n60,n70} | e0 | ACTIVE
+ *
+ * 4. FINISH_LEAVE at e3:
+ *
+ * (10,20] | {n20,n30,n40} | e0 | SEALED
+ * S (10,20] | {n20,n30,n40,n50} | e1 | SEALED
+ * + (10,20] | {n20,n30,n50} | e3 | ACTIVE
+ * (20,30] | {n30,n40,n50} | e0 | SEALED
+ * S (20,30] | {n30,n40,n50,n60} | e1 | SEALED
+ * + (20,30] | {n30,n50,n60} | e3 | ACTIVE
+ * (30,40] | {n40,n50,n60} | e0 | SEALED
+ * S (30,40] | {n40,n50,n60,n70} | e1 | SEALED
+ * S (40,50] | {n50,n60,n70} | e0 | SEALED
+ * + (30,50] | {n50,n60,n70} | e3 | ACTIVE
+ *
+ * [*] At this stage, after sealing the shards, we stream the sstables.
+ * All the offsets belonging to these sealed shards get stripped from
+ * stats component of streamed sstables. Otherwise, the new node
+ * won't be able to compact them or mark them as reconciled/repaired -
+ * it is rightfully unaware of the sealed shards.
+ */
+
+ /**
+ * Seal the shards obsoleted by START_LEAVE before streaming, for both UNBOOTSTRAP and REMOVENODE.
+ */
+ public static void sealShardsAtMidLeave(ClusterMetadata metadata, PlacementDeltas startDelta, NodeId leavingOrDeparted, LeaveStreams.Kind kind)
+ {
+ InetAddressAndPort leavingOrDepartedEndpoint = metadata.directory.endpoint(leavingOrDeparted);
+
+ List<Future<Set<ShardMetadata>>> futures = new ArrayList<>();
+ for (KeyspaceMetadata ksm : metadata.schema.getKeyspaces())
+ {
+ if (!ksm.params.replicationType.isTracked())
+ continue;
+
+ DataPlacement placement = metadata.placements.get(ksm.params.replication);
+ startDelta.get(ksm.params.replication).writes.additions.flattenValues().forEach(added -> {
+ Range<Token> range = added.range();
+ long beforeEpoch = placement.writes.forRange(range).lastModified().getEpoch();
+ Set<InetAddressAndPort> survivors = new HashSet<>(placement.writes.forRange(range).endpoints());
+ survivors.remove(leavingOrDepartedEndpoint);
+ futures.add(FetchShards.fetch(ksm.name, beforeEpoch, range, survivors));
+ });
+ }
+
+ Set<ShardMetadata> shards = new HashSet<>();
+ try
+ {
+ for (Set<ShardMetadata> fetched : FutureCombiner.allOf(futures).get(DatabaseDescriptor.getRpcTimeout(MILLISECONDS), MILLISECONDS))
+ for (ShardMetadata shard : fetched)
+ if (shard.participants.contains(leavingOrDeparted.id()))
+ shards.add(shard);
+ }
+ catch (InterruptedException | ExecutionException | TimeoutException e)
+ {
+ throw new RuntimeException("Failed to fetch shards to seal for leave", e);
+ }
+
+ seal(shards, kind == LeaveStreams.Kind.UNBOOTSTRAP ? null : leavingOrDeparted);
+ }
+
+ /**
+ * Seal the shards that need to be sealed at LEAVE_FINISH for both UNBOOTSTRAP and REMOVENODE.
+ */
+ public static void sealShardsAtFinishLeave(ClusterMetadata postFinish, PlacementDeltas finishDelta, NodeId leavingOrDeparted, LeaveStreams.Kind kind)
+ {
+ long finishEpoch = postFinish.epoch.getEpoch();
+ InetAddressAndPort leavingOrDepartedEndpoint = postFinish.directory.endpoint(leavingOrDeparted);
+
+ List<Future<Set<ShardMetadata>>> futures = new ArrayList<>();
+ for (KeyspaceMetadata ksm : postFinish.schema.getKeyspaces())
+ {
+ if (!ksm.params.replicationType.isTracked())
+ continue;
+
+ DataPlacement placement = postFinish.placements.get(ksm.params.replication);
+ RangesAtEndpoint removals = finishDelta.get(ksm.params.replication).writes.removals.get(leavingOrDepartedEndpoint);
+ for (Replica removedReplica : removals)
+ {
+ // FINISH_LEAVE merges the leavingOrDeparted node's (pre-merge) range with an adjacent one. Query the *merged*
+ // range (not the pre-merge range) so discovery covers both the leavingOrDeparted node's own obsoleted
+ // generations and the merge-half - the adjacent pre-merge range folded into the merged range, whose
+ // shard never listed the leavingOrDeparted node. matchRange() yields the containing merged group + its
+ // current (survivor) write endpoints.
+ VersionedEndpoints.ForRange group = placement.writes.matchRange(removedReplica.range());
+ Range<Token> mergedRange = group.range();
+ Set<InetAddressAndPort> survivors = new HashSet<>(group.endpoints());
+ survivors.remove(leavingOrDepartedEndpoint);
+ futures.add(FetchShards.fetch(ksm.name, finishEpoch, mergedRange, survivors));
+ }
+ }
+
+ Set<ShardMetadata> shards = new HashSet<>();
+ try
+ {
+ for (Set<ShardMetadata> fetched : FutureCombiner.allOf(futures).get(DatabaseDescriptor.getRpcTimeout(MILLISECONDS), MILLISECONDS))
+ shards.addAll(fetched);
+ }
+ catch (InterruptedException | ExecutionException | TimeoutException e)
+ {
+ throw new RuntimeException("Failed to fetch shards to seal for leave finish", e);
+ }
+
+ seal(shards, kind == LeaveStreams.Kind.UNBOOTSTRAP ? null : leavingOrDeparted);
+ }
+
+ /*
+ * Sealing logic.
+ */
+
+ private static void seal(Set<ShardMetadata> shards)
+ {
+ seal(shards, null);
+ }
+
+ private static void seal(Set<ShardMetadata> shards, @Nullable NodeId withoutNode)
+ {
+ initiate(shards, withoutNode); // ACTIVE -> SEALING for each Shard
+ drain(shards, withoutNode); // drain in-flight local writes
+ reconcile(shards, withoutNode); // wait for logs to reconcile
+ complete(shards, withoutNode); // SEALING-> SEALED for each Shard + journal flush
+ }
+
+ private static void initiate(Set<ShardMetadata> shards, @Nullable NodeId withoutNode)
+ {
+ List<Future<Void>> futures = new ArrayList<>(shards.size());
+ for (ShardMetadata shard : shards)
+ futures.add(initiate(shard, withoutNode));
+ try
+ {
+ FutureCombiner.allOf(futures).get(DatabaseDescriptor.getRpcTimeout(MILLISECONDS), MILLISECONDS);
+ }
+ catch (InterruptedException | ExecutionException | TimeoutException e)
+ {
+ throw new RuntimeException("Failed to initiate sealing", e);
+ }
+ }
+
+ /**
+ * Fence allocation on every participant of the obsoleted shard (ACTIVE -> SEALING) by marking the shard
+ * SEALING on each.
+ * TODO (expected): this assumes every participant has already enacted the topology change that obsoleted the shard (so
+ * it no longer allocates new ids on it). The ProgressBarrier gating bootstrap()/MID_JOIN only waits for
+ * EACH_QUORUM of the affected replicas, NOT all live replicas, so a live-but-lagging participant could still
+ * be on the old shard. Bringing each participant up to the seal epoch must be done per-replica (not
+ * per-shard) and will be added separately. TLDR - add a step to guaranteed up-to-speed rest of replicas epochs.
+ */
+ private static AsyncPromise<Void> initiate(ShardMetadata shard, @Nullable NodeId withoutNode)
+ {
+ return InitSealing.initiate(shard.keyspace, shard.sinceEpoch, shard.range, toEndpoints(shard.participants, withoutNode));
+ }
+
+ private static void drain(Set<ShardMetadata> shards, @Nullable NodeId withoutNode)
+ {
+ long deadlineNanos = nanoTime() + DatabaseDescriptor.getRpcTimeout(NANOSECONDS);
Review Comment:
Also, the inline `Drain.poll` also uses the rpc request timeout when awaiting responses from every replica, which feels likely to result in timeouts if any single replica is slow to respond. The operational impact of that seems like it could be quite severe, depending on which point in a `MultiStepOperation` it occurs at (i.e. a single timeout during `sealShardsAtMidJoin|Replace` would leave the MSO in a halted state, like a bootstrap streaming failure would, which requires operator intervention).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CEP-45: Topology Change Support for Mutation Tracking [cassandra]
Posted by "iamaleksey (via GitHub)" <gi...@apache.org>.
iamaleksey commented on code in PR #4910:
URL: https://github.com/apache/cassandra/pull/4910#discussion_r4134697079
##########
src/java/org/apache/cassandra/replication/SealingCoordinator.java:
##########
@@ -0,0 +1,1401 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.cassandra.replication;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.TimeoutException;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.concurrent.locks.LockSupport;
+
+import javax.annotation.Nonnull;
+import javax.annotation.Nullable;
+
+import com.google.common.base.Preconditions;
+
+import org.apache.cassandra.config.DatabaseDescriptor;
+import org.apache.cassandra.db.ColumnFamilyStore;
+import org.apache.cassandra.db.Keyspace;
+import org.apache.cassandra.db.TypeSizes;
+import org.apache.cassandra.dht.AbstractBounds;
+import org.apache.cassandra.dht.IPartitioner;
+import org.apache.cassandra.dht.Range;
+import org.apache.cassandra.dht.Token;
+import org.apache.cassandra.exceptions.RequestFailure;
+import org.apache.cassandra.io.UnversionedSerializer;
+import org.apache.cassandra.io.util.DataInputPlus;
+import org.apache.cassandra.io.util.DataOutputPlus;
+import org.apache.cassandra.locator.EndpointsByReplica;
+import org.apache.cassandra.locator.EndpointsForRange;
+import org.apache.cassandra.locator.InetAddressAndPort;
+import org.apache.cassandra.locator.RangesAtEndpoint;
+import org.apache.cassandra.locator.Replica;
+import org.apache.cassandra.net.IVerbHandler;
+import org.apache.cassandra.net.Message;
+import org.apache.cassandra.net.MessagingService;
+import org.apache.cassandra.net.RequestCallback;
+import org.apache.cassandra.net.Verb;
+import org.apache.cassandra.schema.KeyspaceMetadata;
+import org.apache.cassandra.tcm.ClusterMetadata;
+import org.apache.cassandra.tcm.membership.NodeId;
+import org.apache.cassandra.tcm.ownership.DataPlacement;
+import org.apache.cassandra.tcm.ownership.MovementMap;
+import org.apache.cassandra.tcm.ownership.PlacementDeltas;
+import org.apache.cassandra.tcm.ownership.VersionedEndpoints;
+import org.apache.cassandra.tcm.sequences.LeaveStreams;
+import org.apache.cassandra.utils.CollectionSerializers;
+import org.apache.cassandra.utils.FBUtilities;
+import org.apache.cassandra.utils.concurrent.AsyncPromise;
+import org.apache.cassandra.utils.concurrent.Future;
+import org.apache.cassandra.utils.concurrent.FutureCombiner;
+
+import static java.lang.String.format;
+import static java.util.concurrent.TimeUnit.MILLISECONDS;
+import static java.util.concurrent.TimeUnit.NANOSECONDS;
+import static java.util.concurrent.TimeUnit.SECONDS;
+import static org.apache.cassandra.utils.Clock.Global.nanoTime;
+
+/*
+ * Sealing {@link Shard}s once a topology change obsoletes them.
+ *
+ * Happy Path Sequence:
+ * 1. Bring every participant up to (at least) this seal's epoch and fence allocation, so that everyone
+ * stops writing to the old shards; then wait until all in-progress mutations have been done.
+ * 2. Same for tracked transfers
+ * 3. Wait until all participating replicas have reconciled the union of their witnessed ids
+ * 4. Mark the shard as fully sealed
+ * 5. Stop including the shard in mutation summaries for read requests.
+ *
+ * Unhappy Path Sequence (not yet implemented):
+ * 1. Wait until everyone who's up stops writing to the old shards and all in-progress mutations have been done
+ * 2. Same for tracked transfers
+ * 3. Wait until all *live* participating replicas have reconciled the union of their witnessed ids
+ * 4. Mark the shard as partially sealed, label it with the set of participants that were able to seal
+ * 5. When a down node comes up, have it talk to other nodes and learn about sealing, and what mutations/logs
+ * they have exchanged. Diff with what we have - in system tables and in the journal (though system table
+ * plus local replay should give us an up-to-date metadata state for the shard)
+ */
+public final class SealingCoordinator
+{
+ /*
+ * BOOTSTRAP sequence illustrated. Stages in which write placements
+ * remain unchanged, and only read placements change, are skipped
+ * here as irrelevant to sharding / sealing.
+ *
+ * There is a tracked keyspace with RF=3.
+ * A node n35 with token=35 is added in the middle of the ring.
+ *
+ * 1. Initial state at e0:
+ *
+ * (10,20] | {n20,n30,n40} | e0 | ACTIVE
+ * (20,30] | {n30,n40,n50} | e0 | ACTIVE
+ * (30,40] | {n40,n50,n60} | e0 | ACTIVE
+ *
+ * 2. PREPARE_JOIN at e1:
+ *
+ * (10,20] | {n20,n30,n40} | e0 | ACTIVE
+ * (20,30] | {n30,n40,n50} | e0 | ACTIVE
+ * (30,40] | {n40,n50,n60} | e0 | ACTIVE
+ * + (30,35] | {n40,n50,n60} | e1 | ACTIVE
+ * + (35,40] | {n40,n50,n60} | e1 | ACTIVE
+ *
+ * 3. START_JOIN at e2:
+ *
+ * (10,20] | {n20,n30,n40} | e0 | ACTIVE
+ * + (10,20] | {n20,n30,n35,n40} | e2 | ACTIVE
+ * (20,30] | {n30,n40,n50} | e0 | ACTIVE
+ * + (20,30] | {n30,n35,n40,n50} | e2 | ACTIVE
+ * (30,40] | {n40,n50,n60} | e0 | ACTIVE
+ * (30,35] | {n40,n50,n60} | e1 | ACTIVE
+ * + (30,35] | {n35,n40,n50,n60} | e2 | ACTIVE
+ * (35,40] | {n40,n50,n60} | e1 | ACTIVE
+ *
+ * 4. MID_JOIN at e3 [*]:
+ *
+ * S (10,20] | {n20,n30,n40} | e0 | SEALED
+ * (10,20] | {n20,n30,n35,n40} | e2 | ACTIVE
+ * S (20,30] | {n30,n40,n50} | e0 | SEALED
+ * (20,30] | {n30,n35,n40,n50} | e2 | ACTIVE
+ * S (30,40] | {n40,n50,n60} | e0 | SEALED
+ * S (30,35] | {n40,n50,n60} | e1 | SEALED
+ * (30,35] | {n35,n40,n50,n60} | e2 | ACTIVE
+ * (35,40] | {n40,n50,n60} | e1 | ACTIVE
+ *
+ * 5. FINISH_JOIN at e4:
+ *
+ * (10,20] | {n20,n30,n40} | e0 | SEALED
+ * S (10,20] | {n20,n30,n35,n40} | e2 | SEALED
+ * + (10,20] | {n20,n30,n35} | e4 | ACTIVE
+ * (20,30] | {n30,n40,n50} | e0 | SEALED
+ * S (20,30] | {n30,n35,n40,n50} | e2 | SEALED
+ * + (20,30] | {n30,n35,n40} | e4 | ACTIVE
+ * (30,40] | {n40,n50,n60} | e0 | SEALED
+ * (30,35] | {n40,n50,n60} | e1 | SEALED
+ * S (30,35] | {n35,n40,n50,n60} | e2 | SEALED
+ * + (30,35] | {n35,n40,n50} | e4 | ACTIVE
+ * (35,40] | {n40,n50,n60} | e1 | ACTIVE
+ *
+ * [*] At this stage, after sealing the shards, we stream the sstables.
+ * All the offsets belonging to these sealed shards get stripped from
+ * stats component of streamed sstables. Otherwise, the new node
+ * won't be able to compact them or mark them as reconciled/repaired -
+ * it is rightfully unaware of the sealed shards.
+ */
+
+ /**
+ * Seal the shards obsoleted by bootstrap before the new node starts streaming.
+ */
+ public static void sealShardsAtMidJoin(ClusterMetadata metadata, MovementMap movements)
+ {
+ seal(discoverShardsAtMidJoinOrReplace(metadata, movements));
+ }
+
+ /**
+ * Seal the intermediate shards that were created during START_JOIN.
+ * TODO (required): this MUST eventually run while the affected ranges are still locked - between the
+ * removeFromWrites placement change and the range unlock. FinishJoin/FinishReplace currently apply
+ * removeFromWrites and the unlock atomically in a single transformation, so there is no committed
+ * dropped-but-locked epoch to hook. Until then, this runs just after FINISH_JOIN/FINISH_REPLACE,
+ * when the ranges are already unlocked. Left to implement for the unhappy path (AY).
+ */
+ public static void sealShardsAtFinishJoin(ClusterMetadata metadata, PlacementDeltas finishDelta)
+ {
+ seal(discoverShardsAtFinishJoinOrReplace(metadata, finishDelta, null));
+ }
+
+ /*
+ * REPLACE sequence illustrated. Stages in which write placements
+ * remain unchanged, and only read placements change, are skipped
+ * here as irrelevant to sharding / sealing.
+ *
+ * There is a tracked keyspace with RF=3.
+ * Node n40 is dead, and is being replaced with $n40.
+ *
+ * 1. Initial state at e0:
+ *
+ * (10,20] | {n20,n30,n40} | e0 | ACTIVE
+ * (20,30] | {n30,n40,n50} | e0 | ACTIVE
+ * (30,40] | {n40,n50,n60} | e0 | ACTIVE
+ *
+ * 2. START_REPLACE at e1:
+ *
+ * (10,20] | {n20,n30,n40} | e0 | ACTIVE
+ * + (10,20] | {n20,n30,n40,$n40}| e1 | ACTIVE
+ * (20,30] | {n30,n40,n50} | e0 | ACTIVE
+ * + (20,30] | {n30,n40,$n40,n50}| e1 | ACTIVE
+ * (30,40] | {n40,n50,n60} | e0 | ACTIVE
+ * + (30,40] | {n40,$n40,n50,n60}| e1 | ACTIVE
+ *
+ * 3. MID_REPLACE at e2 [*]:
+ *
+ * S (10,20] | {n20,n30,n40} | e0 | SEALED
+ * (10,20] | {n20,n30,n40,$n40}| e1 | ACTIVE
+ * S (20,30] | {n30,n40,n50} | e0 | SEALED
+ * (20,30] | {n30,n40,$n40,n50}| e1 | ACTIVE
+ * S (30,40] | {n40,n50,n60} | e0 | SEALED
+ * (30,40] | {n40,$n40,n50,n60}| e1 | ACTIVE
+ *
+ * 4. FINISH_REPLACE at e3:
+ *
+ * (10,20] | {n20,n30,n40} | e0 | SEALED
+ * S (10,20] | {n20,n30,n40,$n40}| e1 | SEALED
+ * + (10,20] | {n20,n30,$n40} | e3 | ACTIVE
+ * (20,30] | {n30,n40,n50} | e0 | SEALED
+ * S (20,30] | {n30,n40,$n40,n50}| e1 | SEALED
+ * + (20,30] | {n30,$n40,n50} | e3 | ACTIVE
+ * (30,40] | {n40,n50,n60} | e0 | SEALED
+ * S (30,40] | {n40,$n40,n50,n60}| e1 | SEALED
+ * + (30,40] | {$n40,n50,n60} | e3 | ACTIVE
+ *
+ * [*] At this stage, after sealing the shards, we stream the sstables.
+ * All the offsets belonging to these sealed shards get stripped from
+ * stats component of streamed sstables. Otherwise, the new node
+ * won't be able to compact them or mark them as reconciled/repaired -
+ * it is rightfully unaware of the sealed shards.
+ */
+
+ /**
+ * Seal the shards obsoleted by host replacement before the replacement node starts streaming.
+ * The {@code deadNode} node cannot participate in the sealing.
+ */
+ public static void sealShardsAtMidReplace(ClusterMetadata metadata, MovementMap movements, @Nonnull NodeId deadNode)
+ {
+ seal(discoverShardsAtMidJoinOrReplace(metadata, movements), deadNode);
+ }
+
+ public static void sealShardsAtFinishReplace(ClusterMetadata metadata, PlacementDeltas finishDelta, @Nonnull NodeId deadNode)
+ {
+ seal(discoverShardsAtFinishJoinOrReplace(metadata, finishDelta, deadNode), deadNode);
+ }
+
+ /**
+ * Discover all shards that need to be sealed at MID_JOIN or MID_REPLACE.
+ */
+ private static Set<ShardMetadata> discoverShardsAtMidJoinOrReplace(ClusterMetadata metadata, MovementMap movements)
+ {
+ List<Future<Set<ShardMetadata>>> futures = new ArrayList<>();
+
+ for (KeyspaceMetadata ksm : metadata.schema.getKeyspaces())
+ {
+ if (!ksm.params.replicationType.isTracked())
+ continue;
+
+ EndpointsByReplica acquired = movements.get(ksm.params.replication);
+ if (acquired.isEmpty()) // when the keyspace is not replicated to this DC
+ continue;
+
+ DataPlacement placement = metadata.placements.get(ksm.params.replication);
+
+ for (Map.Entry<Replica, EndpointsForRange> entry : acquired.entrySet())
+ {
+ Range<Token> range = entry.getKey().range();
+ Set<InetAddressAndPort> endpoints = entry.getValue().endpoints();
+ // cutoff: the epoch this node was added to the write placement for the range;
+ // shards over the range with sinceEpoch < cutoff are obsoleted by this bootstrap or replace.
+ long beforeEpoch = placement.writes.forRange(range).lastModified().getEpoch();
+ futures.add(FetchShards.fetch(ksm.name, beforeEpoch, range, endpoints));
+ }
+ }
+
+ Set<ShardMetadata> shards = new HashSet<>();
+ try
+ {
+ for (Set<ShardMetadata> fetched : FutureCombiner.allOf(futures).get(DatabaseDescriptor.getRpcTimeout(MILLISECONDS), MILLISECONDS))
+ shards.addAll(fetched);
+ }
+ catch (InterruptedException | ExecutionException | TimeoutException e)
+ {
+ throw new RuntimeException("Failed to fetch shards to seal", e);
+ }
+ return shards;
+ }
+
+ /**
+ * Discover all shards that need to be sealed at FINISH_JOIN or FINISH_REPLACE.
+ */
+ private static Set<ShardMetadata> discoverShardsAtFinishJoinOrReplace(ClusterMetadata metadata, PlacementDeltas finishDelta, @Nullable NodeId deadNode)
+ {
+ long finishEpoch = metadata.epoch.getEpoch();
+ Set<ShardMetadata> shards = new HashSet<>();
+ for (KeyspaceMetadata ksm : metadata.schema.getKeyspaces())
+ {
+ if (!ksm.params.replicationType.isTracked())
+ continue;
+
+ DataPlacement placement = metadata.placements.get(ksm.params.replication);
+ finishDelta.get(ksm.params.replication).writes.removals.flattenValues().forEach(removed -> {
+ Set<Integer> nodeIDs = new HashSet<>();
+ for (InetAddressAndPort ep : placement.writes.forRange(removed.range()).endpoints())
+ nodeIDs.add(metadata.directory.peerId(ep).id());
+ nodeIDs.add(deadNode != null ? deadNode.id() : metadata.directory.peerId(removed.endpoint()).id());
+ Participants participants = new Participants(nodeIDs);
+
+ MutationTrackingService.instance().forEachShardInKeyspace(ksm.name, shard -> {
+ if (shard.sinceEpoch < finishEpoch
+ && shard.range.equals(removed.range())
+ && shard.participants.equals(participants)
+ && !shard.isSealed())
+ shards.add(new ShardMetadata(shard.keyspace, shard.sinceEpoch, shard.range, shard.participants));
+ });
+ });
+ }
+ return shards;
+ }
+
+ /*
+ * UNBOOTSTRAP and REMOVENODE sequences illustrated (same sequence as far as sharding is concerned).
+ * Stages in which write placements remain unchanged, and only read placements change, are skipped
+ * here as irrelevant to sharding / sealing.
+ *
+ * There is a tracked keyspace with RF=3.
+ * A node n40 with token=40 is removed in the middle of the ring.
+ *
+ * 1. Initial state at e0:
+ *
+ * (10,20] | {n20,n30,n40} | e0 | ACTIVE
+ * (20,30] | {n30,n40,n50} | e0 | ACTIVE
+ * (30,40] | {n40,n50,n60} | e0 | ACTIVE
+ * (40,50] | {n50,n60,n70} | e0 | ACTIVE
+ *
+ * 2. START_LEAVE at e1:
+ *
+ * (10,20] | {n20,n30,n40} | e0 | ACTIVE
+ * + (10,20] | {n20,n30,n40,n50} | e1 | ACTIVE
+ * (20,30] | {n30,n40,n50} | e0 | ACTIVE
+ * + (20,30] | {n30,n40,n50,n60} | e1 | ACTIVE
+ * (30,40] | {n40,n50,n60} | e0 | ACTIVE
+ * + (30,40] | {n40,n50,n60,n70} | e1 | ACTIVE
+ * (40,50] | {n50,n60,n70} | e0 | ACTIVE
+ *
+ * 3. MID_LEAVE at e2 [*]:
+ *
+ * S (10,20] | {n20,n30,n40} | e0 | SEALED
+ * (10,20] | {n20,n30,n40,n50} | e1 | ACTIVE
+ * S (20,30] | {n30,n40,n50} | e0 | SEALED
+ * (20,30] | {n30,n40,n50,n60} | e1 | ACTIVE
+ * S (30,40] | {n40,n50,n60} | e0 | SEALED
+ * (30,40] | {n40,n50,n60,n70} | e1 | ACTIVE
+ * (40,50] | {n50,n60,n70} | e0 | ACTIVE
+ *
+ * 4. FINISH_LEAVE at e3:
+ *
+ * (10,20] | {n20,n30,n40} | e0 | SEALED
+ * S (10,20] | {n20,n30,n40,n50} | e1 | SEALED
+ * + (10,20] | {n20,n30,n50} | e3 | ACTIVE
+ * (20,30] | {n30,n40,n50} | e0 | SEALED
+ * S (20,30] | {n30,n40,n50,n60} | e1 | SEALED
+ * + (20,30] | {n30,n50,n60} | e3 | ACTIVE
+ * (30,40] | {n40,n50,n60} | e0 | SEALED
+ * S (30,40] | {n40,n50,n60,n70} | e1 | SEALED
+ * S (40,50] | {n50,n60,n70} | e0 | SEALED
+ * + (30,50] | {n50,n60,n70} | e3 | ACTIVE
+ *
+ * [*] At this stage, after sealing the shards, we stream the sstables.
+ * All the offsets belonging to these sealed shards get stripped from
+ * stats component of streamed sstables. Otherwise, the new node
+ * won't be able to compact them or mark them as reconciled/repaired -
+ * it is rightfully unaware of the sealed shards.
+ */
+
+ /**
+ * Seal the shards obsoleted by START_LEAVE before streaming, for both UNBOOTSTRAP and REMOVENODE.
+ */
+ public static void sealShardsAtMidLeave(ClusterMetadata metadata, PlacementDeltas startDelta, NodeId leavingOrDeparted, LeaveStreams.Kind kind)
+ {
+ InetAddressAndPort leavingOrDepartedEndpoint = metadata.directory.endpoint(leavingOrDeparted);
+
+ List<Future<Set<ShardMetadata>>> futures = new ArrayList<>();
+ for (KeyspaceMetadata ksm : metadata.schema.getKeyspaces())
+ {
+ if (!ksm.params.replicationType.isTracked())
+ continue;
+
+ DataPlacement placement = metadata.placements.get(ksm.params.replication);
+ startDelta.get(ksm.params.replication).writes.additions.flattenValues().forEach(added -> {
+ Range<Token> range = added.range();
+ long beforeEpoch = placement.writes.forRange(range).lastModified().getEpoch();
+ Set<InetAddressAndPort> survivors = new HashSet<>(placement.writes.forRange(range).endpoints());
+ survivors.remove(leavingOrDepartedEndpoint);
+ futures.add(FetchShards.fetch(ksm.name, beforeEpoch, range, survivors));
+ });
+ }
+
+ Set<ShardMetadata> shards = new HashSet<>();
+ try
+ {
+ for (Set<ShardMetadata> fetched : FutureCombiner.allOf(futures).get(DatabaseDescriptor.getRpcTimeout(MILLISECONDS), MILLISECONDS))
+ for (ShardMetadata shard : fetched)
+ if (shard.participants.contains(leavingOrDeparted.id()))
+ shards.add(shard);
+ }
+ catch (InterruptedException | ExecutionException | TimeoutException e)
+ {
+ throw new RuntimeException("Failed to fetch shards to seal for leave", e);
+ }
+
+ seal(shards, kind == LeaveStreams.Kind.UNBOOTSTRAP ? null : leavingOrDeparted);
+ }
+
+ /**
+ * Seal the shards that need to be sealed at LEAVE_FINISH for both UNBOOTSTRAP and REMOVENODE.
+ */
+ public static void sealShardsAtFinishLeave(ClusterMetadata postFinish, PlacementDeltas finishDelta, NodeId leavingOrDeparted, LeaveStreams.Kind kind)
+ {
+ long finishEpoch = postFinish.epoch.getEpoch();
+ InetAddressAndPort leavingOrDepartedEndpoint = postFinish.directory.endpoint(leavingOrDeparted);
+
+ List<Future<Set<ShardMetadata>>> futures = new ArrayList<>();
+ for (KeyspaceMetadata ksm : postFinish.schema.getKeyspaces())
+ {
+ if (!ksm.params.replicationType.isTracked())
+ continue;
+
+ DataPlacement placement = postFinish.placements.get(ksm.params.replication);
+ RangesAtEndpoint removals = finishDelta.get(ksm.params.replication).writes.removals.get(leavingOrDepartedEndpoint);
+ for (Replica removedReplica : removals)
+ {
+ // FINISH_LEAVE merges the leavingOrDeparted node's (pre-merge) range with an adjacent one. Query the *merged*
+ // range (not the pre-merge range) so discovery covers both the leavingOrDeparted node's own obsoleted
+ // generations and the merge-half - the adjacent pre-merge range folded into the merged range, whose
+ // shard never listed the leavingOrDeparted node. matchRange() yields the containing merged group + its
+ // current (survivor) write endpoints.
+ VersionedEndpoints.ForRange group = placement.writes.matchRange(removedReplica.range());
+ Range<Token> mergedRange = group.range();
+ Set<InetAddressAndPort> survivors = new HashSet<>(group.endpoints());
+ survivors.remove(leavingOrDepartedEndpoint);
+ futures.add(FetchShards.fetch(ksm.name, finishEpoch, mergedRange, survivors));
+ }
+ }
+
+ Set<ShardMetadata> shards = new HashSet<>();
+ try
+ {
+ for (Set<ShardMetadata> fetched : FutureCombiner.allOf(futures).get(DatabaseDescriptor.getRpcTimeout(MILLISECONDS), MILLISECONDS))
+ shards.addAll(fetched);
+ }
+ catch (InterruptedException | ExecutionException | TimeoutException e)
+ {
+ throw new RuntimeException("Failed to fetch shards to seal for leave finish", e);
+ }
+
+ seal(shards, kind == LeaveStreams.Kind.UNBOOTSTRAP ? null : leavingOrDeparted);
+ }
+
+ /*
+ * Sealing logic.
+ */
+
+ private static void seal(Set<ShardMetadata> shards)
+ {
+ seal(shards, null);
+ }
+
+ private static void seal(Set<ShardMetadata> shards, @Nullable NodeId withoutNode)
+ {
+ initiate(shards, withoutNode); // ACTIVE -> SEALING for each Shard
+ drain(shards, withoutNode); // drain in-flight local writes
+ reconcile(shards, withoutNode); // wait for logs to reconcile
+ complete(shards, withoutNode); // SEALING-> SEALED for each Shard + journal flush
+ }
+
+ private static void initiate(Set<ShardMetadata> shards, @Nullable NodeId withoutNode)
+ {
+ List<Future<Void>> futures = new ArrayList<>(shards.size());
+ for (ShardMetadata shard : shards)
+ futures.add(initiate(shard, withoutNode));
+ try
+ {
+ FutureCombiner.allOf(futures).get(DatabaseDescriptor.getRpcTimeout(MILLISECONDS), MILLISECONDS);
+ }
+ catch (InterruptedException | ExecutionException | TimeoutException e)
+ {
+ throw new RuntimeException("Failed to initiate sealing", e);
+ }
+ }
+
+ /**
+ * Fence allocation on every participant of the obsoleted shard (ACTIVE -> SEALING) by marking the shard
+ * SEALING on each.
+ * TODO (expected): this assumes every participant has already enacted the topology change that obsoleted the shard (so
+ * it no longer allocates new ids on it). The ProgressBarrier gating bootstrap()/MID_JOIN only waits for
+ * EACH_QUORUM of the affected replicas, NOT all live replicas, so a live-but-lagging participant could still
+ * be on the old shard. Bringing each participant up to the seal epoch must be done per-replica (not
+ * per-shard) and will be added separately. TLDR - add a step to guaranteed up-to-speed rest of replicas epochs.
+ */
+ private static AsyncPromise<Void> initiate(ShardMetadata shard, @Nullable NodeId withoutNode)
+ {
+ return InitSealing.initiate(shard.keyspace, shard.sinceEpoch, shard.range, toEndpoints(shard.participants, withoutNode));
+ }
+
+ private static void drain(Set<ShardMetadata> shards, @Nullable NodeId withoutNode)
+ {
+ long deadlineNanos = nanoTime() + DatabaseDescriptor.getRpcTimeout(NANOSECONDS);
+ Set<ShardMetadata> pending = new HashSet<>(shards);
+ while (true)
+ {
+ // non-blocking poll: each participant reports its drain status immediately; retry until all drained
+ pending.removeIf(shard -> drain(shard, withoutNode));
+ if (pending.isEmpty())
+ return;
+ if (nanoTime() >= deadlineNanos)
+ throw new RuntimeException("Timed out draining shards; still pending: " + pending);
+ LockSupport.parkNanos(SECONDS.toNanos(1));
+ }
+ }
+
+ /**
+ * Poll every participant of the SEALING shard once for drain status; true iff all report that no mutation
+ * id allocated before the SEALING fence is still applying locally.
+ */
+ private static boolean drain(ShardMetadata shard, @Nullable NodeId withoutNode)
+ {
+ try
+ {
+ return Drain.poll(shard.keyspace, shard.sinceEpoch, shard.range, toEndpoints(shard.participants, withoutNode))
+ .get(DatabaseDescriptor.getRpcTimeout(MILLISECONDS), MILLISECONDS);
+ }
+ catch (InterruptedException | ExecutionException | TimeoutException e)
+ {
+ throw new RuntimeException(format("Failed to poll drain for shard %s", shard), e);
+ }
+ }
+
+ /**
+ * For each of the shards, capture the witnessed union then poll each participant until it has
+ * caught up and witnessed the entire offset union itself.
+ */
+ private static void reconcile(Set<ShardMetadata> shards, @Nullable NodeId withoutNode)
+ {
+ for (ShardMetadata shard : shards)
+ reconcile(shard, withoutNode);
+ }
+
+ private static void reconcile(ShardMetadata shard, @Nullable NodeId withoutNode)
+ {
+ // capture shard's witnessed offsets once
+ Log2OffsetsMap.Mutable offsets = captureWitnessedOffsets(shard, withoutNode);
+ // poll every participant until they've each witnessed the union of offsets
+ for (InetAddressAndPort endpoint : toEndpoints(shard.participants, withoutNode))
+ pollUntilWitnesses(shard, offsets, endpoint);
+ }
+
+ private static Log2OffsetsMap.Mutable captureWitnessedOffsets(ShardMetadata shard, @Nullable NodeId withoutNode)
+ {
+ List<InetAddressAndPort> endpoints = toEndpoints(shard.participants, withoutNode);
+ try
+ {
+ return ReconcileCapture.capture(shard.keyspace, shard.sinceEpoch, shard.range, endpoints)
+ .get(DatabaseDescriptor.getRpcTimeout(MILLISECONDS), MILLISECONDS);
+ }
+ catch (InterruptedException | ExecutionException | TimeoutException e)
+ {
+ throw new RuntimeException(format("Failed to capture witnessed offsets for shard %s", shard), e);
+ }
+ }
+
+ private static void pollUntilWitnesses(ShardMetadata shard, Log2OffsetsMap.Mutable offsets, InetAddressAndPort endpoint)
+ {
+ // TODO (expected): use a longer timeout
+ long deadlineNanos = nanoTime() + DatabaseDescriptor.getRpcTimeout(NANOSECONDS);
+
+ while (true)
+ {
+ if (nanoTime() >= deadlineNanos)
+ throw new RuntimeException("Timed out reconciling shard: " + shard);
+ try
+ {
+ boolean witnessed =
+ ReconcilePoll.poll(shard.keyspace, shard.sinceEpoch, shard.range, offsets, endpoint)
+ .get(DatabaseDescriptor.getRpcTimeout(MILLISECONDS), MILLISECONDS);
+ if (witnessed)
+ return;
+ }
+ catch (InterruptedException | ExecutionException | TimeoutException e)
+ {
+ throw new RuntimeException(format("Failed to poll reconcile for shard %s", shard), e);
+ }
+
+ LockSupport.parkNanos(SECONDS.toNanos(1));
+ }
+ }
+
+ private static void complete(Set<ShardMetadata> shards, @Nullable NodeId withoutNode)
+ {
+ List<Future<Void>> futures = new ArrayList<>(shards.size());
+ for (ShardMetadata shard : shards)
+ futures.add(complete(shard, withoutNode));
+
+ try
+ {
+ FutureCombiner.allOf(futures).get(DatabaseDescriptor.getRpcTimeout(MILLISECONDS), MILLISECONDS);
+ }
+ catch (InterruptedException | ExecutionException | TimeoutException e)
+ {
+ throw new RuntimeException("Failed to complete sealing", e);
+ }
+ }
+
+ /**
+ * Promote every participant of the (drained and reconciled) shard from SEALING to SEALED.
+ */
+ private static AsyncPromise<Void> complete(ShardMetadata shard, @Nullable NodeId withoutNode)
+ {
+ return CompleteSealing.complete(shard.keyspace, shard.sinceEpoch, shard.range, toEndpoints(shard.participants, withoutNode));
+ }
+
+ /**
+ * A bootstrapping node knows nothing about the shards that currently exist,
+ * so it must collect the list of shards from the existing replicas before
+ * it can proceed with the rest of the sealing steps.
+ */
+ public static final class FetchShards
+ {
+ public static final class Request
+ {
+ final String keyspace;
+ final long beforeEpoch;
+ final Range<Token> range;
+
+ Request(String keyspace, long beforeEpoch, Range<Token> range)
+ {
+ this.keyspace = keyspace;
+ this.beforeEpoch = beforeEpoch;
+ this.range = range;
+ }
+ }
+
+ public static final class Response
+ {
+ final Set<ShardMetadata> shards;
+
+ Response(Set<ShardMetadata> shards)
+ {
+ this.shards = shards;
+ }
+ }
+
+ /**
+ * Query all replicas in a group in parallel for the shards over {@code ranges} obsoleted by an
+ * in-flight topology change (those with {@code sinceEpoch < beforeEpoch}). Resembles
+ * {@link ShardMetadataRequest#queryPeers}, but waits for every (live) replica to respond and
+ * merges their deduplicated shards rather than taking the first response.
+ */
+ public static AsyncPromise<Set<ShardMetadata>> fetch(
+ String keyspace, long beforeEpoch, Range<Token> range, Set<InetAddressAndPort> endpoints)
+ {
+ Preconditions.checkArgument(!endpoints.isEmpty());
+ AsyncPromise<Set<ShardMetadata>> promise = new AsyncPromise<>();
+ Set<ShardMetadata> merged = ConcurrentHashMap.newKeySet();
+
+ RequestCallback<Response> callback = new RequestCallback<>()
+ {
+ private final AtomicInteger remaining = new AtomicInteger(endpoints.size());
+
+ @Override
+ public void onResponse(Message<Response> msg)
+ {
+ merged.addAll(msg.payload.shards);
+ if (remaining.decrementAndGet() == 0)
+ promise.trySuccess(merged);
+ }
+
+ @Override
+ public void onFailure(InetAddressAndPort from, RequestFailure failure)
+ {
+ // happy path assumes all replicas are up; fail the whole fetch if any can't respond
+ promise.tryFailure(new RuntimeException(format("Failed to fetch shards to seal from %s for keyspace %s: %s", from, keyspace, failure)));
+ }
+
+ @Override
+ public boolean invokeOnFailure()
+ {
+ return true;
+ }
+ };
+
+ Message<Request> message = Message.out(Verb.MT_FETCH_SHARDS_REQ, new Request(keyspace, beforeEpoch, range));
+ for (InetAddressAndPort peer : endpoints)
+ MessagingService.instance().sendWithCallback(message, peer, callback);
+
+ return promise;
+ }
+
+ public static final IVerbHandler<Request> verbHandler = message ->
+ {
+ MutationTrackingService.ensureEnabled();
+ Request request = message.payload;
+ Set<ShardMetadata> shards = new HashSet<>();
+ MutationTrackingService.instance().forEachShardInKeyspace(request.keyspace, shard -> {
+ if (shard.sinceEpoch < request.beforeEpoch
+ && (request.range.contains(shard.range) || shard.range.contains(request.range))
Review Comment:
This is correct today but `range#intersects()` is also fine and is a little more succinct. Done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CEP-45: Topology Change Support for Mutation Tracking [cassandra]
Posted by "iamaleksey (via GitHub)" <gi...@apache.org>.
iamaleksey commented on code in PR #4910:
URL: https://github.com/apache/cassandra/pull/4910#discussion_r4135303806
##########
src/java/org/apache/cassandra/replication/SealingCoordinator.java:
##########
@@ -0,0 +1,1401 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.cassandra.replication;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.TimeoutException;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.concurrent.locks.LockSupport;
+
+import javax.annotation.Nonnull;
+import javax.annotation.Nullable;
+
+import com.google.common.base.Preconditions;
+
+import org.apache.cassandra.config.DatabaseDescriptor;
+import org.apache.cassandra.db.ColumnFamilyStore;
+import org.apache.cassandra.db.Keyspace;
+import org.apache.cassandra.db.TypeSizes;
+import org.apache.cassandra.dht.AbstractBounds;
+import org.apache.cassandra.dht.IPartitioner;
+import org.apache.cassandra.dht.Range;
+import org.apache.cassandra.dht.Token;
+import org.apache.cassandra.exceptions.RequestFailure;
+import org.apache.cassandra.io.UnversionedSerializer;
+import org.apache.cassandra.io.util.DataInputPlus;
+import org.apache.cassandra.io.util.DataOutputPlus;
+import org.apache.cassandra.locator.EndpointsByReplica;
+import org.apache.cassandra.locator.EndpointsForRange;
+import org.apache.cassandra.locator.InetAddressAndPort;
+import org.apache.cassandra.locator.RangesAtEndpoint;
+import org.apache.cassandra.locator.Replica;
+import org.apache.cassandra.net.IVerbHandler;
+import org.apache.cassandra.net.Message;
+import org.apache.cassandra.net.MessagingService;
+import org.apache.cassandra.net.RequestCallback;
+import org.apache.cassandra.net.Verb;
+import org.apache.cassandra.schema.KeyspaceMetadata;
+import org.apache.cassandra.tcm.ClusterMetadata;
+import org.apache.cassandra.tcm.membership.NodeId;
+import org.apache.cassandra.tcm.ownership.DataPlacement;
+import org.apache.cassandra.tcm.ownership.MovementMap;
+import org.apache.cassandra.tcm.ownership.PlacementDeltas;
+import org.apache.cassandra.tcm.ownership.VersionedEndpoints;
+import org.apache.cassandra.tcm.sequences.LeaveStreams;
+import org.apache.cassandra.utils.CollectionSerializers;
+import org.apache.cassandra.utils.FBUtilities;
+import org.apache.cassandra.utils.concurrent.AsyncPromise;
+import org.apache.cassandra.utils.concurrent.Future;
+import org.apache.cassandra.utils.concurrent.FutureCombiner;
+
+import static java.lang.String.format;
+import static java.util.concurrent.TimeUnit.MILLISECONDS;
+import static java.util.concurrent.TimeUnit.NANOSECONDS;
+import static java.util.concurrent.TimeUnit.SECONDS;
+import static org.apache.cassandra.utils.Clock.Global.nanoTime;
+
+/*
+ * Sealing {@link Shard}s once a topology change obsoletes them.
+ *
+ * Happy Path Sequence:
+ * 1. Bring every participant up to (at least) this seal's epoch and fence allocation, so that everyone
+ * stops writing to the old shards; then wait until all in-progress mutations have been done.
+ * 2. Same for tracked transfers
+ * 3. Wait until all participating replicas have reconciled the union of their witnessed ids
+ * 4. Mark the shard as fully sealed
+ * 5. Stop including the shard in mutation summaries for read requests.
+ *
+ * Unhappy Path Sequence (not yet implemented):
+ * 1. Wait until everyone who's up stops writing to the old shards and all in-progress mutations have been done
+ * 2. Same for tracked transfers
+ * 3. Wait until all *live* participating replicas have reconciled the union of their witnessed ids
+ * 4. Mark the shard as partially sealed, label it with the set of participants that were able to seal
+ * 5. When a down node comes up, have it talk to other nodes and learn about sealing, and what mutations/logs
+ * they have exchanged. Diff with what we have - in system tables and in the journal (though system table
+ * plus local replay should give us an up-to-date metadata state for the shard)
+ */
+public final class SealingCoordinator
+{
+ /*
+ * BOOTSTRAP sequence illustrated. Stages in which write placements
+ * remain unchanged, and only read placements change, are skipped
+ * here as irrelevant to sharding / sealing.
+ *
+ * There is a tracked keyspace with RF=3.
+ * A node n35 with token=35 is added in the middle of the ring.
+ *
+ * 1. Initial state at e0:
+ *
+ * (10,20] | {n20,n30,n40} | e0 | ACTIVE
+ * (20,30] | {n30,n40,n50} | e0 | ACTIVE
+ * (30,40] | {n40,n50,n60} | e0 | ACTIVE
+ *
+ * 2. PREPARE_JOIN at e1:
+ *
+ * (10,20] | {n20,n30,n40} | e0 | ACTIVE
+ * (20,30] | {n30,n40,n50} | e0 | ACTIVE
+ * (30,40] | {n40,n50,n60} | e0 | ACTIVE
+ * + (30,35] | {n40,n50,n60} | e1 | ACTIVE
+ * + (35,40] | {n40,n50,n60} | e1 | ACTIVE
+ *
+ * 3. START_JOIN at e2:
+ *
+ * (10,20] | {n20,n30,n40} | e0 | ACTIVE
+ * + (10,20] | {n20,n30,n35,n40} | e2 | ACTIVE
+ * (20,30] | {n30,n40,n50} | e0 | ACTIVE
+ * + (20,30] | {n30,n35,n40,n50} | e2 | ACTIVE
+ * (30,40] | {n40,n50,n60} | e0 | ACTIVE
+ * (30,35] | {n40,n50,n60} | e1 | ACTIVE
+ * + (30,35] | {n35,n40,n50,n60} | e2 | ACTIVE
+ * (35,40] | {n40,n50,n60} | e1 | ACTIVE
+ *
+ * 4. MID_JOIN at e3 [*]:
+ *
+ * S (10,20] | {n20,n30,n40} | e0 | SEALED
+ * (10,20] | {n20,n30,n35,n40} | e2 | ACTIVE
+ * S (20,30] | {n30,n40,n50} | e0 | SEALED
+ * (20,30] | {n30,n35,n40,n50} | e2 | ACTIVE
+ * S (30,40] | {n40,n50,n60} | e0 | SEALED
+ * S (30,35] | {n40,n50,n60} | e1 | SEALED
+ * (30,35] | {n35,n40,n50,n60} | e2 | ACTIVE
+ * (35,40] | {n40,n50,n60} | e1 | ACTIVE
+ *
+ * 5. FINISH_JOIN at e4:
+ *
+ * (10,20] | {n20,n30,n40} | e0 | SEALED
+ * S (10,20] | {n20,n30,n35,n40} | e2 | SEALED
+ * + (10,20] | {n20,n30,n35} | e4 | ACTIVE
+ * (20,30] | {n30,n40,n50} | e0 | SEALED
+ * S (20,30] | {n30,n35,n40,n50} | e2 | SEALED
+ * + (20,30] | {n30,n35,n40} | e4 | ACTIVE
+ * (30,40] | {n40,n50,n60} | e0 | SEALED
+ * (30,35] | {n40,n50,n60} | e1 | SEALED
+ * S (30,35] | {n35,n40,n50,n60} | e2 | SEALED
+ * + (30,35] | {n35,n40,n50} | e4 | ACTIVE
+ * (35,40] | {n40,n50,n60} | e1 | ACTIVE
+ *
+ * [*] At this stage, after sealing the shards, we stream the sstables.
+ * All the offsets belonging to these sealed shards get stripped from
+ * stats component of streamed sstables. Otherwise, the new node
+ * won't be able to compact them or mark them as reconciled/repaired -
+ * it is rightfully unaware of the sealed shards.
+ */
+
+ /**
+ * Seal the shards obsoleted by bootstrap before the new node starts streaming.
+ */
+ public static void sealShardsAtMidJoin(ClusterMetadata metadata, MovementMap movements)
+ {
+ seal(discoverShardsAtMidJoinOrReplace(metadata, movements));
+ }
+
+ /**
+ * Seal the intermediate shards that were created during START_JOIN.
+ * TODO (required): this MUST eventually run while the affected ranges are still locked - between the
+ * removeFromWrites placement change and the range unlock. FinishJoin/FinishReplace currently apply
+ * removeFromWrites and the unlock atomically in a single transformation, so there is no committed
+ * dropped-but-locked epoch to hook. Until then, this runs just after FINISH_JOIN/FINISH_REPLACE,
+ * when the ranges are already unlocked. Left to implement for the unhappy path (AY).
+ */
+ public static void sealShardsAtFinishJoin(ClusterMetadata metadata, PlacementDeltas finishDelta)
+ {
+ seal(discoverShardsAtFinishJoinOrReplace(metadata, finishDelta, null));
+ }
+
+ /*
+ * REPLACE sequence illustrated. Stages in which write placements
+ * remain unchanged, and only read placements change, are skipped
+ * here as irrelevant to sharding / sealing.
+ *
+ * There is a tracked keyspace with RF=3.
+ * Node n40 is dead, and is being replaced with $n40.
+ *
+ * 1. Initial state at e0:
+ *
+ * (10,20] | {n20,n30,n40} | e0 | ACTIVE
+ * (20,30] | {n30,n40,n50} | e0 | ACTIVE
+ * (30,40] | {n40,n50,n60} | e0 | ACTIVE
+ *
+ * 2. START_REPLACE at e1:
+ *
+ * (10,20] | {n20,n30,n40} | e0 | ACTIVE
+ * + (10,20] | {n20,n30,n40,$n40}| e1 | ACTIVE
+ * (20,30] | {n30,n40,n50} | e0 | ACTIVE
+ * + (20,30] | {n30,n40,$n40,n50}| e1 | ACTIVE
+ * (30,40] | {n40,n50,n60} | e0 | ACTIVE
+ * + (30,40] | {n40,$n40,n50,n60}| e1 | ACTIVE
+ *
+ * 3. MID_REPLACE at e2 [*]:
+ *
+ * S (10,20] | {n20,n30,n40} | e0 | SEALED
+ * (10,20] | {n20,n30,n40,$n40}| e1 | ACTIVE
+ * S (20,30] | {n30,n40,n50} | e0 | SEALED
+ * (20,30] | {n30,n40,$n40,n50}| e1 | ACTIVE
+ * S (30,40] | {n40,n50,n60} | e0 | SEALED
+ * (30,40] | {n40,$n40,n50,n60}| e1 | ACTIVE
+ *
+ * 4. FINISH_REPLACE at e3:
+ *
+ * (10,20] | {n20,n30,n40} | e0 | SEALED
+ * S (10,20] | {n20,n30,n40,$n40}| e1 | SEALED
+ * + (10,20] | {n20,n30,$n40} | e3 | ACTIVE
+ * (20,30] | {n30,n40,n50} | e0 | SEALED
+ * S (20,30] | {n30,n40,$n40,n50}| e1 | SEALED
+ * + (20,30] | {n30,$n40,n50} | e3 | ACTIVE
+ * (30,40] | {n40,n50,n60} | e0 | SEALED
+ * S (30,40] | {n40,$n40,n50,n60}| e1 | SEALED
+ * + (30,40] | {$n40,n50,n60} | e3 | ACTIVE
+ *
+ * [*] At this stage, after sealing the shards, we stream the sstables.
+ * All the offsets belonging to these sealed shards get stripped from
+ * stats component of streamed sstables. Otherwise, the new node
+ * won't be able to compact them or mark them as reconciled/repaired -
+ * it is rightfully unaware of the sealed shards.
+ */
+
+ /**
+ * Seal the shards obsoleted by host replacement before the replacement node starts streaming.
+ * The {@code deadNode} node cannot participate in the sealing.
+ */
+ public static void sealShardsAtMidReplace(ClusterMetadata metadata, MovementMap movements, @Nonnull NodeId deadNode)
+ {
+ seal(discoverShardsAtMidJoinOrReplace(metadata, movements), deadNode);
+ }
+
+ public static void sealShardsAtFinishReplace(ClusterMetadata metadata, PlacementDeltas finishDelta, @Nonnull NodeId deadNode)
+ {
+ seal(discoverShardsAtFinishJoinOrReplace(metadata, finishDelta, deadNode), deadNode);
+ }
+
+ /**
+ * Discover all shards that need to be sealed at MID_JOIN or MID_REPLACE.
+ */
+ private static Set<ShardMetadata> discoverShardsAtMidJoinOrReplace(ClusterMetadata metadata, MovementMap movements)
+ {
+ List<Future<Set<ShardMetadata>>> futures = new ArrayList<>();
+
+ for (KeyspaceMetadata ksm : metadata.schema.getKeyspaces())
+ {
+ if (!ksm.params.replicationType.isTracked())
+ continue;
+
+ EndpointsByReplica acquired = movements.get(ksm.params.replication);
+ if (acquired.isEmpty()) // when the keyspace is not replicated to this DC
+ continue;
+
+ DataPlacement placement = metadata.placements.get(ksm.params.replication);
+
+ for (Map.Entry<Replica, EndpointsForRange> entry : acquired.entrySet())
+ {
+ Range<Token> range = entry.getKey().range();
+ Set<InetAddressAndPort> endpoints = entry.getValue().endpoints();
+ // cutoff: the epoch this node was added to the write placement for the range;
+ // shards over the range with sinceEpoch < cutoff are obsoleted by this bootstrap or replace.
+ long beforeEpoch = placement.writes.forRange(range).lastModified().getEpoch();
+ futures.add(FetchShards.fetch(ksm.name, beforeEpoch, range, endpoints));
+ }
+ }
+
+ Set<ShardMetadata> shards = new HashSet<>();
+ try
+ {
+ for (Set<ShardMetadata> fetched : FutureCombiner.allOf(futures).get(DatabaseDescriptor.getRpcTimeout(MILLISECONDS), MILLISECONDS))
+ shards.addAll(fetched);
+ }
+ catch (InterruptedException | ExecutionException | TimeoutException e)
+ {
+ throw new RuntimeException("Failed to fetch shards to seal", e);
+ }
+ return shards;
+ }
+
+ /**
+ * Discover all shards that need to be sealed at FINISH_JOIN or FINISH_REPLACE.
+ */
+ private static Set<ShardMetadata> discoverShardsAtFinishJoinOrReplace(ClusterMetadata metadata, PlacementDeltas finishDelta, @Nullable NodeId deadNode)
+ {
+ long finishEpoch = metadata.epoch.getEpoch();
+ Set<ShardMetadata> shards = new HashSet<>();
+ for (KeyspaceMetadata ksm : metadata.schema.getKeyspaces())
+ {
+ if (!ksm.params.replicationType.isTracked())
+ continue;
+
+ DataPlacement placement = metadata.placements.get(ksm.params.replication);
+ finishDelta.get(ksm.params.replication).writes.removals.flattenValues().forEach(removed -> {
+ Set<Integer> nodeIDs = new HashSet<>();
+ for (InetAddressAndPort ep : placement.writes.forRange(removed.range()).endpoints())
+ nodeIDs.add(metadata.directory.peerId(ep).id());
+ nodeIDs.add(deadNode != null ? deadNode.id() : metadata.directory.peerId(removed.endpoint()).id());
+ Participants participants = new Participants(nodeIDs);
+
+ MutationTrackingService.instance().forEachShardInKeyspace(ksm.name, shard -> {
+ if (shard.sinceEpoch < finishEpoch
+ && shard.range.equals(removed.range())
+ && shard.participants.equals(participants)
+ && !shard.isSealed())
+ shards.add(new ShardMetadata(shard.keyspace, shard.sinceEpoch, shard.range, shard.participants));
+ });
+ });
+ }
+ return shards;
+ }
+
+ /*
+ * UNBOOTSTRAP and REMOVENODE sequences illustrated (same sequence as far as sharding is concerned).
+ * Stages in which write placements remain unchanged, and only read placements change, are skipped
+ * here as irrelevant to sharding / sealing.
+ *
+ * There is a tracked keyspace with RF=3.
+ * A node n40 with token=40 is removed in the middle of the ring.
+ *
+ * 1. Initial state at e0:
+ *
+ * (10,20] | {n20,n30,n40} | e0 | ACTIVE
+ * (20,30] | {n30,n40,n50} | e0 | ACTIVE
+ * (30,40] | {n40,n50,n60} | e0 | ACTIVE
+ * (40,50] | {n50,n60,n70} | e0 | ACTIVE
+ *
+ * 2. START_LEAVE at e1:
+ *
+ * (10,20] | {n20,n30,n40} | e0 | ACTIVE
+ * + (10,20] | {n20,n30,n40,n50} | e1 | ACTIVE
+ * (20,30] | {n30,n40,n50} | e0 | ACTIVE
+ * + (20,30] | {n30,n40,n50,n60} | e1 | ACTIVE
+ * (30,40] | {n40,n50,n60} | e0 | ACTIVE
+ * + (30,40] | {n40,n50,n60,n70} | e1 | ACTIVE
+ * (40,50] | {n50,n60,n70} | e0 | ACTIVE
+ *
+ * 3. MID_LEAVE at e2 [*]:
+ *
+ * S (10,20] | {n20,n30,n40} | e0 | SEALED
+ * (10,20] | {n20,n30,n40,n50} | e1 | ACTIVE
+ * S (20,30] | {n30,n40,n50} | e0 | SEALED
+ * (20,30] | {n30,n40,n50,n60} | e1 | ACTIVE
+ * S (30,40] | {n40,n50,n60} | e0 | SEALED
+ * (30,40] | {n40,n50,n60,n70} | e1 | ACTIVE
+ * (40,50] | {n50,n60,n70} | e0 | ACTIVE
+ *
+ * 4. FINISH_LEAVE at e3:
+ *
+ * (10,20] | {n20,n30,n40} | e0 | SEALED
+ * S (10,20] | {n20,n30,n40,n50} | e1 | SEALED
+ * + (10,20] | {n20,n30,n50} | e3 | ACTIVE
+ * (20,30] | {n30,n40,n50} | e0 | SEALED
+ * S (20,30] | {n30,n40,n50,n60} | e1 | SEALED
+ * + (20,30] | {n30,n50,n60} | e3 | ACTIVE
+ * (30,40] | {n40,n50,n60} | e0 | SEALED
+ * S (30,40] | {n40,n50,n60,n70} | e1 | SEALED
+ * S (40,50] | {n50,n60,n70} | e0 | SEALED
+ * + (30,50] | {n50,n60,n70} | e3 | ACTIVE
+ *
+ * [*] At this stage, after sealing the shards, we stream the sstables.
+ * All the offsets belonging to these sealed shards get stripped from
+ * stats component of streamed sstables. Otherwise, the new node
+ * won't be able to compact them or mark them as reconciled/repaired -
+ * it is rightfully unaware of the sealed shards.
+ */
+
+ /**
+ * Seal the shards obsoleted by START_LEAVE before streaming, for both UNBOOTSTRAP and REMOVENODE.
+ */
+ public static void sealShardsAtMidLeave(ClusterMetadata metadata, PlacementDeltas startDelta, NodeId leavingOrDeparted, LeaveStreams.Kind kind)
+ {
+ InetAddressAndPort leavingOrDepartedEndpoint = metadata.directory.endpoint(leavingOrDeparted);
+
+ List<Future<Set<ShardMetadata>>> futures = new ArrayList<>();
+ for (KeyspaceMetadata ksm : metadata.schema.getKeyspaces())
+ {
+ if (!ksm.params.replicationType.isTracked())
+ continue;
+
+ DataPlacement placement = metadata.placements.get(ksm.params.replication);
+ startDelta.get(ksm.params.replication).writes.additions.flattenValues().forEach(added -> {
+ Range<Token> range = added.range();
+ long beforeEpoch = placement.writes.forRange(range).lastModified().getEpoch();
+ Set<InetAddressAndPort> survivors = new HashSet<>(placement.writes.forRange(range).endpoints());
+ survivors.remove(leavingOrDepartedEndpoint);
+ futures.add(FetchShards.fetch(ksm.name, beforeEpoch, range, survivors));
+ });
+ }
+
+ Set<ShardMetadata> shards = new HashSet<>();
+ try
+ {
+ for (Set<ShardMetadata> fetched : FutureCombiner.allOf(futures).get(DatabaseDescriptor.getRpcTimeout(MILLISECONDS), MILLISECONDS))
+ for (ShardMetadata shard : fetched)
+ if (shard.participants.contains(leavingOrDeparted.id()))
+ shards.add(shard);
+ }
+ catch (InterruptedException | ExecutionException | TimeoutException e)
+ {
+ throw new RuntimeException("Failed to fetch shards to seal for leave", e);
+ }
+
+ seal(shards, kind == LeaveStreams.Kind.UNBOOTSTRAP ? null : leavingOrDeparted);
+ }
+
+ /**
+ * Seal the shards that need to be sealed at LEAVE_FINISH for both UNBOOTSTRAP and REMOVENODE.
+ */
+ public static void sealShardsAtFinishLeave(ClusterMetadata postFinish, PlacementDeltas finishDelta, NodeId leavingOrDeparted, LeaveStreams.Kind kind)
+ {
+ long finishEpoch = postFinish.epoch.getEpoch();
+ InetAddressAndPort leavingOrDepartedEndpoint = postFinish.directory.endpoint(leavingOrDeparted);
+
+ List<Future<Set<ShardMetadata>>> futures = new ArrayList<>();
+ for (KeyspaceMetadata ksm : postFinish.schema.getKeyspaces())
+ {
+ if (!ksm.params.replicationType.isTracked())
+ continue;
+
+ DataPlacement placement = postFinish.placements.get(ksm.params.replication);
+ RangesAtEndpoint removals = finishDelta.get(ksm.params.replication).writes.removals.get(leavingOrDepartedEndpoint);
+ for (Replica removedReplica : removals)
+ {
+ // FINISH_LEAVE merges the leavingOrDeparted node's (pre-merge) range with an adjacent one. Query the *merged*
+ // range (not the pre-merge range) so discovery covers both the leavingOrDeparted node's own obsoleted
+ // generations and the merge-half - the adjacent pre-merge range folded into the merged range, whose
+ // shard never listed the leavingOrDeparted node. matchRange() yields the containing merged group + its
+ // current (survivor) write endpoints.
+ VersionedEndpoints.ForRange group = placement.writes.matchRange(removedReplica.range());
+ Range<Token> mergedRange = group.range();
+ Set<InetAddressAndPort> survivors = new HashSet<>(group.endpoints());
+ survivors.remove(leavingOrDepartedEndpoint);
+ futures.add(FetchShards.fetch(ksm.name, finishEpoch, mergedRange, survivors));
+ }
+ }
+
+ Set<ShardMetadata> shards = new HashSet<>();
+ try
+ {
+ for (Set<ShardMetadata> fetched : FutureCombiner.allOf(futures).get(DatabaseDescriptor.getRpcTimeout(MILLISECONDS), MILLISECONDS))
+ shards.addAll(fetched);
+ }
+ catch (InterruptedException | ExecutionException | TimeoutException e)
+ {
+ throw new RuntimeException("Failed to fetch shards to seal for leave finish", e);
+ }
+
+ seal(shards, kind == LeaveStreams.Kind.UNBOOTSTRAP ? null : leavingOrDeparted);
+ }
+
+ /*
+ * Sealing logic.
+ */
+
+ private static void seal(Set<ShardMetadata> shards)
+ {
+ seal(shards, null);
+ }
+
+ private static void seal(Set<ShardMetadata> shards, @Nullable NodeId withoutNode)
+ {
+ initiate(shards, withoutNode); // ACTIVE -> SEALING for each Shard
+ drain(shards, withoutNode); // drain in-flight local writes
+ reconcile(shards, withoutNode); // wait for logs to reconcile
+ complete(shards, withoutNode); // SEALING-> SEALED for each Shard + journal flush
+ }
+
+ private static void initiate(Set<ShardMetadata> shards, @Nullable NodeId withoutNode)
+ {
+ List<Future<Void>> futures = new ArrayList<>(shards.size());
+ for (ShardMetadata shard : shards)
+ futures.add(initiate(shard, withoutNode));
+ try
+ {
+ FutureCombiner.allOf(futures).get(DatabaseDescriptor.getRpcTimeout(MILLISECONDS), MILLISECONDS);
+ }
+ catch (InterruptedException | ExecutionException | TimeoutException e)
+ {
+ throw new RuntimeException("Failed to initiate sealing", e);
+ }
+ }
+
+ /**
+ * Fence allocation on every participant of the obsoleted shard (ACTIVE -> SEALING) by marking the shard
+ * SEALING on each.
+ * TODO (expected): this assumes every participant has already enacted the topology change that obsoleted the shard (so
+ * it no longer allocates new ids on it). The ProgressBarrier gating bootstrap()/MID_JOIN only waits for
+ * EACH_QUORUM of the affected replicas, NOT all live replicas, so a live-but-lagging participant could still
+ * be on the old shard. Bringing each participant up to the seal epoch must be done per-replica (not
+ * per-shard) and will be added separately. TLDR - add a step to guaranteed up-to-speed rest of replicas epochs.
+ */
+ private static AsyncPromise<Void> initiate(ShardMetadata shard, @Nullable NodeId withoutNode)
+ {
+ return InitSealing.initiate(shard.keyspace, shard.sinceEpoch, shard.range, toEndpoints(shard.participants, withoutNode));
+ }
+
+ private static void drain(Set<ShardMetadata> shards, @Nullable NodeId withoutNode)
+ {
+ long deadlineNanos = nanoTime() + DatabaseDescriptor.getRpcTimeout(NANOSECONDS);
+ Set<ShardMetadata> pending = new HashSet<>(shards);
+ while (true)
+ {
+ // non-blocking poll: each participant reports its drain status immediately; retry until all drained
+ pending.removeIf(shard -> drain(shard, withoutNode));
+ if (pending.isEmpty())
+ return;
+ if (nanoTime() >= deadlineNanos)
+ throw new RuntimeException("Timed out draining shards; still pending: " + pending);
+ LockSupport.parkNanos(SECONDS.toNanos(1));
+ }
+ }
+
+ /**
+ * Poll every participant of the SEALING shard once for drain status; true iff all report that no mutation
+ * id allocated before the SEALING fence is still applying locally.
+ */
+ private static boolean drain(ShardMetadata shard, @Nullable NodeId withoutNode)
+ {
+ try
+ {
+ return Drain.poll(shard.keyspace, shard.sinceEpoch, shard.range, toEndpoints(shard.participants, withoutNode))
+ .get(DatabaseDescriptor.getRpcTimeout(MILLISECONDS), MILLISECONDS);
+ }
+ catch (InterruptedException | ExecutionException | TimeoutException e)
+ {
+ throw new RuntimeException(format("Failed to poll drain for shard %s", shard), e);
+ }
+ }
+
+ /**
+ * For each of the shards, capture the witnessed union then poll each participant until it has
+ * caught up and witnessed the entire offset union itself.
+ */
+ private static void reconcile(Set<ShardMetadata> shards, @Nullable NodeId withoutNode)
+ {
+ for (ShardMetadata shard : shards)
+ reconcile(shard, withoutNode);
+ }
+
+ private static void reconcile(ShardMetadata shard, @Nullable NodeId withoutNode)
Review Comment:
It looks roughly similar but it is quite different in actuality, I don't see a clean way to reuse it.
(I can go into details why if you want, but TL;DR is that there are about five major ways they differ significantly)
Leaving it to you to resolve this conversation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CEP-45: Topology Change Support for Mutation Tracking [cassandra]
Posted by "iamaleksey (via GitHub)" <gi...@apache.org>.
iamaleksey commented on PR #4910:
URL: https://github.com/apache/cassandra/pull/4910#issuecomment-5779670960
Rebased on top of most recent `cep-45-mutation-tracking` branch, addressing remaining unaddressed feedback.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CEP-45: Topology Change Support for Mutation Tracking [cassandra]
Posted by "iamaleksey (via GitHub)" <gi...@apache.org>.
iamaleksey commented on code in PR #4910:
URL: https://github.com/apache/cassandra/pull/4910#discussion_r4134935755
##########
src/java/org/apache/cassandra/replication/MutationTrackingService.java:
##########
@@ -841,6 +844,44 @@ private KeyspaceShards getOrCreateShards(String keyspace)
return keyspaceShards.computeIfAbsent(keyspace, ignore -> KeyspaceShards.make(ksm, csm, this::nextLogId, this::onNewLog));
}
+ /**
+ * Find an existing shard matching the response's (epoch, range), or create a new one from the response metadata.
+ * TODO (expected): validate if I need to persist the newly created shard, if it was just created here
+ * TODO (expected): validate if this should be called with a shard lock everywhere it's called
+ * TODO (expected): log2Shard map updates even on failed CAS. Need to clean up there(?); Register callbacks?
+ */
+ @Nonnull
+ private Shard getOrCreateShard(
+ String keyspace, long sinceEpoch, Range<Token> range, Participants participants)
+ {
+ int localNodeId = ClusterMetadata.current().myNodeId().id();
+
+ if (!participants.contains(localNodeId))
+ throw new IllegalStateException("A shard looked up that this node doesn't participate in");
+
+ // unlikely, but possible to race here, hence the CAS loop
+ while (true)
+ {
+ KeyspaceShards current = getOrCreateShards(keyspace);
+ Shard shard = current.get(range, sinceEpoch);
+ if (shard != null)
+ return shard;
+ shard = new Shard(localNodeId, keyspace, sinceEpoch, range, participants, this::nextLogId, this::onNewLog);
Review Comment:
Yep. There was a TODO but of course that's insufficient. Addressed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CEP-45: Topology Change Support for Mutation Tracking [cassandra]
Posted by "iamaleksey (via GitHub)" <gi...@apache.org>.
iamaleksey commented on code in PR #4910:
URL: https://github.com/apache/cassandra/pull/4910#discussion_r4134610960
##########
src/java/org/apache/cassandra/replication/SealingCoordinator.java:
##########
@@ -0,0 +1,1401 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.cassandra.replication;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.TimeoutException;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.concurrent.locks.LockSupport;
+
+import javax.annotation.Nonnull;
+import javax.annotation.Nullable;
+
+import com.google.common.base.Preconditions;
+
+import org.apache.cassandra.config.DatabaseDescriptor;
+import org.apache.cassandra.db.ColumnFamilyStore;
+import org.apache.cassandra.db.Keyspace;
+import org.apache.cassandra.db.TypeSizes;
+import org.apache.cassandra.dht.AbstractBounds;
+import org.apache.cassandra.dht.IPartitioner;
+import org.apache.cassandra.dht.Range;
+import org.apache.cassandra.dht.Token;
+import org.apache.cassandra.exceptions.RequestFailure;
+import org.apache.cassandra.io.UnversionedSerializer;
+import org.apache.cassandra.io.util.DataInputPlus;
+import org.apache.cassandra.io.util.DataOutputPlus;
+import org.apache.cassandra.locator.EndpointsByReplica;
+import org.apache.cassandra.locator.EndpointsForRange;
+import org.apache.cassandra.locator.InetAddressAndPort;
+import org.apache.cassandra.locator.RangesAtEndpoint;
+import org.apache.cassandra.locator.Replica;
+import org.apache.cassandra.net.IVerbHandler;
+import org.apache.cassandra.net.Message;
+import org.apache.cassandra.net.MessagingService;
+import org.apache.cassandra.net.RequestCallback;
+import org.apache.cassandra.net.Verb;
+import org.apache.cassandra.schema.KeyspaceMetadata;
+import org.apache.cassandra.tcm.ClusterMetadata;
+import org.apache.cassandra.tcm.membership.NodeId;
+import org.apache.cassandra.tcm.ownership.DataPlacement;
+import org.apache.cassandra.tcm.ownership.MovementMap;
+import org.apache.cassandra.tcm.ownership.PlacementDeltas;
+import org.apache.cassandra.tcm.ownership.VersionedEndpoints;
+import org.apache.cassandra.tcm.sequences.LeaveStreams;
+import org.apache.cassandra.utils.CollectionSerializers;
+import org.apache.cassandra.utils.FBUtilities;
+import org.apache.cassandra.utils.concurrent.AsyncPromise;
+import org.apache.cassandra.utils.concurrent.Future;
+import org.apache.cassandra.utils.concurrent.FutureCombiner;
+
+import static java.lang.String.format;
+import static java.util.concurrent.TimeUnit.MILLISECONDS;
+import static java.util.concurrent.TimeUnit.NANOSECONDS;
+import static java.util.concurrent.TimeUnit.SECONDS;
+import static org.apache.cassandra.utils.Clock.Global.nanoTime;
+
+/*
+ * Sealing {@link Shard}s once a topology change obsoletes them.
+ *
+ * Happy Path Sequence:
+ * 1. Bring every participant up to (at least) this seal's epoch and fence allocation, so that everyone
+ * stops writing to the old shards; then wait until all in-progress mutations have been done.
+ * 2. Same for tracked transfers
+ * 3. Wait until all participating replicas have reconciled the union of their witnessed ids
+ * 4. Mark the shard as fully sealed
+ * 5. Stop including the shard in mutation summaries for read requests.
+ *
+ * Unhappy Path Sequence (not yet implemented):
+ * 1. Wait until everyone who's up stops writing to the old shards and all in-progress mutations have been done
+ * 2. Same for tracked transfers
+ * 3. Wait until all *live* participating replicas have reconciled the union of their witnessed ids
+ * 4. Mark the shard as partially sealed, label it with the set of participants that were able to seal
+ * 5. When a down node comes up, have it talk to other nodes and learn about sealing, and what mutations/logs
+ * they have exchanged. Diff with what we have - in system tables and in the journal (though system table
+ * plus local replay should give us an up-to-date metadata state for the shard)
+ */
+public final class SealingCoordinator
+{
+ /*
+ * BOOTSTRAP sequence illustrated. Stages in which write placements
+ * remain unchanged, and only read placements change, are skipped
+ * here as irrelevant to sharding / sealing.
+ *
+ * There is a tracked keyspace with RF=3.
+ * A node n35 with token=35 is added in the middle of the ring.
+ *
+ * 1. Initial state at e0:
+ *
+ * (10,20] | {n20,n30,n40} | e0 | ACTIVE
+ * (20,30] | {n30,n40,n50} | e0 | ACTIVE
+ * (30,40] | {n40,n50,n60} | e0 | ACTIVE
+ *
+ * 2. PREPARE_JOIN at e1:
+ *
+ * (10,20] | {n20,n30,n40} | e0 | ACTIVE
+ * (20,30] | {n30,n40,n50} | e0 | ACTIVE
+ * (30,40] | {n40,n50,n60} | e0 | ACTIVE
+ * + (30,35] | {n40,n50,n60} | e1 | ACTIVE
+ * + (35,40] | {n40,n50,n60} | e1 | ACTIVE
+ *
+ * 3. START_JOIN at e2:
+ *
+ * (10,20] | {n20,n30,n40} | e0 | ACTIVE
+ * + (10,20] | {n20,n30,n35,n40} | e2 | ACTIVE
+ * (20,30] | {n30,n40,n50} | e0 | ACTIVE
+ * + (20,30] | {n30,n35,n40,n50} | e2 | ACTIVE
+ * (30,40] | {n40,n50,n60} | e0 | ACTIVE
+ * (30,35] | {n40,n50,n60} | e1 | ACTIVE
+ * + (30,35] | {n35,n40,n50,n60} | e2 | ACTIVE
+ * (35,40] | {n40,n50,n60} | e1 | ACTIVE
+ *
+ * 4. MID_JOIN at e3 [*]:
+ *
+ * S (10,20] | {n20,n30,n40} | e0 | SEALED
+ * (10,20] | {n20,n30,n35,n40} | e2 | ACTIVE
+ * S (20,30] | {n30,n40,n50} | e0 | SEALED
+ * (20,30] | {n30,n35,n40,n50} | e2 | ACTIVE
+ * S (30,40] | {n40,n50,n60} | e0 | SEALED
+ * S (30,35] | {n40,n50,n60} | e1 | SEALED
+ * (30,35] | {n35,n40,n50,n60} | e2 | ACTIVE
+ * (35,40] | {n40,n50,n60} | e1 | ACTIVE
+ *
+ * 5. FINISH_JOIN at e4:
+ *
+ * (10,20] | {n20,n30,n40} | e0 | SEALED
+ * S (10,20] | {n20,n30,n35,n40} | e2 | SEALED
+ * + (10,20] | {n20,n30,n35} | e4 | ACTIVE
+ * (20,30] | {n30,n40,n50} | e0 | SEALED
+ * S (20,30] | {n30,n35,n40,n50} | e2 | SEALED
+ * + (20,30] | {n30,n35,n40} | e4 | ACTIVE
+ * (30,40] | {n40,n50,n60} | e0 | SEALED
+ * (30,35] | {n40,n50,n60} | e1 | SEALED
+ * S (30,35] | {n35,n40,n50,n60} | e2 | SEALED
+ * + (30,35] | {n35,n40,n50} | e4 | ACTIVE
+ * (35,40] | {n40,n50,n60} | e1 | ACTIVE
+ *
+ * [*] At this stage, after sealing the shards, we stream the sstables.
+ * All the offsets belonging to these sealed shards get stripped from
+ * stats component of streamed sstables. Otherwise, the new node
+ * won't be able to compact them or mark them as reconciled/repaired -
+ * it is rightfully unaware of the sealed shards.
+ */
+
+ /**
+ * Seal the shards obsoleted by bootstrap before the new node starts streaming.
+ */
+ public static void sealShardsAtMidJoin(ClusterMetadata metadata, MovementMap movements)
+ {
+ seal(discoverShardsAtMidJoinOrReplace(metadata, movements));
+ }
+
+ /**
+ * Seal the intermediate shards that were created during START_JOIN.
+ * TODO (required): this MUST eventually run while the affected ranges are still locked - between the
+ * removeFromWrites placement change and the range unlock. FinishJoin/FinishReplace currently apply
+ * removeFromWrites and the unlock atomically in a single transformation, so there is no committed
+ * dropped-but-locked epoch to hook. Until then, this runs just after FINISH_JOIN/FINISH_REPLACE,
+ * when the ranges are already unlocked. Left to implement for the unhappy path (AY).
+ */
+ public static void sealShardsAtFinishJoin(ClusterMetadata metadata, PlacementDeltas finishDelta)
+ {
+ seal(discoverShardsAtFinishJoinOrReplace(metadata, finishDelta, null));
+ }
+
+ /*
+ * REPLACE sequence illustrated. Stages in which write placements
+ * remain unchanged, and only read placements change, are skipped
+ * here as irrelevant to sharding / sealing.
+ *
+ * There is a tracked keyspace with RF=3.
+ * Node n40 is dead, and is being replaced with $n40.
+ *
+ * 1. Initial state at e0:
+ *
+ * (10,20] | {n20,n30,n40} | e0 | ACTIVE
+ * (20,30] | {n30,n40,n50} | e0 | ACTIVE
+ * (30,40] | {n40,n50,n60} | e0 | ACTIVE
+ *
+ * 2. START_REPLACE at e1:
+ *
+ * (10,20] | {n20,n30,n40} | e0 | ACTIVE
+ * + (10,20] | {n20,n30,n40,$n40}| e1 | ACTIVE
+ * (20,30] | {n30,n40,n50} | e0 | ACTIVE
+ * + (20,30] | {n30,n40,$n40,n50}| e1 | ACTIVE
+ * (30,40] | {n40,n50,n60} | e0 | ACTIVE
+ * + (30,40] | {n40,$n40,n50,n60}| e1 | ACTIVE
+ *
+ * 3. MID_REPLACE at e2 [*]:
+ *
+ * S (10,20] | {n20,n30,n40} | e0 | SEALED
+ * (10,20] | {n20,n30,n40,$n40}| e1 | ACTIVE
+ * S (20,30] | {n30,n40,n50} | e0 | SEALED
+ * (20,30] | {n30,n40,$n40,n50}| e1 | ACTIVE
+ * S (30,40] | {n40,n50,n60} | e0 | SEALED
+ * (30,40] | {n40,$n40,n50,n60}| e1 | ACTIVE
+ *
+ * 4. FINISH_REPLACE at e3:
+ *
+ * (10,20] | {n20,n30,n40} | e0 | SEALED
+ * S (10,20] | {n20,n30,n40,$n40}| e1 | SEALED
+ * + (10,20] | {n20,n30,$n40} | e3 | ACTIVE
+ * (20,30] | {n30,n40,n50} | e0 | SEALED
+ * S (20,30] | {n30,n40,$n40,n50}| e1 | SEALED
+ * + (20,30] | {n30,$n40,n50} | e3 | ACTIVE
+ * (30,40] | {n40,n50,n60} | e0 | SEALED
+ * S (30,40] | {n40,$n40,n50,n60}| e1 | SEALED
+ * + (30,40] | {$n40,n50,n60} | e3 | ACTIVE
+ *
+ * [*] At this stage, after sealing the shards, we stream the sstables.
+ * All the offsets belonging to these sealed shards get stripped from
+ * stats component of streamed sstables. Otherwise, the new node
+ * won't be able to compact them or mark them as reconciled/repaired -
+ * it is rightfully unaware of the sealed shards.
+ */
+
+ /**
+ * Seal the shards obsoleted by host replacement before the replacement node starts streaming.
+ * The {@code deadNode} node cannot participate in the sealing.
+ */
+ public static void sealShardsAtMidReplace(ClusterMetadata metadata, MovementMap movements, @Nonnull NodeId deadNode)
+ {
+ seal(discoverShardsAtMidJoinOrReplace(metadata, movements), deadNode);
+ }
+
+ public static void sealShardsAtFinishReplace(ClusterMetadata metadata, PlacementDeltas finishDelta, @Nonnull NodeId deadNode)
+ {
+ seal(discoverShardsAtFinishJoinOrReplace(metadata, finishDelta, deadNode), deadNode);
+ }
+
+ /**
+ * Discover all shards that need to be sealed at MID_JOIN or MID_REPLACE.
+ */
+ private static Set<ShardMetadata> discoverShardsAtMidJoinOrReplace(ClusterMetadata metadata, MovementMap movements)
+ {
+ List<Future<Set<ShardMetadata>>> futures = new ArrayList<>();
+
+ for (KeyspaceMetadata ksm : metadata.schema.getKeyspaces())
+ {
+ if (!ksm.params.replicationType.isTracked())
+ continue;
+
+ EndpointsByReplica acquired = movements.get(ksm.params.replication);
+ if (acquired.isEmpty()) // when the keyspace is not replicated to this DC
+ continue;
+
+ DataPlacement placement = metadata.placements.get(ksm.params.replication);
+
+ for (Map.Entry<Replica, EndpointsForRange> entry : acquired.entrySet())
+ {
+ Range<Token> range = entry.getKey().range();
+ Set<InetAddressAndPort> endpoints = entry.getValue().endpoints();
+ // cutoff: the epoch this node was added to the write placement for the range;
+ // shards over the range with sinceEpoch < cutoff are obsoleted by this bootstrap or replace.
+ long beforeEpoch = placement.writes.forRange(range).lastModified().getEpoch();
+ futures.add(FetchShards.fetch(ksm.name, beforeEpoch, range, endpoints));
+ }
+ }
+
+ Set<ShardMetadata> shards = new HashSet<>();
+ try
+ {
+ for (Set<ShardMetadata> fetched : FutureCombiner.allOf(futures).get(DatabaseDescriptor.getRpcTimeout(MILLISECONDS), MILLISECONDS))
+ shards.addAll(fetched);
+ }
+ catch (InterruptedException | ExecutionException | TimeoutException e)
+ {
+ throw new RuntimeException("Failed to fetch shards to seal", e);
+ }
+ return shards;
+ }
+
+ /**
+ * Discover all shards that need to be sealed at FINISH_JOIN or FINISH_REPLACE.
+ */
+ private static Set<ShardMetadata> discoverShardsAtFinishJoinOrReplace(ClusterMetadata metadata, PlacementDeltas finishDelta, @Nullable NodeId deadNode)
+ {
+ long finishEpoch = metadata.epoch.getEpoch();
+ Set<ShardMetadata> shards = new HashSet<>();
+ for (KeyspaceMetadata ksm : metadata.schema.getKeyspaces())
+ {
+ if (!ksm.params.replicationType.isTracked())
+ continue;
+
+ DataPlacement placement = metadata.placements.get(ksm.params.replication);
+ finishDelta.get(ksm.params.replication).writes.removals.flattenValues().forEach(removed -> {
+ Set<Integer> nodeIDs = new HashSet<>();
+ for (InetAddressAndPort ep : placement.writes.forRange(removed.range()).endpoints())
+ nodeIDs.add(metadata.directory.peerId(ep).id());
+ nodeIDs.add(deadNode != null ? deadNode.id() : metadata.directory.peerId(removed.endpoint()).id());
+ Participants participants = new Participants(nodeIDs);
+
+ MutationTrackingService.instance().forEachShardInKeyspace(ksm.name, shard -> {
+ if (shard.sinceEpoch < finishEpoch
+ && shard.range.equals(removed.range())
Review Comment:
Here `equals()` is strictly correct. We are going after a very precise set of shards. And `writes.forRange()` would NPE hard if we are wrong here, before we get to this point (but it is not wrong).
(Leaving it to you to mark this conversation as resolved, in case you have further questions)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CEP-45: Topology Change Support for Mutation Tracking [cassandra]
Posted by "iamaleksey (via GitHub)" <gi...@apache.org>.
iamaleksey commented on code in PR #4910:
URL: https://github.com/apache/cassandra/pull/4910#discussion_r4135078475
##########
src/java/org/apache/cassandra/replication/MutationTrackingService.java:
##########
@@ -950,7 +988,6 @@ private synchronized void onNewClusterMetadata(@Nullable ClusterMetadata prev, C
}
catch (Throwable t)
{
- log2ShardMap = originalLog2ShardMap;
Review Comment:
Yes. Perhaps. This code is a little funky (`onNewClusterMetadata()` and `applyUpdatedMetadata()` and arguably needs some further massaging / rewrite. In the meantime, I've reverted this, and added a different TODO.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CEP-45: Topology Change Support for Mutation Tracking [cassandra]
Posted by "bdeggleston (via GitHub)" <gi...@apache.org>.
bdeggleston commented on code in PR #4910:
URL: https://github.com/apache/cassandra/pull/4910#discussion_r3539483348
##########
src/java/org/apache/cassandra/replication/MutationTrackingService.java:
##########
@@ -483,52 +398,105 @@ public void updateReplicatedOffsets(String keyspace, Range<Token> range, List<?
}
}
- public void recordFullyReconciledOffsets(ReconciledLogSnapshot reconciledSnapshot)
+ public boolean startWriting(Mutation mutation)
{
+ Preconditions.checkArgument(!mutation.id().isNone());
+ // resolve the shard first - might trigger a blocking network call
+ Shard shard = getOrCreateShardForMutation(mutation);
shardLock.readLock().lock();
try
{
- reconciledSnapshot.forEach((keyspace, keyspaceOffsets) -> {
- KeyspaceShards ksShards = getOrCreateShards(keyspace);
- if (ksShards != null)
- ksShards.recordFullyReconciledOffsets(keyspaceOffsets);
- });
+ return shard.startWriting(mutation);
}
finally
{
shardLock.readLock().unlock();
}
}
- public boolean startWriting(Mutation mutation)
+ public void finishWriting(Mutation mutation)
{
+ Preconditions.checkArgument(!mutation.id().isNone());
shardLock.readLock().lock();
try
{
- Preconditions.checkArgument(!mutation.id().isNone());
- return getOrCreateShards(mutation.getKeyspaceName()).startWriting(mutation);
+ getShard(mutation.id().asLogId()).finishWriting(mutation);
+ incomingMutations.invokeListeners(mutation.id());
}
finally
{
shardLock.readLock().unlock();
}
}
- public void finishWriting(Mutation mutation)
+ /**
+ * Must be called exactly once per {@code nextId()} invocation.
+ */
+ public void completeLocalWrite(MutationId id)
Review Comment:
it's a little awkward having to remember remember to call this for each locally applied mutation, and any mistake is going to basically hang sealing which it waits for a decrement that never comes. Should we apply all local reads in a try with resources block or something?
##########
src/java/org/apache/cassandra/replication/MutationTrackingService.java:
##########
@@ -483,52 +398,105 @@ public void updateReplicatedOffsets(String keyspace, Range<Token> range, List<?
}
}
- public void recordFullyReconciledOffsets(ReconciledLogSnapshot reconciledSnapshot)
+ public boolean startWriting(Mutation mutation)
{
+ Preconditions.checkArgument(!mutation.id().isNone());
+ // resolve the shard first - might trigger a blocking network call
+ Shard shard = getOrCreateShardForMutation(mutation);
shardLock.readLock().lock();
try
{
- reconciledSnapshot.forEach((keyspace, keyspaceOffsets) -> {
- KeyspaceShards ksShards = getOrCreateShards(keyspace);
- if (ksShards != null)
- ksShards.recordFullyReconciledOffsets(keyspaceOffsets);
- });
+ return shard.startWriting(mutation);
}
finally
{
shardLock.readLock().unlock();
}
}
- public boolean startWriting(Mutation mutation)
+ public void finishWriting(Mutation mutation)
{
+ Preconditions.checkArgument(!mutation.id().isNone());
shardLock.readLock().lock();
try
{
- Preconditions.checkArgument(!mutation.id().isNone());
- return getOrCreateShards(mutation.getKeyspaceName()).startWriting(mutation);
+ getShard(mutation.id().asLogId()).finishWriting(mutation);
+ incomingMutations.invokeListeners(mutation.id());
}
finally
{
shardLock.readLock().unlock();
}
}
- public void finishWriting(Mutation mutation)
+ /**
+ * Must be called exactly once per {@code nextId()} invocation.
+ */
+ public void completeLocalWrite(MutationId id)
+ {
+ Preconditions.checkArgument(!id.isNone());
+ Shard shard = getShardNullable(id.asLogId());
+ if (null == shard)
+ throw new IllegalStateException(format("Shard for log %s was not found in log2ShardMap", id.asLogId()));
+ shard.completeLocalWrite();
+ }
+
+ /**
+ * Check the log-to-shard index first; if the log ID is locally unknown, query all peers for the shard metadata,
+ * then find or create the matching shard locally under the read lock.
+ * <p>
+ * The returned shard is safe to write to even if a topology change interleaves between this
+ * call and the subsequent write, because shard recreation carries existing shards forward rather
+ * than discarding them (see {@link KeyspaceShards#withNewShards}).
+ */
+ @Nonnull
+ private Shard getOrCreateShardForMutation(Mutation mutation)
{
+ CoordinatorLogId logId = mutation.id().asLogId();
+
+ Shard shard = getShardNullable(logId);
+ if (null != shard)
+ return shard;
+
+ ShardMetadata metadata = queryPeersForShardMetadata(logId, mutation.getKeyspaceName());
shardLock.readLock().lock();
try
{
- Preconditions.checkArgument(!mutation.id().isNone());
- getOrCreateShards(mutation.getKeyspaceName()).finishWriting(mutation);
- incomingMutations.invokeListeners(mutation.id());
+ return getOrCreateShard(metadata.keyspace, metadata.sinceEpoch, metadata.range, metadata.participants);
}
finally
{
shardLock.readLock().unlock();
}
}
+ /**
+ * When we see a new coordinator log id for the first time, we may not know what shard to place it into.
+ * Queries all peers for the shard metadata, finds or creates the matching shard locally, and returns it.
+ * Throws if resolution fails entirely.
+ */
+ private ShardMetadata queryPeersForShardMetadata(CoordinatorLogId logId, String keyspace)
Review Comment:
This is probably the right way to do this for log ids we don't know about, but I'm worried about this causing a flurry of requests and increase write latencies every time a log rolls over. I thin kwe should address this a few ways.
1) Proactive communication from the allocator of the new log. Putting the range in every mutation id is probably overkill, but the node creating the log should immediately notify it's peers of a new log and it's ranges/epoch/keyspace, or at least attempt to. If it's possible to send the notification before we start issuing mutation ids that would be ideal, but at least sending them as soon as the log is created is probably good enough, so mutations don't have a period where there's an additional round trip on the apply process.
2) The request/response process here should not be happening in the write stage. We should probably detect that we don't have the coordinator id and start the ShardMetadataRequest asynchronously, resubmitting the mutation when it's complete. We should also dedupe calls against a log id as well. One thing I'm not sure of is if that might cause problems with futures waiting on mutation apply?
##########
src/java/org/apache/cassandra/replication/SealingCoordinator.java:
##########
@@ -0,0 +1,1401 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.cassandra.replication;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.TimeoutException;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.concurrent.locks.LockSupport;
+
+import javax.annotation.Nonnull;
+import javax.annotation.Nullable;
+
+import com.google.common.base.Preconditions;
+
+import org.apache.cassandra.config.DatabaseDescriptor;
+import org.apache.cassandra.db.ColumnFamilyStore;
+import org.apache.cassandra.db.Keyspace;
+import org.apache.cassandra.db.TypeSizes;
+import org.apache.cassandra.dht.AbstractBounds;
+import org.apache.cassandra.dht.IPartitioner;
+import org.apache.cassandra.dht.Range;
+import org.apache.cassandra.dht.Token;
+import org.apache.cassandra.exceptions.RequestFailure;
+import org.apache.cassandra.io.UnversionedSerializer;
+import org.apache.cassandra.io.util.DataInputPlus;
+import org.apache.cassandra.io.util.DataOutputPlus;
+import org.apache.cassandra.locator.EndpointsByReplica;
+import org.apache.cassandra.locator.EndpointsForRange;
+import org.apache.cassandra.locator.InetAddressAndPort;
+import org.apache.cassandra.locator.RangesAtEndpoint;
+import org.apache.cassandra.locator.Replica;
+import org.apache.cassandra.net.IVerbHandler;
+import org.apache.cassandra.net.Message;
+import org.apache.cassandra.net.MessagingService;
+import org.apache.cassandra.net.RequestCallback;
+import org.apache.cassandra.net.Verb;
+import org.apache.cassandra.schema.KeyspaceMetadata;
+import org.apache.cassandra.tcm.ClusterMetadata;
+import org.apache.cassandra.tcm.membership.NodeId;
+import org.apache.cassandra.tcm.ownership.DataPlacement;
+import org.apache.cassandra.tcm.ownership.MovementMap;
+import org.apache.cassandra.tcm.ownership.PlacementDeltas;
+import org.apache.cassandra.tcm.ownership.VersionedEndpoints;
+import org.apache.cassandra.tcm.sequences.LeaveStreams;
+import org.apache.cassandra.utils.CollectionSerializers;
+import org.apache.cassandra.utils.FBUtilities;
+import org.apache.cassandra.utils.concurrent.AsyncPromise;
+import org.apache.cassandra.utils.concurrent.Future;
+import org.apache.cassandra.utils.concurrent.FutureCombiner;
+
+import static java.lang.String.format;
+import static java.util.concurrent.TimeUnit.MILLISECONDS;
+import static java.util.concurrent.TimeUnit.NANOSECONDS;
+import static java.util.concurrent.TimeUnit.SECONDS;
+import static org.apache.cassandra.utils.Clock.Global.nanoTime;
+
+/*
+ * Sealing {@link Shard}s once a topology change obsoletes them.
+ *
+ * Happy Path Sequence:
+ * 1. Bring every participant up to (at least) this seal's epoch and fence allocation, so that everyone
+ * stops writing to the old shards; then wait until all in-progress mutations have been done.
+ * 2. Same for tracked transfers
+ * 3. Wait until all participating replicas have reconciled the union of their witnessed ids
+ * 4. Mark the shard as fully sealed
+ * 5. Stop including the shard in mutation summaries for read requests.
+ *
+ * Unhappy Path Sequence (not yet implemented):
+ * 1. Wait until everyone who's up stops writing to the old shards and all in-progress mutations have been done
+ * 2. Same for tracked transfers
+ * 3. Wait until all *live* participating replicas have reconciled the union of their witnessed ids
+ * 4. Mark the shard as partially sealed, label it with the set of participants that were able to seal
+ * 5. When a down node comes up, have it talk to other nodes and learn about sealing, and what mutations/logs
+ * they have exchanged. Diff with what we have - in system tables and in the journal (though system table
+ * plus local replay should give us an up-to-date metadata state for the shard)
+ */
+public final class SealingCoordinator
+{
+ /*
+ * BOOTSTRAP sequence illustrated. Stages in which write placements
+ * remain unchanged, and only read placements change, are skipped
+ * here as irrelevant to sharding / sealing.
+ *
+ * There is a tracked keyspace with RF=3.
+ * A node n35 with token=35 is added in the middle of the ring.
+ *
+ * 1. Initial state at e0:
+ *
+ * (10,20] | {n20,n30,n40} | e0 | ACTIVE
+ * (20,30] | {n30,n40,n50} | e0 | ACTIVE
+ * (30,40] | {n40,n50,n60} | e0 | ACTIVE
+ *
+ * 2. PREPARE_JOIN at e1:
+ *
+ * (10,20] | {n20,n30,n40} | e0 | ACTIVE
+ * (20,30] | {n30,n40,n50} | e0 | ACTIVE
+ * (30,40] | {n40,n50,n60} | e0 | ACTIVE
+ * + (30,35] | {n40,n50,n60} | e1 | ACTIVE
+ * + (35,40] | {n40,n50,n60} | e1 | ACTIVE
+ *
+ * 3. START_JOIN at e2:
+ *
+ * (10,20] | {n20,n30,n40} | e0 | ACTIVE
+ * + (10,20] | {n20,n30,n35,n40} | e2 | ACTIVE
+ * (20,30] | {n30,n40,n50} | e0 | ACTIVE
+ * + (20,30] | {n30,n35,n40,n50} | e2 | ACTIVE
+ * (30,40] | {n40,n50,n60} | e0 | ACTIVE
+ * (30,35] | {n40,n50,n60} | e1 | ACTIVE
+ * + (30,35] | {n35,n40,n50,n60} | e2 | ACTIVE
+ * (35,40] | {n40,n50,n60} | e1 | ACTIVE
+ *
+ * 4. MID_JOIN at e3 [*]:
+ *
+ * S (10,20] | {n20,n30,n40} | e0 | SEALED
+ * (10,20] | {n20,n30,n35,n40} | e2 | ACTIVE
+ * S (20,30] | {n30,n40,n50} | e0 | SEALED
+ * (20,30] | {n30,n35,n40,n50} | e2 | ACTIVE
+ * S (30,40] | {n40,n50,n60} | e0 | SEALED
+ * S (30,35] | {n40,n50,n60} | e1 | SEALED
+ * (30,35] | {n35,n40,n50,n60} | e2 | ACTIVE
+ * (35,40] | {n40,n50,n60} | e1 | ACTIVE
+ *
+ * 5. FINISH_JOIN at e4:
+ *
+ * (10,20] | {n20,n30,n40} | e0 | SEALED
+ * S (10,20] | {n20,n30,n35,n40} | e2 | SEALED
+ * + (10,20] | {n20,n30,n35} | e4 | ACTIVE
+ * (20,30] | {n30,n40,n50} | e0 | SEALED
+ * S (20,30] | {n30,n35,n40,n50} | e2 | SEALED
+ * + (20,30] | {n30,n35,n40} | e4 | ACTIVE
+ * (30,40] | {n40,n50,n60} | e0 | SEALED
+ * (30,35] | {n40,n50,n60} | e1 | SEALED
+ * S (30,35] | {n35,n40,n50,n60} | e2 | SEALED
+ * + (30,35] | {n35,n40,n50} | e4 | ACTIVE
+ * (35,40] | {n40,n50,n60} | e1 | ACTIVE
+ *
+ * [*] At this stage, after sealing the shards, we stream the sstables.
+ * All the offsets belonging to these sealed shards get stripped from
+ * stats component of streamed sstables. Otherwise, the new node
+ * won't be able to compact them or mark them as reconciled/repaired -
+ * it is rightfully unaware of the sealed shards.
+ */
+
+ /**
+ * Seal the shards obsoleted by bootstrap before the new node starts streaming.
+ */
+ public static void sealShardsAtMidJoin(ClusterMetadata metadata, MovementMap movements)
+ {
+ seal(discoverShardsAtMidJoinOrReplace(metadata, movements));
+ }
+
+ /**
+ * Seal the intermediate shards that were created during START_JOIN.
+ * TODO (required): this MUST eventually run while the affected ranges are still locked - between the
+ * removeFromWrites placement change and the range unlock. FinishJoin/FinishReplace currently apply
+ * removeFromWrites and the unlock atomically in a single transformation, so there is no committed
+ * dropped-but-locked epoch to hook. Until then, this runs just after FINISH_JOIN/FINISH_REPLACE,
+ * when the ranges are already unlocked. Left to implement for the unhappy path (AY).
+ */
+ public static void sealShardsAtFinishJoin(ClusterMetadata metadata, PlacementDeltas finishDelta)
+ {
+ seal(discoverShardsAtFinishJoinOrReplace(metadata, finishDelta, null));
+ }
+
+ /*
+ * REPLACE sequence illustrated. Stages in which write placements
+ * remain unchanged, and only read placements change, are skipped
+ * here as irrelevant to sharding / sealing.
+ *
+ * There is a tracked keyspace with RF=3.
+ * Node n40 is dead, and is being replaced with $n40.
+ *
+ * 1. Initial state at e0:
+ *
+ * (10,20] | {n20,n30,n40} | e0 | ACTIVE
+ * (20,30] | {n30,n40,n50} | e0 | ACTIVE
+ * (30,40] | {n40,n50,n60} | e0 | ACTIVE
+ *
+ * 2. START_REPLACE at e1:
+ *
+ * (10,20] | {n20,n30,n40} | e0 | ACTIVE
+ * + (10,20] | {n20,n30,n40,$n40}| e1 | ACTIVE
+ * (20,30] | {n30,n40,n50} | e0 | ACTIVE
+ * + (20,30] | {n30,n40,$n40,n50}| e1 | ACTIVE
+ * (30,40] | {n40,n50,n60} | e0 | ACTIVE
+ * + (30,40] | {n40,$n40,n50,n60}| e1 | ACTIVE
+ *
+ * 3. MID_REPLACE at e2 [*]:
+ *
+ * S (10,20] | {n20,n30,n40} | e0 | SEALED
+ * (10,20] | {n20,n30,n40,$n40}| e1 | ACTIVE
+ * S (20,30] | {n30,n40,n50} | e0 | SEALED
+ * (20,30] | {n30,n40,$n40,n50}| e1 | ACTIVE
+ * S (30,40] | {n40,n50,n60} | e0 | SEALED
+ * (30,40] | {n40,$n40,n50,n60}| e1 | ACTIVE
+ *
+ * 4. FINISH_REPLACE at e3:
+ *
+ * (10,20] | {n20,n30,n40} | e0 | SEALED
+ * S (10,20] | {n20,n30,n40,$n40}| e1 | SEALED
+ * + (10,20] | {n20,n30,$n40} | e3 | ACTIVE
+ * (20,30] | {n30,n40,n50} | e0 | SEALED
+ * S (20,30] | {n30,n40,$n40,n50}| e1 | SEALED
+ * + (20,30] | {n30,$n40,n50} | e3 | ACTIVE
+ * (30,40] | {n40,n50,n60} | e0 | SEALED
+ * S (30,40] | {n40,$n40,n50,n60}| e1 | SEALED
+ * + (30,40] | {$n40,n50,n60} | e3 | ACTIVE
+ *
+ * [*] At this stage, after sealing the shards, we stream the sstables.
+ * All the offsets belonging to these sealed shards get stripped from
+ * stats component of streamed sstables. Otherwise, the new node
+ * won't be able to compact them or mark them as reconciled/repaired -
+ * it is rightfully unaware of the sealed shards.
+ */
+
+ /**
+ * Seal the shards obsoleted by host replacement before the replacement node starts streaming.
+ * The {@code deadNode} node cannot participate in the sealing.
+ */
+ public static void sealShardsAtMidReplace(ClusterMetadata metadata, MovementMap movements, @Nonnull NodeId deadNode)
+ {
+ seal(discoverShardsAtMidJoinOrReplace(metadata, movements), deadNode);
+ }
+
+ public static void sealShardsAtFinishReplace(ClusterMetadata metadata, PlacementDeltas finishDelta, @Nonnull NodeId deadNode)
+ {
+ seal(discoverShardsAtFinishJoinOrReplace(metadata, finishDelta, deadNode), deadNode);
+ }
+
+ /**
+ * Discover all shards that need to be sealed at MID_JOIN or MID_REPLACE.
+ */
+ private static Set<ShardMetadata> discoverShardsAtMidJoinOrReplace(ClusterMetadata metadata, MovementMap movements)
+ {
+ List<Future<Set<ShardMetadata>>> futures = new ArrayList<>();
+
+ for (KeyspaceMetadata ksm : metadata.schema.getKeyspaces())
+ {
+ if (!ksm.params.replicationType.isTracked())
+ continue;
+
+ EndpointsByReplica acquired = movements.get(ksm.params.replication);
+ if (acquired.isEmpty()) // when the keyspace is not replicated to this DC
+ continue;
+
+ DataPlacement placement = metadata.placements.get(ksm.params.replication);
+
+ for (Map.Entry<Replica, EndpointsForRange> entry : acquired.entrySet())
+ {
+ Range<Token> range = entry.getKey().range();
+ Set<InetAddressAndPort> endpoints = entry.getValue().endpoints();
+ // cutoff: the epoch this node was added to the write placement for the range;
+ // shards over the range with sinceEpoch < cutoff are obsoleted by this bootstrap or replace.
+ long beforeEpoch = placement.writes.forRange(range).lastModified().getEpoch();
+ futures.add(FetchShards.fetch(ksm.name, beforeEpoch, range, endpoints));
+ }
+ }
+
+ Set<ShardMetadata> shards = new HashSet<>();
+ try
+ {
+ for (Set<ShardMetadata> fetched : FutureCombiner.allOf(futures).get(DatabaseDescriptor.getRpcTimeout(MILLISECONDS), MILLISECONDS))
+ shards.addAll(fetched);
+ }
+ catch (InterruptedException | ExecutionException | TimeoutException e)
+ {
+ throw new RuntimeException("Failed to fetch shards to seal", e);
+ }
+ return shards;
+ }
+
+ /**
+ * Discover all shards that need to be sealed at FINISH_JOIN or FINISH_REPLACE.
+ */
+ private static Set<ShardMetadata> discoverShardsAtFinishJoinOrReplace(ClusterMetadata metadata, PlacementDeltas finishDelta, @Nullable NodeId deadNode)
+ {
+ long finishEpoch = metadata.epoch.getEpoch();
+ Set<ShardMetadata> shards = new HashSet<>();
+ for (KeyspaceMetadata ksm : metadata.schema.getKeyspaces())
+ {
+ if (!ksm.params.replicationType.isTracked())
+ continue;
+
+ DataPlacement placement = metadata.placements.get(ksm.params.replication);
+ finishDelta.get(ksm.params.replication).writes.removals.flattenValues().forEach(removed -> {
+ Set<Integer> nodeIDs = new HashSet<>();
+ for (InetAddressAndPort ep : placement.writes.forRange(removed.range()).endpoints())
+ nodeIDs.add(metadata.directory.peerId(ep).id());
+ nodeIDs.add(deadNode != null ? deadNode.id() : metadata.directory.peerId(removed.endpoint()).id());
+ Participants participants = new Participants(nodeIDs);
+
+ MutationTrackingService.instance().forEachShardInKeyspace(ksm.name, shard -> {
+ if (shard.sinceEpoch < finishEpoch
+ && shard.range.equals(removed.range())
+ && shard.participants.equals(participants)
+ && !shard.isSealed())
+ shards.add(new ShardMetadata(shard.keyspace, shard.sinceEpoch, shard.range, shard.participants));
+ });
+ });
+ }
+ return shards;
+ }
+
+ /*
+ * UNBOOTSTRAP and REMOVENODE sequences illustrated (same sequence as far as sharding is concerned).
+ * Stages in which write placements remain unchanged, and only read placements change, are skipped
+ * here as irrelevant to sharding / sealing.
+ *
+ * There is a tracked keyspace with RF=3.
+ * A node n40 with token=40 is removed in the middle of the ring.
+ *
+ * 1. Initial state at e0:
+ *
+ * (10,20] | {n20,n30,n40} | e0 | ACTIVE
+ * (20,30] | {n30,n40,n50} | e0 | ACTIVE
+ * (30,40] | {n40,n50,n60} | e0 | ACTIVE
+ * (40,50] | {n50,n60,n70} | e0 | ACTIVE
+ *
+ * 2. START_LEAVE at e1:
+ *
+ * (10,20] | {n20,n30,n40} | e0 | ACTIVE
+ * + (10,20] | {n20,n30,n40,n50} | e1 | ACTIVE
+ * (20,30] | {n30,n40,n50} | e0 | ACTIVE
+ * + (20,30] | {n30,n40,n50,n60} | e1 | ACTIVE
+ * (30,40] | {n40,n50,n60} | e0 | ACTIVE
+ * + (30,40] | {n40,n50,n60,n70} | e1 | ACTIVE
+ * (40,50] | {n50,n60,n70} | e0 | ACTIVE
+ *
+ * 3. MID_LEAVE at e2 [*]:
+ *
+ * S (10,20] | {n20,n30,n40} | e0 | SEALED
+ * (10,20] | {n20,n30,n40,n50} | e1 | ACTIVE
+ * S (20,30] | {n30,n40,n50} | e0 | SEALED
+ * (20,30] | {n30,n40,n50,n60} | e1 | ACTIVE
+ * S (30,40] | {n40,n50,n60} | e0 | SEALED
+ * (30,40] | {n40,n50,n60,n70} | e1 | ACTIVE
+ * (40,50] | {n50,n60,n70} | e0 | ACTIVE
+ *
+ * 4. FINISH_LEAVE at e3:
+ *
+ * (10,20] | {n20,n30,n40} | e0 | SEALED
+ * S (10,20] | {n20,n30,n40,n50} | e1 | SEALED
+ * + (10,20] | {n20,n30,n50} | e3 | ACTIVE
+ * (20,30] | {n30,n40,n50} | e0 | SEALED
+ * S (20,30] | {n30,n40,n50,n60} | e1 | SEALED
+ * + (20,30] | {n30,n50,n60} | e3 | ACTIVE
+ * (30,40] | {n40,n50,n60} | e0 | SEALED
+ * S (30,40] | {n40,n50,n60,n70} | e1 | SEALED
+ * S (40,50] | {n50,n60,n70} | e0 | SEALED
+ * + (30,50] | {n50,n60,n70} | e3 | ACTIVE
+ *
+ * [*] At this stage, after sealing the shards, we stream the sstables.
+ * All the offsets belonging to these sealed shards get stripped from
+ * stats component of streamed sstables. Otherwise, the new node
+ * won't be able to compact them or mark them as reconciled/repaired -
+ * it is rightfully unaware of the sealed shards.
+ */
+
+ /**
+ * Seal the shards obsoleted by START_LEAVE before streaming, for both UNBOOTSTRAP and REMOVENODE.
+ */
+ public static void sealShardsAtMidLeave(ClusterMetadata metadata, PlacementDeltas startDelta, NodeId leavingOrDeparted, LeaveStreams.Kind kind)
+ {
+ InetAddressAndPort leavingOrDepartedEndpoint = metadata.directory.endpoint(leavingOrDeparted);
+
+ List<Future<Set<ShardMetadata>>> futures = new ArrayList<>();
+ for (KeyspaceMetadata ksm : metadata.schema.getKeyspaces())
+ {
+ if (!ksm.params.replicationType.isTracked())
+ continue;
+
+ DataPlacement placement = metadata.placements.get(ksm.params.replication);
+ startDelta.get(ksm.params.replication).writes.additions.flattenValues().forEach(added -> {
+ Range<Token> range = added.range();
+ long beforeEpoch = placement.writes.forRange(range).lastModified().getEpoch();
+ Set<InetAddressAndPort> survivors = new HashSet<>(placement.writes.forRange(range).endpoints());
+ survivors.remove(leavingOrDepartedEndpoint);
+ futures.add(FetchShards.fetch(ksm.name, beforeEpoch, range, survivors));
+ });
+ }
+
+ Set<ShardMetadata> shards = new HashSet<>();
+ try
+ {
+ for (Set<ShardMetadata> fetched : FutureCombiner.allOf(futures).get(DatabaseDescriptor.getRpcTimeout(MILLISECONDS), MILLISECONDS))
+ for (ShardMetadata shard : fetched)
+ if (shard.participants.contains(leavingOrDeparted.id()))
+ shards.add(shard);
+ }
+ catch (InterruptedException | ExecutionException | TimeoutException e)
+ {
+ throw new RuntimeException("Failed to fetch shards to seal for leave", e);
+ }
+
+ seal(shards, kind == LeaveStreams.Kind.UNBOOTSTRAP ? null : leavingOrDeparted);
+ }
+
+ /**
+ * Seal the shards that need to be sealed at LEAVE_FINISH for both UNBOOTSTRAP and REMOVENODE.
+ */
+ public static void sealShardsAtFinishLeave(ClusterMetadata postFinish, PlacementDeltas finishDelta, NodeId leavingOrDeparted, LeaveStreams.Kind kind)
+ {
+ long finishEpoch = postFinish.epoch.getEpoch();
+ InetAddressAndPort leavingOrDepartedEndpoint = postFinish.directory.endpoint(leavingOrDeparted);
+
+ List<Future<Set<ShardMetadata>>> futures = new ArrayList<>();
+ for (KeyspaceMetadata ksm : postFinish.schema.getKeyspaces())
+ {
+ if (!ksm.params.replicationType.isTracked())
+ continue;
+
+ DataPlacement placement = postFinish.placements.get(ksm.params.replication);
+ RangesAtEndpoint removals = finishDelta.get(ksm.params.replication).writes.removals.get(leavingOrDepartedEndpoint);
+ for (Replica removedReplica : removals)
+ {
+ // FINISH_LEAVE merges the leavingOrDeparted node's (pre-merge) range with an adjacent one. Query the *merged*
+ // range (not the pre-merge range) so discovery covers both the leavingOrDeparted node's own obsoleted
+ // generations and the merge-half - the adjacent pre-merge range folded into the merged range, whose
+ // shard never listed the leavingOrDeparted node. matchRange() yields the containing merged group + its
+ // current (survivor) write endpoints.
+ VersionedEndpoints.ForRange group = placement.writes.matchRange(removedReplica.range());
+ Range<Token> mergedRange = group.range();
+ Set<InetAddressAndPort> survivors = new HashSet<>(group.endpoints());
+ survivors.remove(leavingOrDepartedEndpoint);
+ futures.add(FetchShards.fetch(ksm.name, finishEpoch, mergedRange, survivors));
+ }
+ }
+
+ Set<ShardMetadata> shards = new HashSet<>();
+ try
+ {
+ for (Set<ShardMetadata> fetched : FutureCombiner.allOf(futures).get(DatabaseDescriptor.getRpcTimeout(MILLISECONDS), MILLISECONDS))
+ shards.addAll(fetched);
+ }
+ catch (InterruptedException | ExecutionException | TimeoutException e)
+ {
+ throw new RuntimeException("Failed to fetch shards to seal for leave finish", e);
+ }
+
+ seal(shards, kind == LeaveStreams.Kind.UNBOOTSTRAP ? null : leavingOrDeparted);
+ }
+
+ /*
+ * Sealing logic.
+ */
+
+ private static void seal(Set<ShardMetadata> shards)
+ {
+ seal(shards, null);
+ }
+
+ private static void seal(Set<ShardMetadata> shards, @Nullable NodeId withoutNode)
+ {
+ initiate(shards, withoutNode); // ACTIVE -> SEALING for each Shard
+ drain(shards, withoutNode); // drain in-flight local writes
+ reconcile(shards, withoutNode); // wait for logs to reconcile
+ complete(shards, withoutNode); // SEALING-> SEALED for each Shard + journal flush
+ }
+
+ private static void initiate(Set<ShardMetadata> shards, @Nullable NodeId withoutNode)
+ {
+ List<Future<Void>> futures = new ArrayList<>(shards.size());
+ for (ShardMetadata shard : shards)
+ futures.add(initiate(shard, withoutNode));
+ try
+ {
+ FutureCombiner.allOf(futures).get(DatabaseDescriptor.getRpcTimeout(MILLISECONDS), MILLISECONDS);
+ }
+ catch (InterruptedException | ExecutionException | TimeoutException e)
+ {
+ throw new RuntimeException("Failed to initiate sealing", e);
+ }
+ }
+
+ /**
+ * Fence allocation on every participant of the obsoleted shard (ACTIVE -> SEALING) by marking the shard
+ * SEALING on each.
+ * TODO (expected): this assumes every participant has already enacted the topology change that obsoleted the shard (so
+ * it no longer allocates new ids on it). The ProgressBarrier gating bootstrap()/MID_JOIN only waits for
+ * EACH_QUORUM of the affected replicas, NOT all live replicas, so a live-but-lagging participant could still
+ * be on the old shard. Bringing each participant up to the seal epoch must be done per-replica (not
+ * per-shard) and will be added separately. TLDR - add a step to guaranteed up-to-speed rest of replicas epochs.
+ */
+ private static AsyncPromise<Void> initiate(ShardMetadata shard, @Nullable NodeId withoutNode)
+ {
+ return InitSealing.initiate(shard.keyspace, shard.sinceEpoch, shard.range, toEndpoints(shard.participants, withoutNode));
+ }
+
+ private static void drain(Set<ShardMetadata> shards, @Nullable NodeId withoutNode)
+ {
+ long deadlineNanos = nanoTime() + DatabaseDescriptor.getRpcTimeout(NANOSECONDS);
Review Comment:
We should probably use the repair request timeout here, and maybe add some additional buffer with backoff and retry on top. Tight deadlines caused a lot of issues on paxos repair for topology changes and it's not time critical.
##########
src/java/org/apache/cassandra/replication/SealingCoordinator.java:
##########
@@ -0,0 +1,1401 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.cassandra.replication;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.TimeoutException;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.concurrent.locks.LockSupport;
+
+import javax.annotation.Nonnull;
+import javax.annotation.Nullable;
+
+import com.google.common.base.Preconditions;
+
+import org.apache.cassandra.config.DatabaseDescriptor;
+import org.apache.cassandra.db.ColumnFamilyStore;
+import org.apache.cassandra.db.Keyspace;
+import org.apache.cassandra.db.TypeSizes;
+import org.apache.cassandra.dht.AbstractBounds;
+import org.apache.cassandra.dht.IPartitioner;
+import org.apache.cassandra.dht.Range;
+import org.apache.cassandra.dht.Token;
+import org.apache.cassandra.exceptions.RequestFailure;
+import org.apache.cassandra.io.UnversionedSerializer;
+import org.apache.cassandra.io.util.DataInputPlus;
+import org.apache.cassandra.io.util.DataOutputPlus;
+import org.apache.cassandra.locator.EndpointsByReplica;
+import org.apache.cassandra.locator.EndpointsForRange;
+import org.apache.cassandra.locator.InetAddressAndPort;
+import org.apache.cassandra.locator.RangesAtEndpoint;
+import org.apache.cassandra.locator.Replica;
+import org.apache.cassandra.net.IVerbHandler;
+import org.apache.cassandra.net.Message;
+import org.apache.cassandra.net.MessagingService;
+import org.apache.cassandra.net.RequestCallback;
+import org.apache.cassandra.net.Verb;
+import org.apache.cassandra.schema.KeyspaceMetadata;
+import org.apache.cassandra.tcm.ClusterMetadata;
+import org.apache.cassandra.tcm.membership.NodeId;
+import org.apache.cassandra.tcm.ownership.DataPlacement;
+import org.apache.cassandra.tcm.ownership.MovementMap;
+import org.apache.cassandra.tcm.ownership.PlacementDeltas;
+import org.apache.cassandra.tcm.ownership.VersionedEndpoints;
+import org.apache.cassandra.tcm.sequences.LeaveStreams;
+import org.apache.cassandra.utils.CollectionSerializers;
+import org.apache.cassandra.utils.FBUtilities;
+import org.apache.cassandra.utils.concurrent.AsyncPromise;
+import org.apache.cassandra.utils.concurrent.Future;
+import org.apache.cassandra.utils.concurrent.FutureCombiner;
+
+import static java.lang.String.format;
+import static java.util.concurrent.TimeUnit.MILLISECONDS;
+import static java.util.concurrent.TimeUnit.NANOSECONDS;
+import static java.util.concurrent.TimeUnit.SECONDS;
+import static org.apache.cassandra.utils.Clock.Global.nanoTime;
+
+/*
+ * Sealing {@link Shard}s once a topology change obsoletes them.
+ *
+ * Happy Path Sequence:
+ * 1. Bring every participant up to (at least) this seal's epoch and fence allocation, so that everyone
+ * stops writing to the old shards; then wait until all in-progress mutations have been done.
+ * 2. Same for tracked transfers
+ * 3. Wait until all participating replicas have reconciled the union of their witnessed ids
+ * 4. Mark the shard as fully sealed
+ * 5. Stop including the shard in mutation summaries for read requests.
+ *
+ * Unhappy Path Sequence (not yet implemented):
+ * 1. Wait until everyone who's up stops writing to the old shards and all in-progress mutations have been done
+ * 2. Same for tracked transfers
+ * 3. Wait until all *live* participating replicas have reconciled the union of their witnessed ids
+ * 4. Mark the shard as partially sealed, label it with the set of participants that were able to seal
+ * 5. When a down node comes up, have it talk to other nodes and learn about sealing, and what mutations/logs
+ * they have exchanged. Diff with what we have - in system tables and in the journal (though system table
+ * plus local replay should give us an up-to-date metadata state for the shard)
+ */
+public final class SealingCoordinator
+{
+ /*
+ * BOOTSTRAP sequence illustrated. Stages in which write placements
+ * remain unchanged, and only read placements change, are skipped
+ * here as irrelevant to sharding / sealing.
+ *
+ * There is a tracked keyspace with RF=3.
+ * A node n35 with token=35 is added in the middle of the ring.
+ *
+ * 1. Initial state at e0:
+ *
+ * (10,20] | {n20,n30,n40} | e0 | ACTIVE
+ * (20,30] | {n30,n40,n50} | e0 | ACTIVE
+ * (30,40] | {n40,n50,n60} | e0 | ACTIVE
+ *
+ * 2. PREPARE_JOIN at e1:
+ *
+ * (10,20] | {n20,n30,n40} | e0 | ACTIVE
+ * (20,30] | {n30,n40,n50} | e0 | ACTIVE
+ * (30,40] | {n40,n50,n60} | e0 | ACTIVE
+ * + (30,35] | {n40,n50,n60} | e1 | ACTIVE
+ * + (35,40] | {n40,n50,n60} | e1 | ACTIVE
+ *
+ * 3. START_JOIN at e2:
+ *
+ * (10,20] | {n20,n30,n40} | e0 | ACTIVE
+ * + (10,20] | {n20,n30,n35,n40} | e2 | ACTIVE
+ * (20,30] | {n30,n40,n50} | e0 | ACTIVE
+ * + (20,30] | {n30,n35,n40,n50} | e2 | ACTIVE
+ * (30,40] | {n40,n50,n60} | e0 | ACTIVE
+ * (30,35] | {n40,n50,n60} | e1 | ACTIVE
+ * + (30,35] | {n35,n40,n50,n60} | e2 | ACTIVE
+ * (35,40] | {n40,n50,n60} | e1 | ACTIVE
+ *
+ * 4. MID_JOIN at e3 [*]:
+ *
+ * S (10,20] | {n20,n30,n40} | e0 | SEALED
+ * (10,20] | {n20,n30,n35,n40} | e2 | ACTIVE
+ * S (20,30] | {n30,n40,n50} | e0 | SEALED
+ * (20,30] | {n30,n35,n40,n50} | e2 | ACTIVE
+ * S (30,40] | {n40,n50,n60} | e0 | SEALED
+ * S (30,35] | {n40,n50,n60} | e1 | SEALED
+ * (30,35] | {n35,n40,n50,n60} | e2 | ACTIVE
+ * (35,40] | {n40,n50,n60} | e1 | ACTIVE
+ *
+ * 5. FINISH_JOIN at e4:
+ *
+ * (10,20] | {n20,n30,n40} | e0 | SEALED
+ * S (10,20] | {n20,n30,n35,n40} | e2 | SEALED
+ * + (10,20] | {n20,n30,n35} | e4 | ACTIVE
+ * (20,30] | {n30,n40,n50} | e0 | SEALED
+ * S (20,30] | {n30,n35,n40,n50} | e2 | SEALED
+ * + (20,30] | {n30,n35,n40} | e4 | ACTIVE
+ * (30,40] | {n40,n50,n60} | e0 | SEALED
+ * (30,35] | {n40,n50,n60} | e1 | SEALED
+ * S (30,35] | {n35,n40,n50,n60} | e2 | SEALED
+ * + (30,35] | {n35,n40,n50} | e4 | ACTIVE
+ * (35,40] | {n40,n50,n60} | e1 | ACTIVE
+ *
+ * [*] At this stage, after sealing the shards, we stream the sstables.
+ * All the offsets belonging to these sealed shards get stripped from
+ * stats component of streamed sstables. Otherwise, the new node
+ * won't be able to compact them or mark them as reconciled/repaired -
+ * it is rightfully unaware of the sealed shards.
+ */
+
+ /**
+ * Seal the shards obsoleted by bootstrap before the new node starts streaming.
+ */
+ public static void sealShardsAtMidJoin(ClusterMetadata metadata, MovementMap movements)
+ {
+ seal(discoverShardsAtMidJoinOrReplace(metadata, movements));
+ }
+
+ /**
+ * Seal the intermediate shards that were created during START_JOIN.
+ * TODO (required): this MUST eventually run while the affected ranges are still locked - between the
+ * removeFromWrites placement change and the range unlock. FinishJoin/FinishReplace currently apply
+ * removeFromWrites and the unlock atomically in a single transformation, so there is no committed
+ * dropped-but-locked epoch to hook. Until then, this runs just after FINISH_JOIN/FINISH_REPLACE,
+ * when the ranges are already unlocked. Left to implement for the unhappy path (AY).
+ */
+ public static void sealShardsAtFinishJoin(ClusterMetadata metadata, PlacementDeltas finishDelta)
+ {
+ seal(discoverShardsAtFinishJoinOrReplace(metadata, finishDelta, null));
+ }
+
+ /*
+ * REPLACE sequence illustrated. Stages in which write placements
+ * remain unchanged, and only read placements change, are skipped
+ * here as irrelevant to sharding / sealing.
+ *
+ * There is a tracked keyspace with RF=3.
+ * Node n40 is dead, and is being replaced with $n40.
+ *
+ * 1. Initial state at e0:
+ *
+ * (10,20] | {n20,n30,n40} | e0 | ACTIVE
+ * (20,30] | {n30,n40,n50} | e0 | ACTIVE
+ * (30,40] | {n40,n50,n60} | e0 | ACTIVE
+ *
+ * 2. START_REPLACE at e1:
+ *
+ * (10,20] | {n20,n30,n40} | e0 | ACTIVE
+ * + (10,20] | {n20,n30,n40,$n40}| e1 | ACTIVE
+ * (20,30] | {n30,n40,n50} | e0 | ACTIVE
+ * + (20,30] | {n30,n40,$n40,n50}| e1 | ACTIVE
+ * (30,40] | {n40,n50,n60} | e0 | ACTIVE
+ * + (30,40] | {n40,$n40,n50,n60}| e1 | ACTIVE
+ *
+ * 3. MID_REPLACE at e2 [*]:
+ *
+ * S (10,20] | {n20,n30,n40} | e0 | SEALED
+ * (10,20] | {n20,n30,n40,$n40}| e1 | ACTIVE
+ * S (20,30] | {n30,n40,n50} | e0 | SEALED
+ * (20,30] | {n30,n40,$n40,n50}| e1 | ACTIVE
+ * S (30,40] | {n40,n50,n60} | e0 | SEALED
+ * (30,40] | {n40,$n40,n50,n60}| e1 | ACTIVE
+ *
+ * 4. FINISH_REPLACE at e3:
+ *
+ * (10,20] | {n20,n30,n40} | e0 | SEALED
+ * S (10,20] | {n20,n30,n40,$n40}| e1 | SEALED
+ * + (10,20] | {n20,n30,$n40} | e3 | ACTIVE
+ * (20,30] | {n30,n40,n50} | e0 | SEALED
+ * S (20,30] | {n30,n40,$n40,n50}| e1 | SEALED
+ * + (20,30] | {n30,$n40,n50} | e3 | ACTIVE
+ * (30,40] | {n40,n50,n60} | e0 | SEALED
+ * S (30,40] | {n40,$n40,n50,n60}| e1 | SEALED
+ * + (30,40] | {$n40,n50,n60} | e3 | ACTIVE
+ *
+ * [*] At this stage, after sealing the shards, we stream the sstables.
+ * All the offsets belonging to these sealed shards get stripped from
+ * stats component of streamed sstables. Otherwise, the new node
+ * won't be able to compact them or mark them as reconciled/repaired -
+ * it is rightfully unaware of the sealed shards.
+ */
+
+ /**
+ * Seal the shards obsoleted by host replacement before the replacement node starts streaming.
+ * The {@code deadNode} node cannot participate in the sealing.
+ */
+ public static void sealShardsAtMidReplace(ClusterMetadata metadata, MovementMap movements, @Nonnull NodeId deadNode)
+ {
+ seal(discoverShardsAtMidJoinOrReplace(metadata, movements), deadNode);
+ }
+
+ public static void sealShardsAtFinishReplace(ClusterMetadata metadata, PlacementDeltas finishDelta, @Nonnull NodeId deadNode)
+ {
+ seal(discoverShardsAtFinishJoinOrReplace(metadata, finishDelta, deadNode), deadNode);
+ }
+
+ /**
+ * Discover all shards that need to be sealed at MID_JOIN or MID_REPLACE.
+ */
+ private static Set<ShardMetadata> discoverShardsAtMidJoinOrReplace(ClusterMetadata metadata, MovementMap movements)
+ {
+ List<Future<Set<ShardMetadata>>> futures = new ArrayList<>();
+
+ for (KeyspaceMetadata ksm : metadata.schema.getKeyspaces())
+ {
+ if (!ksm.params.replicationType.isTracked())
+ continue;
+
+ EndpointsByReplica acquired = movements.get(ksm.params.replication);
+ if (acquired.isEmpty()) // when the keyspace is not replicated to this DC
+ continue;
+
+ DataPlacement placement = metadata.placements.get(ksm.params.replication);
+
+ for (Map.Entry<Replica, EndpointsForRange> entry : acquired.entrySet())
+ {
+ Range<Token> range = entry.getKey().range();
+ Set<InetAddressAndPort> endpoints = entry.getValue().endpoints();
+ // cutoff: the epoch this node was added to the write placement for the range;
+ // shards over the range with sinceEpoch < cutoff are obsoleted by this bootstrap or replace.
+ long beforeEpoch = placement.writes.forRange(range).lastModified().getEpoch();
+ futures.add(FetchShards.fetch(ksm.name, beforeEpoch, range, endpoints));
+ }
+ }
+
+ Set<ShardMetadata> shards = new HashSet<>();
+ try
+ {
+ for (Set<ShardMetadata> fetched : FutureCombiner.allOf(futures).get(DatabaseDescriptor.getRpcTimeout(MILLISECONDS), MILLISECONDS))
+ shards.addAll(fetched);
+ }
+ catch (InterruptedException | ExecutionException | TimeoutException e)
+ {
+ throw new RuntimeException("Failed to fetch shards to seal", e);
+ }
+ return shards;
+ }
+
+ /**
+ * Discover all shards that need to be sealed at FINISH_JOIN or FINISH_REPLACE.
+ */
+ private static Set<ShardMetadata> discoverShardsAtFinishJoinOrReplace(ClusterMetadata metadata, PlacementDeltas finishDelta, @Nullable NodeId deadNode)
+ {
+ long finishEpoch = metadata.epoch.getEpoch();
+ Set<ShardMetadata> shards = new HashSet<>();
+ for (KeyspaceMetadata ksm : metadata.schema.getKeyspaces())
+ {
+ if (!ksm.params.replicationType.isTracked())
+ continue;
+
+ DataPlacement placement = metadata.placements.get(ksm.params.replication);
+ finishDelta.get(ksm.params.replication).writes.removals.flattenValues().forEach(removed -> {
+ Set<Integer> nodeIDs = new HashSet<>();
+ for (InetAddressAndPort ep : placement.writes.forRange(removed.range()).endpoints())
+ nodeIDs.add(metadata.directory.peerId(ep).id());
+ nodeIDs.add(deadNode != null ? deadNode.id() : metadata.directory.peerId(removed.endpoint()).id());
+ Participants participants = new Participants(nodeIDs);
+
+ MutationTrackingService.instance().forEachShardInKeyspace(ksm.name, shard -> {
+ if (shard.sinceEpoch < finishEpoch
+ && shard.range.equals(removed.range())
+ && shard.participants.equals(participants)
+ && !shard.isSealed())
+ shards.add(new ShardMetadata(shard.keyspace, shard.sinceEpoch, shard.range, shard.participants));
+ });
+ });
+ }
+ return shards;
+ }
+
+ /*
+ * UNBOOTSTRAP and REMOVENODE sequences illustrated (same sequence as far as sharding is concerned).
+ * Stages in which write placements remain unchanged, and only read placements change, are skipped
+ * here as irrelevant to sharding / sealing.
+ *
+ * There is a tracked keyspace with RF=3.
+ * A node n40 with token=40 is removed in the middle of the ring.
+ *
+ * 1. Initial state at e0:
+ *
+ * (10,20] | {n20,n30,n40} | e0 | ACTIVE
+ * (20,30] | {n30,n40,n50} | e0 | ACTIVE
+ * (30,40] | {n40,n50,n60} | e0 | ACTIVE
+ * (40,50] | {n50,n60,n70} | e0 | ACTIVE
+ *
+ * 2. START_LEAVE at e1:
+ *
+ * (10,20] | {n20,n30,n40} | e0 | ACTIVE
+ * + (10,20] | {n20,n30,n40,n50} | e1 | ACTIVE
+ * (20,30] | {n30,n40,n50} | e0 | ACTIVE
+ * + (20,30] | {n30,n40,n50,n60} | e1 | ACTIVE
+ * (30,40] | {n40,n50,n60} | e0 | ACTIVE
+ * + (30,40] | {n40,n50,n60,n70} | e1 | ACTIVE
+ * (40,50] | {n50,n60,n70} | e0 | ACTIVE
+ *
+ * 3. MID_LEAVE at e2 [*]:
+ *
+ * S (10,20] | {n20,n30,n40} | e0 | SEALED
+ * (10,20] | {n20,n30,n40,n50} | e1 | ACTIVE
+ * S (20,30] | {n30,n40,n50} | e0 | SEALED
+ * (20,30] | {n30,n40,n50,n60} | e1 | ACTIVE
+ * S (30,40] | {n40,n50,n60} | e0 | SEALED
+ * (30,40] | {n40,n50,n60,n70} | e1 | ACTIVE
+ * (40,50] | {n50,n60,n70} | e0 | ACTIVE
+ *
+ * 4. FINISH_LEAVE at e3:
+ *
+ * (10,20] | {n20,n30,n40} | e0 | SEALED
+ * S (10,20] | {n20,n30,n40,n50} | e1 | SEALED
+ * + (10,20] | {n20,n30,n50} | e3 | ACTIVE
+ * (20,30] | {n30,n40,n50} | e0 | SEALED
+ * S (20,30] | {n30,n40,n50,n60} | e1 | SEALED
+ * + (20,30] | {n30,n50,n60} | e3 | ACTIVE
+ * (30,40] | {n40,n50,n60} | e0 | SEALED
+ * S (30,40] | {n40,n50,n60,n70} | e1 | SEALED
+ * S (40,50] | {n50,n60,n70} | e0 | SEALED
+ * + (30,50] | {n50,n60,n70} | e3 | ACTIVE
+ *
+ * [*] At this stage, after sealing the shards, we stream the sstables.
+ * All the offsets belonging to these sealed shards get stripped from
+ * stats component of streamed sstables. Otherwise, the new node
+ * won't be able to compact them or mark them as reconciled/repaired -
+ * it is rightfully unaware of the sealed shards.
+ */
+
+ /**
+ * Seal the shards obsoleted by START_LEAVE before streaming, for both UNBOOTSTRAP and REMOVENODE.
+ */
+ public static void sealShardsAtMidLeave(ClusterMetadata metadata, PlacementDeltas startDelta, NodeId leavingOrDeparted, LeaveStreams.Kind kind)
+ {
+ InetAddressAndPort leavingOrDepartedEndpoint = metadata.directory.endpoint(leavingOrDeparted);
+
+ List<Future<Set<ShardMetadata>>> futures = new ArrayList<>();
+ for (KeyspaceMetadata ksm : metadata.schema.getKeyspaces())
+ {
+ if (!ksm.params.replicationType.isTracked())
+ continue;
+
+ DataPlacement placement = metadata.placements.get(ksm.params.replication);
+ startDelta.get(ksm.params.replication).writes.additions.flattenValues().forEach(added -> {
+ Range<Token> range = added.range();
+ long beforeEpoch = placement.writes.forRange(range).lastModified().getEpoch();
+ Set<InetAddressAndPort> survivors = new HashSet<>(placement.writes.forRange(range).endpoints());
+ survivors.remove(leavingOrDepartedEndpoint);
+ futures.add(FetchShards.fetch(ksm.name, beforeEpoch, range, survivors));
+ });
+ }
+
+ Set<ShardMetadata> shards = new HashSet<>();
+ try
+ {
+ for (Set<ShardMetadata> fetched : FutureCombiner.allOf(futures).get(DatabaseDescriptor.getRpcTimeout(MILLISECONDS), MILLISECONDS))
+ for (ShardMetadata shard : fetched)
+ if (shard.participants.contains(leavingOrDeparted.id()))
+ shards.add(shard);
+ }
+ catch (InterruptedException | ExecutionException | TimeoutException e)
+ {
+ throw new RuntimeException("Failed to fetch shards to seal for leave", e);
+ }
+
+ seal(shards, kind == LeaveStreams.Kind.UNBOOTSTRAP ? null : leavingOrDeparted);
+ }
+
+ /**
+ * Seal the shards that need to be sealed at LEAVE_FINISH for both UNBOOTSTRAP and REMOVENODE.
+ */
+ public static void sealShardsAtFinishLeave(ClusterMetadata postFinish, PlacementDeltas finishDelta, NodeId leavingOrDeparted, LeaveStreams.Kind kind)
+ {
+ long finishEpoch = postFinish.epoch.getEpoch();
+ InetAddressAndPort leavingOrDepartedEndpoint = postFinish.directory.endpoint(leavingOrDeparted);
+
+ List<Future<Set<ShardMetadata>>> futures = new ArrayList<>();
+ for (KeyspaceMetadata ksm : postFinish.schema.getKeyspaces())
+ {
+ if (!ksm.params.replicationType.isTracked())
+ continue;
+
+ DataPlacement placement = postFinish.placements.get(ksm.params.replication);
+ RangesAtEndpoint removals = finishDelta.get(ksm.params.replication).writes.removals.get(leavingOrDepartedEndpoint);
+ for (Replica removedReplica : removals)
+ {
+ // FINISH_LEAVE merges the leavingOrDeparted node's (pre-merge) range with an adjacent one. Query the *merged*
+ // range (not the pre-merge range) so discovery covers both the leavingOrDeparted node's own obsoleted
+ // generations and the merge-half - the adjacent pre-merge range folded into the merged range, whose
+ // shard never listed the leavingOrDeparted node. matchRange() yields the containing merged group + its
+ // current (survivor) write endpoints.
+ VersionedEndpoints.ForRange group = placement.writes.matchRange(removedReplica.range());
+ Range<Token> mergedRange = group.range();
+ Set<InetAddressAndPort> survivors = new HashSet<>(group.endpoints());
+ survivors.remove(leavingOrDepartedEndpoint);
+ futures.add(FetchShards.fetch(ksm.name, finishEpoch, mergedRange, survivors));
+ }
+ }
+
+ Set<ShardMetadata> shards = new HashSet<>();
+ try
+ {
+ for (Set<ShardMetadata> fetched : FutureCombiner.allOf(futures).get(DatabaseDescriptor.getRpcTimeout(MILLISECONDS), MILLISECONDS))
+ shards.addAll(fetched);
+ }
+ catch (InterruptedException | ExecutionException | TimeoutException e)
+ {
+ throw new RuntimeException("Failed to fetch shards to seal for leave finish", e);
+ }
+
+ seal(shards, kind == LeaveStreams.Kind.UNBOOTSTRAP ? null : leavingOrDeparted);
+ }
+
+ /*
+ * Sealing logic.
+ */
+
+ private static void seal(Set<ShardMetadata> shards)
+ {
+ seal(shards, null);
+ }
+
+ private static void seal(Set<ShardMetadata> shards, @Nullable NodeId withoutNode)
+ {
+ initiate(shards, withoutNode); // ACTIVE -> SEALING for each Shard
+ drain(shards, withoutNode); // drain in-flight local writes
+ reconcile(shards, withoutNode); // wait for logs to reconcile
+ complete(shards, withoutNode); // SEALING-> SEALED for each Shard + journal flush
+ }
+
+ private static void initiate(Set<ShardMetadata> shards, @Nullable NodeId withoutNode)
+ {
+ List<Future<Void>> futures = new ArrayList<>(shards.size());
+ for (ShardMetadata shard : shards)
+ futures.add(initiate(shard, withoutNode));
+ try
+ {
+ FutureCombiner.allOf(futures).get(DatabaseDescriptor.getRpcTimeout(MILLISECONDS), MILLISECONDS);
+ }
+ catch (InterruptedException | ExecutionException | TimeoutException e)
+ {
+ throw new RuntimeException("Failed to initiate sealing", e);
+ }
+ }
+
+ /**
+ * Fence allocation on every participant of the obsoleted shard (ACTIVE -> SEALING) by marking the shard
+ * SEALING on each.
+ * TODO (expected): this assumes every participant has already enacted the topology change that obsoleted the shard (so
+ * it no longer allocates new ids on it). The ProgressBarrier gating bootstrap()/MID_JOIN only waits for
+ * EACH_QUORUM of the affected replicas, NOT all live replicas, so a live-but-lagging participant could still
+ * be on the old shard. Bringing each participant up to the seal epoch must be done per-replica (not
+ * per-shard) and will be added separately. TLDR - add a step to guaranteed up-to-speed rest of replicas epochs.
+ */
+ private static AsyncPromise<Void> initiate(ShardMetadata shard, @Nullable NodeId withoutNode)
+ {
+ return InitSealing.initiate(shard.keyspace, shard.sinceEpoch, shard.range, toEndpoints(shard.participants, withoutNode));
+ }
+
+ private static void drain(Set<ShardMetadata> shards, @Nullable NodeId withoutNode)
+ {
+ long deadlineNanos = nanoTime() + DatabaseDescriptor.getRpcTimeout(NANOSECONDS);
+ Set<ShardMetadata> pending = new HashSet<>(shards);
+ while (true)
+ {
+ // non-blocking poll: each participant reports its drain status immediately; retry until all drained
+ pending.removeIf(shard -> drain(shard, withoutNode));
+ if (pending.isEmpty())
+ return;
+ if (nanoTime() >= deadlineNanos)
+ throw new RuntimeException("Timed out draining shards; still pending: " + pending);
+ LockSupport.parkNanos(SECONDS.toNanos(1));
+ }
+ }
+
+ /**
+ * Poll every participant of the SEALING shard once for drain status; true iff all report that no mutation
+ * id allocated before the SEALING fence is still applying locally.
+ */
+ private static boolean drain(ShardMetadata shard, @Nullable NodeId withoutNode)
+ {
+ try
+ {
+ return Drain.poll(shard.keyspace, shard.sinceEpoch, shard.range, toEndpoints(shard.participants, withoutNode))
+ .get(DatabaseDescriptor.getRpcTimeout(MILLISECONDS), MILLISECONDS);
+ }
+ catch (InterruptedException | ExecutionException | TimeoutException e)
+ {
+ throw new RuntimeException(format("Failed to poll drain for shard %s", shard), e);
+ }
+ }
+
+ /**
+ * For each of the shards, capture the witnessed union then poll each participant until it has
+ * caught up and witnessed the entire offset union itself.
+ */
+ private static void reconcile(Set<ShardMetadata> shards, @Nullable NodeId withoutNode)
+ {
+ for (ShardMetadata shard : shards)
+ reconcile(shard, withoutNode);
+ }
+
+ private static void reconcile(ShardMetadata shard, @Nullable NodeId withoutNode)
+ {
+ // capture shard's witnessed offsets once
+ Log2OffsetsMap.Mutable offsets = captureWitnessedOffsets(shard, withoutNode);
+ // poll every participant until they've each witnessed the union of offsets
+ for (InetAddressAndPort endpoint : toEndpoints(shard.participants, withoutNode))
+ pollUntilWitnesses(shard, offsets, endpoint);
+ }
+
+ private static Log2OffsetsMap.Mutable captureWitnessedOffsets(ShardMetadata shard, @Nullable NodeId withoutNode)
+ {
+ List<InetAddressAndPort> endpoints = toEndpoints(shard.participants, withoutNode);
+ try
+ {
+ return ReconcileCapture.capture(shard.keyspace, shard.sinceEpoch, shard.range, endpoints)
+ .get(DatabaseDescriptor.getRpcTimeout(MILLISECONDS), MILLISECONDS);
+ }
+ catch (InterruptedException | ExecutionException | TimeoutException e)
+ {
+ throw new RuntimeException(format("Failed to capture witnessed offsets for shard %s", shard), e);
+ }
+ }
+
+ private static void pollUntilWitnesses(ShardMetadata shard, Log2OffsetsMap.Mutable offsets, InetAddressAndPort endpoint)
+ {
+ // TODO (expected): use a longer timeout
+ long deadlineNanos = nanoTime() + DatabaseDescriptor.getRpcTimeout(NANOSECONDS);
Review Comment:
Same timeout+retry+backoff comment here
##########
src/java/org/apache/cassandra/replication/MutationTrackingService.java:
##########
@@ -702,14 +703,15 @@ public long getUnreconciledMutationCount()
return count[0];
}
+ // TODO (expected): what? this is not the way; this should grab an immutable snapshot instead
Review Comment:
```suggestion
// TODO (expected): this should grab an immutable snapshot instead
```
##########
src/java/org/apache/cassandra/replication/SealingCoordinator.java:
##########
@@ -0,0 +1,1401 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.cassandra.replication;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.TimeoutException;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.concurrent.locks.LockSupport;
+
+import javax.annotation.Nonnull;
+import javax.annotation.Nullable;
+
+import com.google.common.base.Preconditions;
+
+import org.apache.cassandra.config.DatabaseDescriptor;
+import org.apache.cassandra.db.ColumnFamilyStore;
+import org.apache.cassandra.db.Keyspace;
+import org.apache.cassandra.db.TypeSizes;
+import org.apache.cassandra.dht.AbstractBounds;
+import org.apache.cassandra.dht.IPartitioner;
+import org.apache.cassandra.dht.Range;
+import org.apache.cassandra.dht.Token;
+import org.apache.cassandra.exceptions.RequestFailure;
+import org.apache.cassandra.io.UnversionedSerializer;
+import org.apache.cassandra.io.util.DataInputPlus;
+import org.apache.cassandra.io.util.DataOutputPlus;
+import org.apache.cassandra.locator.EndpointsByReplica;
+import org.apache.cassandra.locator.EndpointsForRange;
+import org.apache.cassandra.locator.InetAddressAndPort;
+import org.apache.cassandra.locator.RangesAtEndpoint;
+import org.apache.cassandra.locator.Replica;
+import org.apache.cassandra.net.IVerbHandler;
+import org.apache.cassandra.net.Message;
+import org.apache.cassandra.net.MessagingService;
+import org.apache.cassandra.net.RequestCallback;
+import org.apache.cassandra.net.Verb;
+import org.apache.cassandra.schema.KeyspaceMetadata;
+import org.apache.cassandra.tcm.ClusterMetadata;
+import org.apache.cassandra.tcm.membership.NodeId;
+import org.apache.cassandra.tcm.ownership.DataPlacement;
+import org.apache.cassandra.tcm.ownership.MovementMap;
+import org.apache.cassandra.tcm.ownership.PlacementDeltas;
+import org.apache.cassandra.tcm.ownership.VersionedEndpoints;
+import org.apache.cassandra.tcm.sequences.LeaveStreams;
+import org.apache.cassandra.utils.CollectionSerializers;
+import org.apache.cassandra.utils.FBUtilities;
+import org.apache.cassandra.utils.concurrent.AsyncPromise;
+import org.apache.cassandra.utils.concurrent.Future;
+import org.apache.cassandra.utils.concurrent.FutureCombiner;
+
+import static java.lang.String.format;
+import static java.util.concurrent.TimeUnit.MILLISECONDS;
+import static java.util.concurrent.TimeUnit.NANOSECONDS;
+import static java.util.concurrent.TimeUnit.SECONDS;
+import static org.apache.cassandra.utils.Clock.Global.nanoTime;
+
+/*
+ * Sealing {@link Shard}s once a topology change obsoletes them.
+ *
+ * Happy Path Sequence:
+ * 1. Bring every participant up to (at least) this seal's epoch and fence allocation, so that everyone
+ * stops writing to the old shards; then wait until all in-progress mutations have been done.
+ * 2. Same for tracked transfers
+ * 3. Wait until all participating replicas have reconciled the union of their witnessed ids
+ * 4. Mark the shard as fully sealed
+ * 5. Stop including the shard in mutation summaries for read requests.
+ *
+ * Unhappy Path Sequence (not yet implemented):
+ * 1. Wait until everyone who's up stops writing to the old shards and all in-progress mutations have been done
+ * 2. Same for tracked transfers
+ * 3. Wait until all *live* participating replicas have reconciled the union of their witnessed ids
+ * 4. Mark the shard as partially sealed, label it with the set of participants that were able to seal
+ * 5. When a down node comes up, have it talk to other nodes and learn about sealing, and what mutations/logs
+ * they have exchanged. Diff with what we have - in system tables and in the journal (though system table
+ * plus local replay should give us an up-to-date metadata state for the shard)
+ */
+public final class SealingCoordinator
+{
+ /*
+ * BOOTSTRAP sequence illustrated. Stages in which write placements
+ * remain unchanged, and only read placements change, are skipped
+ * here as irrelevant to sharding / sealing.
+ *
+ * There is a tracked keyspace with RF=3.
+ * A node n35 with token=35 is added in the middle of the ring.
+ *
+ * 1. Initial state at e0:
+ *
+ * (10,20] | {n20,n30,n40} | e0 | ACTIVE
+ * (20,30] | {n30,n40,n50} | e0 | ACTIVE
+ * (30,40] | {n40,n50,n60} | e0 | ACTIVE
+ *
+ * 2. PREPARE_JOIN at e1:
+ *
+ * (10,20] | {n20,n30,n40} | e0 | ACTIVE
+ * (20,30] | {n30,n40,n50} | e0 | ACTIVE
+ * (30,40] | {n40,n50,n60} | e0 | ACTIVE
+ * + (30,35] | {n40,n50,n60} | e1 | ACTIVE
+ * + (35,40] | {n40,n50,n60} | e1 | ACTIVE
+ *
+ * 3. START_JOIN at e2:
+ *
+ * (10,20] | {n20,n30,n40} | e0 | ACTIVE
+ * + (10,20] | {n20,n30,n35,n40} | e2 | ACTIVE
+ * (20,30] | {n30,n40,n50} | e0 | ACTIVE
+ * + (20,30] | {n30,n35,n40,n50} | e2 | ACTIVE
+ * (30,40] | {n40,n50,n60} | e0 | ACTIVE
+ * (30,35] | {n40,n50,n60} | e1 | ACTIVE
+ * + (30,35] | {n35,n40,n50,n60} | e2 | ACTIVE
+ * (35,40] | {n40,n50,n60} | e1 | ACTIVE
+ *
+ * 4. MID_JOIN at e3 [*]:
+ *
+ * S (10,20] | {n20,n30,n40} | e0 | SEALED
+ * (10,20] | {n20,n30,n35,n40} | e2 | ACTIVE
+ * S (20,30] | {n30,n40,n50} | e0 | SEALED
+ * (20,30] | {n30,n35,n40,n50} | e2 | ACTIVE
+ * S (30,40] | {n40,n50,n60} | e0 | SEALED
+ * S (30,35] | {n40,n50,n60} | e1 | SEALED
+ * (30,35] | {n35,n40,n50,n60} | e2 | ACTIVE
+ * (35,40] | {n40,n50,n60} | e1 | ACTIVE
+ *
+ * 5. FINISH_JOIN at e4:
+ *
+ * (10,20] | {n20,n30,n40} | e0 | SEALED
+ * S (10,20] | {n20,n30,n35,n40} | e2 | SEALED
+ * + (10,20] | {n20,n30,n35} | e4 | ACTIVE
+ * (20,30] | {n30,n40,n50} | e0 | SEALED
+ * S (20,30] | {n30,n35,n40,n50} | e2 | SEALED
+ * + (20,30] | {n30,n35,n40} | e4 | ACTIVE
+ * (30,40] | {n40,n50,n60} | e0 | SEALED
+ * (30,35] | {n40,n50,n60} | e1 | SEALED
+ * S (30,35] | {n35,n40,n50,n60} | e2 | SEALED
+ * + (30,35] | {n35,n40,n50} | e4 | ACTIVE
+ * (35,40] | {n40,n50,n60} | e1 | ACTIVE
+ *
+ * [*] At this stage, after sealing the shards, we stream the sstables.
+ * All the offsets belonging to these sealed shards get stripped from
+ * stats component of streamed sstables. Otherwise, the new node
+ * won't be able to compact them or mark them as reconciled/repaired -
+ * it is rightfully unaware of the sealed shards.
+ */
+
+ /**
+ * Seal the shards obsoleted by bootstrap before the new node starts streaming.
+ */
+ public static void sealShardsAtMidJoin(ClusterMetadata metadata, MovementMap movements)
+ {
+ seal(discoverShardsAtMidJoinOrReplace(metadata, movements));
+ }
+
+ /**
+ * Seal the intermediate shards that were created during START_JOIN.
+ * TODO (required): this MUST eventually run while the affected ranges are still locked - between the
+ * removeFromWrites placement change and the range unlock. FinishJoin/FinishReplace currently apply
+ * removeFromWrites and the unlock atomically in a single transformation, so there is no committed
+ * dropped-but-locked epoch to hook. Until then, this runs just after FINISH_JOIN/FINISH_REPLACE,
+ * when the ranges are already unlocked. Left to implement for the unhappy path (AY).
+ */
+ public static void sealShardsAtFinishJoin(ClusterMetadata metadata, PlacementDeltas finishDelta)
+ {
+ seal(discoverShardsAtFinishJoinOrReplace(metadata, finishDelta, null));
+ }
+
+ /*
+ * REPLACE sequence illustrated. Stages in which write placements
+ * remain unchanged, and only read placements change, are skipped
+ * here as irrelevant to sharding / sealing.
+ *
+ * There is a tracked keyspace with RF=3.
+ * Node n40 is dead, and is being replaced with $n40.
+ *
+ * 1. Initial state at e0:
+ *
+ * (10,20] | {n20,n30,n40} | e0 | ACTIVE
+ * (20,30] | {n30,n40,n50} | e0 | ACTIVE
+ * (30,40] | {n40,n50,n60} | e0 | ACTIVE
+ *
+ * 2. START_REPLACE at e1:
+ *
+ * (10,20] | {n20,n30,n40} | e0 | ACTIVE
+ * + (10,20] | {n20,n30,n40,$n40}| e1 | ACTIVE
+ * (20,30] | {n30,n40,n50} | e0 | ACTIVE
+ * + (20,30] | {n30,n40,$n40,n50}| e1 | ACTIVE
+ * (30,40] | {n40,n50,n60} | e0 | ACTIVE
+ * + (30,40] | {n40,$n40,n50,n60}| e1 | ACTIVE
+ *
+ * 3. MID_REPLACE at e2 [*]:
+ *
+ * S (10,20] | {n20,n30,n40} | e0 | SEALED
+ * (10,20] | {n20,n30,n40,$n40}| e1 | ACTIVE
+ * S (20,30] | {n30,n40,n50} | e0 | SEALED
+ * (20,30] | {n30,n40,$n40,n50}| e1 | ACTIVE
+ * S (30,40] | {n40,n50,n60} | e0 | SEALED
+ * (30,40] | {n40,$n40,n50,n60}| e1 | ACTIVE
+ *
+ * 4. FINISH_REPLACE at e3:
+ *
+ * (10,20] | {n20,n30,n40} | e0 | SEALED
+ * S (10,20] | {n20,n30,n40,$n40}| e1 | SEALED
+ * + (10,20] | {n20,n30,$n40} | e3 | ACTIVE
+ * (20,30] | {n30,n40,n50} | e0 | SEALED
+ * S (20,30] | {n30,n40,$n40,n50}| e1 | SEALED
+ * + (20,30] | {n30,$n40,n50} | e3 | ACTIVE
+ * (30,40] | {n40,n50,n60} | e0 | SEALED
+ * S (30,40] | {n40,$n40,n50,n60}| e1 | SEALED
+ * + (30,40] | {$n40,n50,n60} | e3 | ACTIVE
+ *
+ * [*] At this stage, after sealing the shards, we stream the sstables.
+ * All the offsets belonging to these sealed shards get stripped from
+ * stats component of streamed sstables. Otherwise, the new node
+ * won't be able to compact them or mark them as reconciled/repaired -
+ * it is rightfully unaware of the sealed shards.
+ */
+
+ /**
+ * Seal the shards obsoleted by host replacement before the replacement node starts streaming.
+ * The {@code deadNode} node cannot participate in the sealing.
+ */
+ public static void sealShardsAtMidReplace(ClusterMetadata metadata, MovementMap movements, @Nonnull NodeId deadNode)
+ {
+ seal(discoverShardsAtMidJoinOrReplace(metadata, movements), deadNode);
+ }
+
+ public static void sealShardsAtFinishReplace(ClusterMetadata metadata, PlacementDeltas finishDelta, @Nonnull NodeId deadNode)
+ {
+ seal(discoverShardsAtFinishJoinOrReplace(metadata, finishDelta, deadNode), deadNode);
+ }
+
+ /**
+ * Discover all shards that need to be sealed at MID_JOIN or MID_REPLACE.
+ */
+ private static Set<ShardMetadata> discoverShardsAtMidJoinOrReplace(ClusterMetadata metadata, MovementMap movements)
+ {
+ List<Future<Set<ShardMetadata>>> futures = new ArrayList<>();
+
+ for (KeyspaceMetadata ksm : metadata.schema.getKeyspaces())
+ {
+ if (!ksm.params.replicationType.isTracked())
+ continue;
+
+ EndpointsByReplica acquired = movements.get(ksm.params.replication);
+ if (acquired.isEmpty()) // when the keyspace is not replicated to this DC
+ continue;
+
+ DataPlacement placement = metadata.placements.get(ksm.params.replication);
+
+ for (Map.Entry<Replica, EndpointsForRange> entry : acquired.entrySet())
+ {
+ Range<Token> range = entry.getKey().range();
+ Set<InetAddressAndPort> endpoints = entry.getValue().endpoints();
+ // cutoff: the epoch this node was added to the write placement for the range;
+ // shards over the range with sinceEpoch < cutoff are obsoleted by this bootstrap or replace.
+ long beforeEpoch = placement.writes.forRange(range).lastModified().getEpoch();
+ futures.add(FetchShards.fetch(ksm.name, beforeEpoch, range, endpoints));
+ }
+ }
+
+ Set<ShardMetadata> shards = new HashSet<>();
+ try
+ {
+ for (Set<ShardMetadata> fetched : FutureCombiner.allOf(futures).get(DatabaseDescriptor.getRpcTimeout(MILLISECONDS), MILLISECONDS))
+ shards.addAll(fetched);
+ }
+ catch (InterruptedException | ExecutionException | TimeoutException e)
+ {
+ throw new RuntimeException("Failed to fetch shards to seal", e);
+ }
+ return shards;
+ }
+
+ /**
+ * Discover all shards that need to be sealed at FINISH_JOIN or FINISH_REPLACE.
+ */
+ private static Set<ShardMetadata> discoverShardsAtFinishJoinOrReplace(ClusterMetadata metadata, PlacementDeltas finishDelta, @Nullable NodeId deadNode)
+ {
+ long finishEpoch = metadata.epoch.getEpoch();
+ Set<ShardMetadata> shards = new HashSet<>();
+ for (KeyspaceMetadata ksm : metadata.schema.getKeyspaces())
+ {
+ if (!ksm.params.replicationType.isTracked())
+ continue;
+
+ DataPlacement placement = metadata.placements.get(ksm.params.replication);
+ finishDelta.get(ksm.params.replication).writes.removals.flattenValues().forEach(removed -> {
+ Set<Integer> nodeIDs = new HashSet<>();
+ for (InetAddressAndPort ep : placement.writes.forRange(removed.range()).endpoints())
+ nodeIDs.add(metadata.directory.peerId(ep).id());
+ nodeIDs.add(deadNode != null ? deadNode.id() : metadata.directory.peerId(removed.endpoint()).id());
+ Participants participants = new Participants(nodeIDs);
+
+ MutationTrackingService.instance().forEachShardInKeyspace(ksm.name, shard -> {
+ if (shard.sinceEpoch < finishEpoch
+ && shard.range.equals(removed.range())
+ && shard.participants.equals(participants)
+ && !shard.isSealed())
+ shards.add(new ShardMetadata(shard.keyspace, shard.sinceEpoch, shard.range, shard.participants));
+ });
+ });
+ }
+ return shards;
+ }
+
+ /*
+ * UNBOOTSTRAP and REMOVENODE sequences illustrated (same sequence as far as sharding is concerned).
+ * Stages in which write placements remain unchanged, and only read placements change, are skipped
+ * here as irrelevant to sharding / sealing.
+ *
+ * There is a tracked keyspace with RF=3.
+ * A node n40 with token=40 is removed in the middle of the ring.
+ *
+ * 1. Initial state at e0:
+ *
+ * (10,20] | {n20,n30,n40} | e0 | ACTIVE
+ * (20,30] | {n30,n40,n50} | e0 | ACTIVE
+ * (30,40] | {n40,n50,n60} | e0 | ACTIVE
+ * (40,50] | {n50,n60,n70} | e0 | ACTIVE
+ *
+ * 2. START_LEAVE at e1:
+ *
+ * (10,20] | {n20,n30,n40} | e0 | ACTIVE
+ * + (10,20] | {n20,n30,n40,n50} | e1 | ACTIVE
+ * (20,30] | {n30,n40,n50} | e0 | ACTIVE
+ * + (20,30] | {n30,n40,n50,n60} | e1 | ACTIVE
+ * (30,40] | {n40,n50,n60} | e0 | ACTIVE
+ * + (30,40] | {n40,n50,n60,n70} | e1 | ACTIVE
+ * (40,50] | {n50,n60,n70} | e0 | ACTIVE
+ *
+ * 3. MID_LEAVE at e2 [*]:
+ *
+ * S (10,20] | {n20,n30,n40} | e0 | SEALED
+ * (10,20] | {n20,n30,n40,n50} | e1 | ACTIVE
+ * S (20,30] | {n30,n40,n50} | e0 | SEALED
+ * (20,30] | {n30,n40,n50,n60} | e1 | ACTIVE
+ * S (30,40] | {n40,n50,n60} | e0 | SEALED
+ * (30,40] | {n40,n50,n60,n70} | e1 | ACTIVE
+ * (40,50] | {n50,n60,n70} | e0 | ACTIVE
+ *
+ * 4. FINISH_LEAVE at e3:
+ *
+ * (10,20] | {n20,n30,n40} | e0 | SEALED
+ * S (10,20] | {n20,n30,n40,n50} | e1 | SEALED
+ * + (10,20] | {n20,n30,n50} | e3 | ACTIVE
+ * (20,30] | {n30,n40,n50} | e0 | SEALED
+ * S (20,30] | {n30,n40,n50,n60} | e1 | SEALED
+ * + (20,30] | {n30,n50,n60} | e3 | ACTIVE
+ * (30,40] | {n40,n50,n60} | e0 | SEALED
+ * S (30,40] | {n40,n50,n60,n70} | e1 | SEALED
+ * S (40,50] | {n50,n60,n70} | e0 | SEALED
+ * + (30,50] | {n50,n60,n70} | e3 | ACTIVE
+ *
+ * [*] At this stage, after sealing the shards, we stream the sstables.
+ * All the offsets belonging to these sealed shards get stripped from
+ * stats component of streamed sstables. Otherwise, the new node
+ * won't be able to compact them or mark them as reconciled/repaired -
+ * it is rightfully unaware of the sealed shards.
+ */
+
+ /**
+ * Seal the shards obsoleted by START_LEAVE before streaming, for both UNBOOTSTRAP and REMOVENODE.
+ */
+ public static void sealShardsAtMidLeave(ClusterMetadata metadata, PlacementDeltas startDelta, NodeId leavingOrDeparted, LeaveStreams.Kind kind)
+ {
+ InetAddressAndPort leavingOrDepartedEndpoint = metadata.directory.endpoint(leavingOrDeparted);
+
+ List<Future<Set<ShardMetadata>>> futures = new ArrayList<>();
+ for (KeyspaceMetadata ksm : metadata.schema.getKeyspaces())
+ {
+ if (!ksm.params.replicationType.isTracked())
+ continue;
+
+ DataPlacement placement = metadata.placements.get(ksm.params.replication);
+ startDelta.get(ksm.params.replication).writes.additions.flattenValues().forEach(added -> {
+ Range<Token> range = added.range();
+ long beforeEpoch = placement.writes.forRange(range).lastModified().getEpoch();
+ Set<InetAddressAndPort> survivors = new HashSet<>(placement.writes.forRange(range).endpoints());
+ survivors.remove(leavingOrDepartedEndpoint);
+ futures.add(FetchShards.fetch(ksm.name, beforeEpoch, range, survivors));
+ });
+ }
+
+ Set<ShardMetadata> shards = new HashSet<>();
+ try
+ {
+ for (Set<ShardMetadata> fetched : FutureCombiner.allOf(futures).get(DatabaseDescriptor.getRpcTimeout(MILLISECONDS), MILLISECONDS))
+ for (ShardMetadata shard : fetched)
+ if (shard.participants.contains(leavingOrDeparted.id()))
+ shards.add(shard);
+ }
+ catch (InterruptedException | ExecutionException | TimeoutException e)
+ {
+ throw new RuntimeException("Failed to fetch shards to seal for leave", e);
+ }
+
+ seal(shards, kind == LeaveStreams.Kind.UNBOOTSTRAP ? null : leavingOrDeparted);
+ }
+
+ /**
+ * Seal the shards that need to be sealed at LEAVE_FINISH for both UNBOOTSTRAP and REMOVENODE.
+ */
+ public static void sealShardsAtFinishLeave(ClusterMetadata postFinish, PlacementDeltas finishDelta, NodeId leavingOrDeparted, LeaveStreams.Kind kind)
+ {
+ long finishEpoch = postFinish.epoch.getEpoch();
+ InetAddressAndPort leavingOrDepartedEndpoint = postFinish.directory.endpoint(leavingOrDeparted);
+
+ List<Future<Set<ShardMetadata>>> futures = new ArrayList<>();
+ for (KeyspaceMetadata ksm : postFinish.schema.getKeyspaces())
+ {
+ if (!ksm.params.replicationType.isTracked())
+ continue;
+
+ DataPlacement placement = postFinish.placements.get(ksm.params.replication);
+ RangesAtEndpoint removals = finishDelta.get(ksm.params.replication).writes.removals.get(leavingOrDepartedEndpoint);
+ for (Replica removedReplica : removals)
+ {
+ // FINISH_LEAVE merges the leavingOrDeparted node's (pre-merge) range with an adjacent one. Query the *merged*
+ // range (not the pre-merge range) so discovery covers both the leavingOrDeparted node's own obsoleted
+ // generations and the merge-half - the adjacent pre-merge range folded into the merged range, whose
+ // shard never listed the leavingOrDeparted node. matchRange() yields the containing merged group + its
+ // current (survivor) write endpoints.
+ VersionedEndpoints.ForRange group = placement.writes.matchRange(removedReplica.range());
+ Range<Token> mergedRange = group.range();
+ Set<InetAddressAndPort> survivors = new HashSet<>(group.endpoints());
+ survivors.remove(leavingOrDepartedEndpoint);
+ futures.add(FetchShards.fetch(ksm.name, finishEpoch, mergedRange, survivors));
+ }
+ }
+
+ Set<ShardMetadata> shards = new HashSet<>();
+ try
+ {
+ for (Set<ShardMetadata> fetched : FutureCombiner.allOf(futures).get(DatabaseDescriptor.getRpcTimeout(MILLISECONDS), MILLISECONDS))
+ shards.addAll(fetched);
+ }
+ catch (InterruptedException | ExecutionException | TimeoutException e)
+ {
+ throw new RuntimeException("Failed to fetch shards to seal for leave finish", e);
+ }
+
+ seal(shards, kind == LeaveStreams.Kind.UNBOOTSTRAP ? null : leavingOrDeparted);
+ }
+
+ /*
+ * Sealing logic.
+ */
+
+ private static void seal(Set<ShardMetadata> shards)
+ {
+ seal(shards, null);
+ }
+
+ private static void seal(Set<ShardMetadata> shards, @Nullable NodeId withoutNode)
+ {
+ initiate(shards, withoutNode); // ACTIVE -> SEALING for each Shard
+ drain(shards, withoutNode); // drain in-flight local writes
+ reconcile(shards, withoutNode); // wait for logs to reconcile
+ complete(shards, withoutNode); // SEALING-> SEALED for each Shard + journal flush
+ }
+
+ private static void initiate(Set<ShardMetadata> shards, @Nullable NodeId withoutNode)
+ {
+ List<Future<Void>> futures = new ArrayList<>(shards.size());
+ for (ShardMetadata shard : shards)
+ futures.add(initiate(shard, withoutNode));
+ try
+ {
+ FutureCombiner.allOf(futures).get(DatabaseDescriptor.getRpcTimeout(MILLISECONDS), MILLISECONDS);
+ }
+ catch (InterruptedException | ExecutionException | TimeoutException e)
+ {
+ throw new RuntimeException("Failed to initiate sealing", e);
+ }
+ }
+
+ /**
+ * Fence allocation on every participant of the obsoleted shard (ACTIVE -> SEALING) by marking the shard
+ * SEALING on each.
+ * TODO (expected): this assumes every participant has already enacted the topology change that obsoleted the shard (so
+ * it no longer allocates new ids on it). The ProgressBarrier gating bootstrap()/MID_JOIN only waits for
+ * EACH_QUORUM of the affected replicas, NOT all live replicas, so a live-but-lagging participant could still
+ * be on the old shard. Bringing each participant up to the seal epoch must be done per-replica (not
+ * per-shard) and will be added separately. TLDR - add a step to guaranteed up-to-speed rest of replicas epochs.
+ */
+ private static AsyncPromise<Void> initiate(ShardMetadata shard, @Nullable NodeId withoutNode)
+ {
+ return InitSealing.initiate(shard.keyspace, shard.sinceEpoch, shard.range, toEndpoints(shard.participants, withoutNode));
+ }
+
+ private static void drain(Set<ShardMetadata> shards, @Nullable NodeId withoutNode)
+ {
+ long deadlineNanos = nanoTime() + DatabaseDescriptor.getRpcTimeout(NANOSECONDS);
+ Set<ShardMetadata> pending = new HashSet<>(shards);
+ while (true)
+ {
+ // non-blocking poll: each participant reports its drain status immediately; retry until all drained
+ pending.removeIf(shard -> drain(shard, withoutNode));
+ if (pending.isEmpty())
+ return;
+ if (nanoTime() >= deadlineNanos)
+ throw new RuntimeException("Timed out draining shards; still pending: " + pending);
+ LockSupport.parkNanos(SECONDS.toNanos(1));
+ }
+ }
+
+ /**
+ * Poll every participant of the SEALING shard once for drain status; true iff all report that no mutation
+ * id allocated before the SEALING fence is still applying locally.
+ */
+ private static boolean drain(ShardMetadata shard, @Nullable NodeId withoutNode)
+ {
+ try
+ {
+ return Drain.poll(shard.keyspace, shard.sinceEpoch, shard.range, toEndpoints(shard.participants, withoutNode))
+ .get(DatabaseDescriptor.getRpcTimeout(MILLISECONDS), MILLISECONDS);
+ }
+ catch (InterruptedException | ExecutionException | TimeoutException e)
+ {
+ throw new RuntimeException(format("Failed to poll drain for shard %s", shard), e);
+ }
+ }
+
+ /**
+ * For each of the shards, capture the witnessed union then poll each participant until it has
+ * caught up and witnessed the entire offset union itself.
+ */
+ private static void reconcile(Set<ShardMetadata> shards, @Nullable NodeId withoutNode)
+ {
+ for (ShardMetadata shard : shards)
+ reconcile(shard, withoutNode);
+ }
+
+ private static void reconcile(ShardMetadata shard, @Nullable NodeId withoutNode)
+ {
+ // capture shard's witnessed offsets once
+ Log2OffsetsMap.Mutable offsets = captureWitnessedOffsets(shard, withoutNode);
+ // poll every participant until they've each witnessed the union of offsets
+ for (InetAddressAndPort endpoint : toEndpoints(shard.participants, withoutNode))
+ pollUntilWitnesses(shard, offsets, endpoint);
+ }
+
+ private static Log2OffsetsMap.Mutable captureWitnessedOffsets(ShardMetadata shard, @Nullable NodeId withoutNode)
+ {
+ List<InetAddressAndPort> endpoints = toEndpoints(shard.participants, withoutNode);
+ try
+ {
+ return ReconcileCapture.capture(shard.keyspace, shard.sinceEpoch, shard.range, endpoints)
+ .get(DatabaseDescriptor.getRpcTimeout(MILLISECONDS), MILLISECONDS);
+ }
+ catch (InterruptedException | ExecutionException | TimeoutException e)
+ {
+ throw new RuntimeException(format("Failed to capture witnessed offsets for shard %s", shard), e);
+ }
+ }
+
+ private static void pollUntilWitnesses(ShardMetadata shard, Log2OffsetsMap.Mutable offsets, InetAddressAndPort endpoint)
+ {
+ // TODO (expected): use a longer timeout
+ long deadlineNanos = nanoTime() + DatabaseDescriptor.getRpcTimeout(NANOSECONDS);
+
+ while (true)
+ {
+ if (nanoTime() >= deadlineNanos)
+ throw new RuntimeException("Timed out reconciling shard: " + shard);
+ try
+ {
+ boolean witnessed =
+ ReconcilePoll.poll(shard.keyspace, shard.sinceEpoch, shard.range, offsets, endpoint)
+ .get(DatabaseDescriptor.getRpcTimeout(MILLISECONDS), MILLISECONDS);
+ if (witnessed)
+ return;
+ }
+ catch (InterruptedException | ExecutionException | TimeoutException e)
+ {
+ throw new RuntimeException(format("Failed to poll reconcile for shard %s", shard), e);
+ }
+
+ LockSupport.parkNanos(SECONDS.toNanos(1));
+ }
+ }
+
+ private static void complete(Set<ShardMetadata> shards, @Nullable NodeId withoutNode)
+ {
+ List<Future<Void>> futures = new ArrayList<>(shards.size());
+ for (ShardMetadata shard : shards)
+ futures.add(complete(shard, withoutNode));
+
+ try
+ {
+ FutureCombiner.allOf(futures).get(DatabaseDescriptor.getRpcTimeout(MILLISECONDS), MILLISECONDS);
+ }
+ catch (InterruptedException | ExecutionException | TimeoutException e)
+ {
+ throw new RuntimeException("Failed to complete sealing", e);
+ }
+ }
+
+ /**
+ * Promote every participant of the (drained and reconciled) shard from SEALING to SEALED.
+ */
+ private static AsyncPromise<Void> complete(ShardMetadata shard, @Nullable NodeId withoutNode)
+ {
+ return CompleteSealing.complete(shard.keyspace, shard.sinceEpoch, shard.range, toEndpoints(shard.participants, withoutNode));
+ }
+
+ /**
+ * A bootstrapping node knows nothing about the shards that currently exist,
+ * so it must collect the list of shards from the existing replicas before
+ * it can proceed with the rest of the sealing steps.
+ */
+ public static final class FetchShards
+ {
+ public static final class Request
+ {
+ final String keyspace;
+ final long beforeEpoch;
+ final Range<Token> range;
+
+ Request(String keyspace, long beforeEpoch, Range<Token> range)
+ {
+ this.keyspace = keyspace;
+ this.beforeEpoch = beforeEpoch;
+ this.range = range;
+ }
+ }
+
+ public static final class Response
+ {
+ final Set<ShardMetadata> shards;
+
+ Response(Set<ShardMetadata> shards)
+ {
+ this.shards = shards;
+ }
+ }
+
+ /**
+ * Query all replicas in a group in parallel for the shards over {@code ranges} obsoleted by an
+ * in-flight topology change (those with {@code sinceEpoch < beforeEpoch}). Resembles
+ * {@link ShardMetadataRequest#queryPeers}, but waits for every (live) replica to respond and
+ * merges their deduplicated shards rather than taking the first response.
+ */
+ public static AsyncPromise<Set<ShardMetadata>> fetch(
+ String keyspace, long beforeEpoch, Range<Token> range, Set<InetAddressAndPort> endpoints)
+ {
+ Preconditions.checkArgument(!endpoints.isEmpty());
+ AsyncPromise<Set<ShardMetadata>> promise = new AsyncPromise<>();
+ Set<ShardMetadata> merged = ConcurrentHashMap.newKeySet();
+
+ RequestCallback<Response> callback = new RequestCallback<>()
+ {
+ private final AtomicInteger remaining = new AtomicInteger(endpoints.size());
+
+ @Override
+ public void onResponse(Message<Response> msg)
+ {
+ merged.addAll(msg.payload.shards);
+ if (remaining.decrementAndGet() == 0)
+ promise.trySuccess(merged);
+ }
+
+ @Override
+ public void onFailure(InetAddressAndPort from, RequestFailure failure)
+ {
+ // happy path assumes all replicas are up; fail the whole fetch if any can't respond
+ promise.tryFailure(new RuntimeException(format("Failed to fetch shards to seal from %s for keyspace %s: %s", from, keyspace, failure)));
+ }
+
+ @Override
+ public boolean invokeOnFailure()
+ {
+ return true;
+ }
+ };
+
+ Message<Request> message = Message.out(Verb.MT_FETCH_SHARDS_REQ, new Request(keyspace, beforeEpoch, range));
+ for (InetAddressAndPort peer : endpoints)
+ MessagingService.instance().sendWithCallback(message, peer, callback);
+
+ return promise;
+ }
+
+ public static final IVerbHandler<Request> verbHandler = message ->
+ {
+ MutationTrackingService.ensureEnabled();
+ Request request = message.payload;
+ Set<ShardMetadata> shards = new HashSet<>();
+ MutationTrackingService.instance().forEachShardInKeyspace(request.keyspace, shard -> {
+ if (shard.sinceEpoch < request.beforeEpoch
+ && (request.range.contains(shard.range) || shard.range.contains(request.range))
+ && !shard.isSealed())
+ shards.add(new ShardMetadata(shard.keyspace, shard.sinceEpoch, shard.range, shard.participants));
+ });
+ MessagingService.instance().send(message.responseWith(new Response(shards)), message.from());
+ };
+
+ public static final VersionedSerializer<Request> requestSerializer = new VersionedSerializer<>()
+ {
+ @Override
+ public void serialize(Request r, DataOutputPlus out, Version version) throws IOException
+ {
+ out.writeUTF(r.keyspace);
+ out.writeLong(r.beforeEpoch);
+ AbstractBounds.tokenSerializer.serialize(r.range, out, version.messagingVersion());
+ }
+
+ @Override
+ public Request deserialize(DataInputPlus in, Version version) throws IOException
+ {
+ String keyspace = in.readUTF();
+ long beforeEpoch = in.readLong();
+ Range<Token> range = (Range<Token>) AbstractBounds.tokenSerializer.deserialize(in, IPartitioner.global(), version.messagingVersion());
+ return new Request(keyspace, beforeEpoch, range);
+ }
+
+ @Override
+ public long serializedSize(Request r, Version version)
+ {
+ long size = TypeSizes.sizeof(r.keyspace);
+ size += TypeSizes.sizeof(r.beforeEpoch);
+ size += AbstractBounds.tokenSerializer.serializedSize(r.range, version.messagingVersion());
+ return size;
+ }
+ };
+
+ public static final VersionedSerializer<Response> responseSerializer = new VersionedSerializer<>()
+ {
+ @Override
+ public void serialize(Response r, DataOutputPlus out, Version version) throws IOException
+ {
+ CollectionSerializers.serializeCollection(r.shards, out, version, ShardMetadata.serializer);
+ }
+
+ @Override
+ public Response deserialize(DataInputPlus in, Version version) throws IOException
+ {
+ return new Response(CollectionSerializers.deserializeSet(in, version, ShardMetadata.serializer));
+ }
+
+ @Override
+ public long serializedSize(Response r, Version version)
+ {
+ return CollectionSerializers.serializedCollectionSize(r.shards, version, ShardMetadata.serializer);
+ }
+ };
+ }
+
+ public static final class InitSealing
+ {
+ public static final class Request
+ {
+ final String keyspace;
+ final long sinceEpoch;
+ final Range<Token> range;
+
+ Request(String keyspace, long sinceEpoch, Range<Token> range)
+ {
+ this.keyspace = keyspace;
+ this.sinceEpoch = sinceEpoch;
+ this.range = range;
+ }
+ }
+
+ // cannot use NoPayload, because its serializer cannot be wrapped inside mtEmbedded()
+ public static final class Response
+ {
+ private static final Response instance = new Response();
+ }
+
+ /**
+ * Tell every participant of an obsoleted shard to fence id allocation (mark it SEALING)
+ */
+ public static AsyncPromise<Void> initiate(
+ String keyspace, long sinceEpoch, Range<Token> range, List<InetAddressAndPort> endpoints)
+ {
+ AsyncPromise<Void> promise = new AsyncPromise<>();
+
+ RequestCallback<Response> callback = new RequestCallback<>()
+ {
+ private final AtomicInteger remaining = new AtomicInteger(endpoints.size());
+
+ @Override
+ public void onResponse(Message<Response> msg)
+ {
+ if (remaining.decrementAndGet() == 0)
+ promise.trySuccess(null);
+ }
+
+ @Override
+ public void onFailure(InetAddressAndPort from, RequestFailure failure)
+ {
+ promise.tryFailure(new RuntimeException(format("Failed to initiate sealing on %s for keyspace %s: %s", from, keyspace, failure)));
+ }
+
+ @Override
+ public boolean invokeOnFailure()
+ {
+ return true;
+ }
+ };
+
+ Message<Request> message = Message.out(Verb.MT_INIT_SEALING_REQ, new Request(keyspace, sinceEpoch, range));
+ for (InetAddressAndPort peer : endpoints)
+ MessagingService.instance().sendWithCallback(message, peer, callback);
+
+ return promise;
+ }
+
+ public static final IVerbHandler<Request> verbHandler = message -> {
+ MutationTrackingService.ensureEnabled();
+ Request request = message.payload;
+ MutationTrackingService.instance().markShardSealing(request.keyspace, request.sinceEpoch, request.range);
+ MessagingService.instance().send(message.responseWith(Response.instance), message.from());
+ };
+
+ public static final VersionedSerializer<Request> requestSerializer = new VersionedSerializer<>()
+ {
+ @Override
+ public void serialize(Request r, DataOutputPlus out, Version version) throws IOException
+ {
+ out.writeUTF(r.keyspace);
+ out.writeLong(r.sinceEpoch);
+ AbstractBounds.tokenSerializer.serialize(r.range, out, version.messagingVersion());
+ }
+
+ @Override
+ public Request deserialize(DataInputPlus in, Version version) throws IOException
+ {
+ String keyspace = in.readUTF();
+ long sinceEpoch = in.readLong();
+ Range<Token> range = (Range<Token>) AbstractBounds.tokenSerializer.deserialize(in, IPartitioner.global(), version.messagingVersion());
+ return new Request(keyspace, sinceEpoch, range);
+ }
+
+ @Override
+ public long serializedSize(Request r, Version version)
+ {
+ long size = TypeSizes.sizeof(r.keyspace);
+ size += TypeSizes.sizeof(r.sinceEpoch);
+ size += AbstractBounds.tokenSerializer.serializedSize(r.range, version.messagingVersion());
+ return size;
+ }
+ };
+
+ public static final UnversionedSerializer<Response> responseSerializer = new UnversionedSerializer<>()
+ {
+ @Override
+ public void serialize(Response r, DataOutputPlus out)
+ {
+ }
+
+ @Override
+ public Response deserialize(DataInputPlus in)
+ {
+ return Response.instance;
+ }
+
+ @Override
+ public long serializedSize(Response r)
+ {
+ return 0;
+ }
+ };
+ }
+
+ public static final class Drain
+ {
+ public static final class Request
+ {
+ final String keyspace;
+ final long sinceEpoch;
+ final Range<Token> range;
+
+ Request(String keyspace, long sinceEpoch, Range<Token> range)
+ {
+ this.keyspace = keyspace;
+ this.sinceEpoch = sinceEpoch;
+ this.range = range;
+ }
+ }
+
+ public static final class Response
+ {
+ final boolean drained;
+
+ Response(boolean drained)
+ {
+ this.drained = drained;
+ }
+ }
+
+ /**
+ * Poll every participant of a SEALING shard for whether it has drained its in-flight local
+ * writes. Succeeds with {@code true} iff every participant reports drained.
+ */
+ public static AsyncPromise<Boolean> poll(
+ String keyspace, long sinceEpoch, Range<Token> range, List<InetAddressAndPort> endpoints)
+ {
+ AsyncPromise<Boolean> promise = new AsyncPromise<>();
+
+ RequestCallback<Response> callback = new RequestCallback<>()
+ {
+ private final AtomicInteger remaining = new AtomicInteger(endpoints.size());
+ private volatile boolean allDrained = true;
+
+ @Override
+ public void onResponse(Message<Response> msg)
+ {
+ if (!msg.payload.drained)
+ allDrained = false;
+ if (remaining.decrementAndGet() == 0)
+ promise.trySuccess(allDrained);
+ }
+
+ @Override
+ public void onFailure(InetAddressAndPort from, RequestFailure failure)
+ {
+ promise.tryFailure(new RuntimeException(format("Failed to poll drain on %s for keyspace %s: %s", from, keyspace, failure)));
+ }
+
+ @Override
+ public boolean invokeOnFailure()
+ {
+ return true;
+ }
+ };
+
+ Message<Request> message = Message.out(Verb.MT_DRAIN_REQ, new Request(keyspace, sinceEpoch, range));
+ for (InetAddressAndPort peer : endpoints)
+ MessagingService.instance().sendWithCallback(message, peer, callback);
+
+ return promise;
+ }
+
+ public static final IVerbHandler<Request> verbHandler = message -> {
+ MutationTrackingService.ensureEnabled();
+ Request request = message.payload;
+ boolean drained = MutationTrackingService.instance().isShardDrained(request.keyspace, request.sinceEpoch, request.range);
+ MessagingService.instance().send(message.responseWith(new Response(drained)), message.from());
+ };
+
+ public static final VersionedSerializer<Request> requestSerializer = new VersionedSerializer<>()
+ {
+ @Override
+ public void serialize(Request r, DataOutputPlus out, Version version) throws IOException
+ {
+ out.writeUTF(r.keyspace);
+ out.writeLong(r.sinceEpoch);
+ AbstractBounds.tokenSerializer.serialize(r.range, out, version.messagingVersion());
+ }
+
+ @Override
+ public Request deserialize(DataInputPlus in, Version version) throws IOException
+ {
+ String keyspace = in.readUTF();
+ long sinceEpoch = in.readLong();
+ Range<Token> range = (Range<Token>) AbstractBounds.tokenSerializer.deserialize(in, IPartitioner.global(), version.messagingVersion());
+ return new Request(keyspace, sinceEpoch, range);
+ }
+
+ @Override
+ public long serializedSize(Request r, Version version)
+ {
+ long size = TypeSizes.sizeof(r.keyspace);
+ size += TypeSizes.sizeof(r.sinceEpoch);
+ size += AbstractBounds.tokenSerializer.serializedSize(r.range, version.messagingVersion());
+ return size;
+ }
+ };
+
+ public static final UnversionedSerializer<Response> responseSerializer = new UnversionedSerializer<>()
+ {
+ @Override
+ public void serialize(Response r, DataOutputPlus out) throws IOException
+ {
+ out.writeBoolean(r.drained);
+ }
+
+ @Override
+ public Response deserialize(DataInputPlus in) throws IOException
+ {
+ return new Response(in.readBoolean());
+ }
+
+ @Override
+ public long serializedSize(Response r)
+ {
+ return TypeSizes.sizeof(r.drained);
+ }
+ };
+ }
+
+ public static final class ReconcileCapture
+ {
+ public static final class Request
+ {
+ final String keyspace;
+ final long sinceEpoch;
+ final Range<Token> range;
+
+ Request(String keyspace, long sinceEpoch, Range<Token> range)
+ {
+ this.keyspace = keyspace;
+ this.sinceEpoch = sinceEpoch;
+ this.range = range;
+ }
+ }
+
+ public static final class Response
+ {
+ final Log2OffsetsMap<?> witnessed;
+
+ Response(Log2OffsetsMap<?> witnessed)
+ {
+ this.witnessed = witnessed;
+ }
+ }
+
+ public static AsyncPromise<Log2OffsetsMap.Mutable> capture(
+ String keyspace, long sinceEpoch, Range<Token> range, List<InetAddressAndPort> endpoints)
+ {
+ AsyncPromise<Log2OffsetsMap.Mutable> promise = new AsyncPromise<>();
+ Log2OffsetsMap.Mutable union = new Log2OffsetsMap.Mutable();
+
+ RequestCallback<Response> callback = new RequestCallback<>()
+ {
+ private final AtomicInteger remaining = new AtomicInteger(endpoints.size());
+
+ @Override
+ public void onResponse(Message<Response> msg)
+ {
+ synchronized (union)
+ {
+ union.addAll(msg.payload.witnessed);
+ }
+ if (remaining.decrementAndGet() == 0)
+ promise.trySuccess(union);
+ }
+
+ @Override
+ public void onFailure(InetAddressAndPort from, RequestFailure failure)
+ {
+ promise.tryFailure(new RuntimeException(format("Failed to capture witnessed offsets on %s for keyspace %s: %s", from, keyspace, failure)));
+ }
+
+ @Override
+ public boolean invokeOnFailure()
+ {
+ return true;
+ }
+ };
+
+ Message<Request> message = Message.out(Verb.MT_RECONCILE_CAPTURE_REQ, new Request(keyspace, sinceEpoch, range));
+ for (InetAddressAndPort peer : endpoints)
+ MessagingService.instance().sendWithCallback(message, peer, callback);
+
+ return promise;
+ }
+
+ public static final IVerbHandler<Request> verbHandler = message -> {
+ MutationTrackingService.ensureEnabled();
+ Request request = message.payload;
+ Log2OffsetsMap<?> witnessed =
+ MutationTrackingService.instance()
+ .collectLocallyWitnessedOffsets(request.keyspace, request.sinceEpoch, request.range);
Review Comment:
I think we'd want to capture all offsets from all shards with a `sinceEpoch <= request.sinceEpoch` here? It may not be technically unneccesary, but it seems like a reasonable hedge against racing shard seal operations.
##########
src/java/org/apache/cassandra/replication/TrackedImportTransfer.java:
##########
@@ -92,14 +92,14 @@ public class TrackedImportTransfer extends CoordinatedTransfer
@VisibleForTesting
TrackedImportTransfer(Range<Token> range, MutationId id)
{
- super(id, null, range);
+ super(id, null, 0L, range);
Review Comment:
this class has toString/equals/hashCode that should to be updated to include this (though the base class does not)
##########
src/java/org/apache/cassandra/replication/MutationTrackingService.java:
##########
@@ -950,7 +988,6 @@ private synchronized void onNewClusterMetadata(@Nullable ClusterMetadata prev, C
}
catch (Throwable t)
{
- log2ShardMap = originalLog2ShardMap;
Review Comment:
removing log2ShardMap rollback like this will cause the contents of log2ShardMap and keyspaceShards to get out of sync if an exception is thrown in `applyUpdatedMetadata`
##########
src/java/org/apache/cassandra/replication/SealingCoordinator.java:
##########
@@ -0,0 +1,1401 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.cassandra.replication;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.TimeoutException;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.concurrent.locks.LockSupport;
+
+import javax.annotation.Nonnull;
+import javax.annotation.Nullable;
+
+import com.google.common.base.Preconditions;
+
+import org.apache.cassandra.config.DatabaseDescriptor;
+import org.apache.cassandra.db.ColumnFamilyStore;
+import org.apache.cassandra.db.Keyspace;
+import org.apache.cassandra.db.TypeSizes;
+import org.apache.cassandra.dht.AbstractBounds;
+import org.apache.cassandra.dht.IPartitioner;
+import org.apache.cassandra.dht.Range;
+import org.apache.cassandra.dht.Token;
+import org.apache.cassandra.exceptions.RequestFailure;
+import org.apache.cassandra.io.UnversionedSerializer;
+import org.apache.cassandra.io.util.DataInputPlus;
+import org.apache.cassandra.io.util.DataOutputPlus;
+import org.apache.cassandra.locator.EndpointsByReplica;
+import org.apache.cassandra.locator.EndpointsForRange;
+import org.apache.cassandra.locator.InetAddressAndPort;
+import org.apache.cassandra.locator.RangesAtEndpoint;
+import org.apache.cassandra.locator.Replica;
+import org.apache.cassandra.net.IVerbHandler;
+import org.apache.cassandra.net.Message;
+import org.apache.cassandra.net.MessagingService;
+import org.apache.cassandra.net.RequestCallback;
+import org.apache.cassandra.net.Verb;
+import org.apache.cassandra.schema.KeyspaceMetadata;
+import org.apache.cassandra.tcm.ClusterMetadata;
+import org.apache.cassandra.tcm.membership.NodeId;
+import org.apache.cassandra.tcm.ownership.DataPlacement;
+import org.apache.cassandra.tcm.ownership.MovementMap;
+import org.apache.cassandra.tcm.ownership.PlacementDeltas;
+import org.apache.cassandra.tcm.ownership.VersionedEndpoints;
+import org.apache.cassandra.tcm.sequences.LeaveStreams;
+import org.apache.cassandra.utils.CollectionSerializers;
+import org.apache.cassandra.utils.FBUtilities;
+import org.apache.cassandra.utils.concurrent.AsyncPromise;
+import org.apache.cassandra.utils.concurrent.Future;
+import org.apache.cassandra.utils.concurrent.FutureCombiner;
+
+import static java.lang.String.format;
+import static java.util.concurrent.TimeUnit.MILLISECONDS;
+import static java.util.concurrent.TimeUnit.NANOSECONDS;
+import static java.util.concurrent.TimeUnit.SECONDS;
+import static org.apache.cassandra.utils.Clock.Global.nanoTime;
+
+/*
+ * Sealing {@link Shard}s once a topology change obsoletes them.
+ *
+ * Happy Path Sequence:
+ * 1. Bring every participant up to (at least) this seal's epoch and fence allocation, so that everyone
+ * stops writing to the old shards; then wait until all in-progress mutations have been done.
+ * 2. Same for tracked transfers
+ * 3. Wait until all participating replicas have reconciled the union of their witnessed ids
+ * 4. Mark the shard as fully sealed
+ * 5. Stop including the shard in mutation summaries for read requests.
+ *
+ * Unhappy Path Sequence (not yet implemented):
+ * 1. Wait until everyone who's up stops writing to the old shards and all in-progress mutations have been done
+ * 2. Same for tracked transfers
+ * 3. Wait until all *live* participating replicas have reconciled the union of their witnessed ids
+ * 4. Mark the shard as partially sealed, label it with the set of participants that were able to seal
+ * 5. When a down node comes up, have it talk to other nodes and learn about sealing, and what mutations/logs
+ * they have exchanged. Diff with what we have - in system tables and in the journal (though system table
+ * plus local replay should give us an up-to-date metadata state for the shard)
+ */
+public final class SealingCoordinator
+{
+ /*
+ * BOOTSTRAP sequence illustrated. Stages in which write placements
+ * remain unchanged, and only read placements change, are skipped
+ * here as irrelevant to sharding / sealing.
+ *
+ * There is a tracked keyspace with RF=3.
+ * A node n35 with token=35 is added in the middle of the ring.
+ *
+ * 1. Initial state at e0:
+ *
+ * (10,20] | {n20,n30,n40} | e0 | ACTIVE
+ * (20,30] | {n30,n40,n50} | e0 | ACTIVE
+ * (30,40] | {n40,n50,n60} | e0 | ACTIVE
+ *
+ * 2. PREPARE_JOIN at e1:
+ *
+ * (10,20] | {n20,n30,n40} | e0 | ACTIVE
+ * (20,30] | {n30,n40,n50} | e0 | ACTIVE
+ * (30,40] | {n40,n50,n60} | e0 | ACTIVE
+ * + (30,35] | {n40,n50,n60} | e1 | ACTIVE
+ * + (35,40] | {n40,n50,n60} | e1 | ACTIVE
+ *
+ * 3. START_JOIN at e2:
+ *
+ * (10,20] | {n20,n30,n40} | e0 | ACTIVE
+ * + (10,20] | {n20,n30,n35,n40} | e2 | ACTIVE
+ * (20,30] | {n30,n40,n50} | e0 | ACTIVE
+ * + (20,30] | {n30,n35,n40,n50} | e2 | ACTIVE
+ * (30,40] | {n40,n50,n60} | e0 | ACTIVE
+ * (30,35] | {n40,n50,n60} | e1 | ACTIVE
+ * + (30,35] | {n35,n40,n50,n60} | e2 | ACTIVE
+ * (35,40] | {n40,n50,n60} | e1 | ACTIVE
+ *
+ * 4. MID_JOIN at e3 [*]:
+ *
+ * S (10,20] | {n20,n30,n40} | e0 | SEALED
+ * (10,20] | {n20,n30,n35,n40} | e2 | ACTIVE
+ * S (20,30] | {n30,n40,n50} | e0 | SEALED
+ * (20,30] | {n30,n35,n40,n50} | e2 | ACTIVE
+ * S (30,40] | {n40,n50,n60} | e0 | SEALED
+ * S (30,35] | {n40,n50,n60} | e1 | SEALED
+ * (30,35] | {n35,n40,n50,n60} | e2 | ACTIVE
+ * (35,40] | {n40,n50,n60} | e1 | ACTIVE
+ *
+ * 5. FINISH_JOIN at e4:
+ *
+ * (10,20] | {n20,n30,n40} | e0 | SEALED
+ * S (10,20] | {n20,n30,n35,n40} | e2 | SEALED
+ * + (10,20] | {n20,n30,n35} | e4 | ACTIVE
+ * (20,30] | {n30,n40,n50} | e0 | SEALED
+ * S (20,30] | {n30,n35,n40,n50} | e2 | SEALED
+ * + (20,30] | {n30,n35,n40} | e4 | ACTIVE
+ * (30,40] | {n40,n50,n60} | e0 | SEALED
+ * (30,35] | {n40,n50,n60} | e1 | SEALED
+ * S (30,35] | {n35,n40,n50,n60} | e2 | SEALED
+ * + (30,35] | {n35,n40,n50} | e4 | ACTIVE
+ * (35,40] | {n40,n50,n60} | e1 | ACTIVE
+ *
+ * [*] At this stage, after sealing the shards, we stream the sstables.
+ * All the offsets belonging to these sealed shards get stripped from
+ * stats component of streamed sstables. Otherwise, the new node
+ * won't be able to compact them or mark them as reconciled/repaired -
+ * it is rightfully unaware of the sealed shards.
+ */
+
+ /**
+ * Seal the shards obsoleted by bootstrap before the new node starts streaming.
+ */
+ public static void sealShardsAtMidJoin(ClusterMetadata metadata, MovementMap movements)
+ {
+ seal(discoverShardsAtMidJoinOrReplace(metadata, movements));
+ }
+
+ /**
+ * Seal the intermediate shards that were created during START_JOIN.
+ * TODO (required): this MUST eventually run while the affected ranges are still locked - between the
+ * removeFromWrites placement change and the range unlock. FinishJoin/FinishReplace currently apply
+ * removeFromWrites and the unlock atomically in a single transformation, so there is no committed
+ * dropped-but-locked epoch to hook. Until then, this runs just after FINISH_JOIN/FINISH_REPLACE,
+ * when the ranges are already unlocked. Left to implement for the unhappy path (AY).
+ */
+ public static void sealShardsAtFinishJoin(ClusterMetadata metadata, PlacementDeltas finishDelta)
+ {
+ seal(discoverShardsAtFinishJoinOrReplace(metadata, finishDelta, null));
+ }
+
+ /*
+ * REPLACE sequence illustrated. Stages in which write placements
+ * remain unchanged, and only read placements change, are skipped
+ * here as irrelevant to sharding / sealing.
+ *
+ * There is a tracked keyspace with RF=3.
+ * Node n40 is dead, and is being replaced with $n40.
+ *
+ * 1. Initial state at e0:
+ *
+ * (10,20] | {n20,n30,n40} | e0 | ACTIVE
+ * (20,30] | {n30,n40,n50} | e0 | ACTIVE
+ * (30,40] | {n40,n50,n60} | e0 | ACTIVE
+ *
+ * 2. START_REPLACE at e1:
+ *
+ * (10,20] | {n20,n30,n40} | e0 | ACTIVE
+ * + (10,20] | {n20,n30,n40,$n40}| e1 | ACTIVE
+ * (20,30] | {n30,n40,n50} | e0 | ACTIVE
+ * + (20,30] | {n30,n40,$n40,n50}| e1 | ACTIVE
+ * (30,40] | {n40,n50,n60} | e0 | ACTIVE
+ * + (30,40] | {n40,$n40,n50,n60}| e1 | ACTIVE
+ *
+ * 3. MID_REPLACE at e2 [*]:
+ *
+ * S (10,20] | {n20,n30,n40} | e0 | SEALED
+ * (10,20] | {n20,n30,n40,$n40}| e1 | ACTIVE
+ * S (20,30] | {n30,n40,n50} | e0 | SEALED
+ * (20,30] | {n30,n40,$n40,n50}| e1 | ACTIVE
+ * S (30,40] | {n40,n50,n60} | e0 | SEALED
+ * (30,40] | {n40,$n40,n50,n60}| e1 | ACTIVE
+ *
+ * 4. FINISH_REPLACE at e3:
+ *
+ * (10,20] | {n20,n30,n40} | e0 | SEALED
+ * S (10,20] | {n20,n30,n40,$n40}| e1 | SEALED
+ * + (10,20] | {n20,n30,$n40} | e3 | ACTIVE
+ * (20,30] | {n30,n40,n50} | e0 | SEALED
+ * S (20,30] | {n30,n40,$n40,n50}| e1 | SEALED
+ * + (20,30] | {n30,$n40,n50} | e3 | ACTIVE
+ * (30,40] | {n40,n50,n60} | e0 | SEALED
+ * S (30,40] | {n40,$n40,n50,n60}| e1 | SEALED
+ * + (30,40] | {$n40,n50,n60} | e3 | ACTIVE
+ *
+ * [*] At this stage, after sealing the shards, we stream the sstables.
+ * All the offsets belonging to these sealed shards get stripped from
+ * stats component of streamed sstables. Otherwise, the new node
+ * won't be able to compact them or mark them as reconciled/repaired -
+ * it is rightfully unaware of the sealed shards.
+ */
+
+ /**
+ * Seal the shards obsoleted by host replacement before the replacement node starts streaming.
+ * The {@code deadNode} node cannot participate in the sealing.
+ */
+ public static void sealShardsAtMidReplace(ClusterMetadata metadata, MovementMap movements, @Nonnull NodeId deadNode)
+ {
+ seal(discoverShardsAtMidJoinOrReplace(metadata, movements), deadNode);
+ }
+
+ public static void sealShardsAtFinishReplace(ClusterMetadata metadata, PlacementDeltas finishDelta, @Nonnull NodeId deadNode)
+ {
+ seal(discoverShardsAtFinishJoinOrReplace(metadata, finishDelta, deadNode), deadNode);
+ }
+
+ /**
+ * Discover all shards that need to be sealed at MID_JOIN or MID_REPLACE.
+ */
+ private static Set<ShardMetadata> discoverShardsAtMidJoinOrReplace(ClusterMetadata metadata, MovementMap movements)
+ {
+ List<Future<Set<ShardMetadata>>> futures = new ArrayList<>();
+
+ for (KeyspaceMetadata ksm : metadata.schema.getKeyspaces())
+ {
+ if (!ksm.params.replicationType.isTracked())
+ continue;
+
+ EndpointsByReplica acquired = movements.get(ksm.params.replication);
+ if (acquired.isEmpty()) // when the keyspace is not replicated to this DC
+ continue;
+
+ DataPlacement placement = metadata.placements.get(ksm.params.replication);
+
+ for (Map.Entry<Replica, EndpointsForRange> entry : acquired.entrySet())
+ {
+ Range<Token> range = entry.getKey().range();
+ Set<InetAddressAndPort> endpoints = entry.getValue().endpoints();
+ // cutoff: the epoch this node was added to the write placement for the range;
+ // shards over the range with sinceEpoch < cutoff are obsoleted by this bootstrap or replace.
+ long beforeEpoch = placement.writes.forRange(range).lastModified().getEpoch();
+ futures.add(FetchShards.fetch(ksm.name, beforeEpoch, range, endpoints));
+ }
+ }
+
+ Set<ShardMetadata> shards = new HashSet<>();
+ try
+ {
+ for (Set<ShardMetadata> fetched : FutureCombiner.allOf(futures).get(DatabaseDescriptor.getRpcTimeout(MILLISECONDS), MILLISECONDS))
+ shards.addAll(fetched);
+ }
+ catch (InterruptedException | ExecutionException | TimeoutException e)
+ {
+ throw new RuntimeException("Failed to fetch shards to seal", e);
+ }
+ return shards;
+ }
+
+ /**
+ * Discover all shards that need to be sealed at FINISH_JOIN or FINISH_REPLACE.
+ */
+ private static Set<ShardMetadata> discoverShardsAtFinishJoinOrReplace(ClusterMetadata metadata, PlacementDeltas finishDelta, @Nullable NodeId deadNode)
+ {
+ long finishEpoch = metadata.epoch.getEpoch();
+ Set<ShardMetadata> shards = new HashSet<>();
+ for (KeyspaceMetadata ksm : metadata.schema.getKeyspaces())
+ {
+ if (!ksm.params.replicationType.isTracked())
+ continue;
+
+ DataPlacement placement = metadata.placements.get(ksm.params.replication);
+ finishDelta.get(ksm.params.replication).writes.removals.flattenValues().forEach(removed -> {
+ Set<Integer> nodeIDs = new HashSet<>();
+ for (InetAddressAndPort ep : placement.writes.forRange(removed.range()).endpoints())
+ nodeIDs.add(metadata.directory.peerId(ep).id());
+ nodeIDs.add(deadNode != null ? deadNode.id() : metadata.directory.peerId(removed.endpoint()).id());
+ Participants participants = new Participants(nodeIDs);
+
+ MutationTrackingService.instance().forEachShardInKeyspace(ksm.name, shard -> {
+ if (shard.sinceEpoch < finishEpoch
+ && shard.range.equals(removed.range())
+ && shard.participants.equals(participants)
+ && !shard.isSealed())
+ shards.add(new ShardMetadata(shard.keyspace, shard.sinceEpoch, shard.range, shard.participants));
+ });
+ });
+ }
+ return shards;
+ }
+
+ /*
+ * UNBOOTSTRAP and REMOVENODE sequences illustrated (same sequence as far as sharding is concerned).
+ * Stages in which write placements remain unchanged, and only read placements change, are skipped
+ * here as irrelevant to sharding / sealing.
+ *
+ * There is a tracked keyspace with RF=3.
+ * A node n40 with token=40 is removed in the middle of the ring.
+ *
+ * 1. Initial state at e0:
+ *
+ * (10,20] | {n20,n30,n40} | e0 | ACTIVE
+ * (20,30] | {n30,n40,n50} | e0 | ACTIVE
+ * (30,40] | {n40,n50,n60} | e0 | ACTIVE
+ * (40,50] | {n50,n60,n70} | e0 | ACTIVE
+ *
+ * 2. START_LEAVE at e1:
+ *
+ * (10,20] | {n20,n30,n40} | e0 | ACTIVE
+ * + (10,20] | {n20,n30,n40,n50} | e1 | ACTIVE
+ * (20,30] | {n30,n40,n50} | e0 | ACTIVE
+ * + (20,30] | {n30,n40,n50,n60} | e1 | ACTIVE
+ * (30,40] | {n40,n50,n60} | e0 | ACTIVE
+ * + (30,40] | {n40,n50,n60,n70} | e1 | ACTIVE
+ * (40,50] | {n50,n60,n70} | e0 | ACTIVE
+ *
+ * 3. MID_LEAVE at e2 [*]:
+ *
+ * S (10,20] | {n20,n30,n40} | e0 | SEALED
+ * (10,20] | {n20,n30,n40,n50} | e1 | ACTIVE
+ * S (20,30] | {n30,n40,n50} | e0 | SEALED
+ * (20,30] | {n30,n40,n50,n60} | e1 | ACTIVE
+ * S (30,40] | {n40,n50,n60} | e0 | SEALED
+ * (30,40] | {n40,n50,n60,n70} | e1 | ACTIVE
+ * (40,50] | {n50,n60,n70} | e0 | ACTIVE
+ *
+ * 4. FINISH_LEAVE at e3:
+ *
+ * (10,20] | {n20,n30,n40} | e0 | SEALED
+ * S (10,20] | {n20,n30,n40,n50} | e1 | SEALED
+ * + (10,20] | {n20,n30,n50} | e3 | ACTIVE
+ * (20,30] | {n30,n40,n50} | e0 | SEALED
+ * S (20,30] | {n30,n40,n50,n60} | e1 | SEALED
+ * + (20,30] | {n30,n50,n60} | e3 | ACTIVE
+ * (30,40] | {n40,n50,n60} | e0 | SEALED
+ * S (30,40] | {n40,n50,n60,n70} | e1 | SEALED
+ * S (40,50] | {n50,n60,n70} | e0 | SEALED
+ * + (30,50] | {n50,n60,n70} | e3 | ACTIVE
+ *
+ * [*] At this stage, after sealing the shards, we stream the sstables.
+ * All the offsets belonging to these sealed shards get stripped from
+ * stats component of streamed sstables. Otherwise, the new node
+ * won't be able to compact them or mark them as reconciled/repaired -
+ * it is rightfully unaware of the sealed shards.
+ */
+
+ /**
+ * Seal the shards obsoleted by START_LEAVE before streaming, for both UNBOOTSTRAP and REMOVENODE.
+ */
+ public static void sealShardsAtMidLeave(ClusterMetadata metadata, PlacementDeltas startDelta, NodeId leavingOrDeparted, LeaveStreams.Kind kind)
+ {
+ InetAddressAndPort leavingOrDepartedEndpoint = metadata.directory.endpoint(leavingOrDeparted);
+
+ List<Future<Set<ShardMetadata>>> futures = new ArrayList<>();
+ for (KeyspaceMetadata ksm : metadata.schema.getKeyspaces())
+ {
+ if (!ksm.params.replicationType.isTracked())
+ continue;
+
+ DataPlacement placement = metadata.placements.get(ksm.params.replication);
+ startDelta.get(ksm.params.replication).writes.additions.flattenValues().forEach(added -> {
+ Range<Token> range = added.range();
+ long beforeEpoch = placement.writes.forRange(range).lastModified().getEpoch();
+ Set<InetAddressAndPort> survivors = new HashSet<>(placement.writes.forRange(range).endpoints());
+ survivors.remove(leavingOrDepartedEndpoint);
+ futures.add(FetchShards.fetch(ksm.name, beforeEpoch, range, survivors));
+ });
+ }
+
+ Set<ShardMetadata> shards = new HashSet<>();
+ try
+ {
+ for (Set<ShardMetadata> fetched : FutureCombiner.allOf(futures).get(DatabaseDescriptor.getRpcTimeout(MILLISECONDS), MILLISECONDS))
+ for (ShardMetadata shard : fetched)
+ if (shard.participants.contains(leavingOrDeparted.id()))
+ shards.add(shard);
+ }
+ catch (InterruptedException | ExecutionException | TimeoutException e)
+ {
+ throw new RuntimeException("Failed to fetch shards to seal for leave", e);
+ }
+
+ seal(shards, kind == LeaveStreams.Kind.UNBOOTSTRAP ? null : leavingOrDeparted);
+ }
+
+ /**
+ * Seal the shards that need to be sealed at LEAVE_FINISH for both UNBOOTSTRAP and REMOVENODE.
+ */
+ public static void sealShardsAtFinishLeave(ClusterMetadata postFinish, PlacementDeltas finishDelta, NodeId leavingOrDeparted, LeaveStreams.Kind kind)
+ {
+ long finishEpoch = postFinish.epoch.getEpoch();
+ InetAddressAndPort leavingOrDepartedEndpoint = postFinish.directory.endpoint(leavingOrDeparted);
+
+ List<Future<Set<ShardMetadata>>> futures = new ArrayList<>();
+ for (KeyspaceMetadata ksm : postFinish.schema.getKeyspaces())
+ {
+ if (!ksm.params.replicationType.isTracked())
+ continue;
+
+ DataPlacement placement = postFinish.placements.get(ksm.params.replication);
+ RangesAtEndpoint removals = finishDelta.get(ksm.params.replication).writes.removals.get(leavingOrDepartedEndpoint);
+ for (Replica removedReplica : removals)
+ {
+ // FINISH_LEAVE merges the leavingOrDeparted node's (pre-merge) range with an adjacent one. Query the *merged*
+ // range (not the pre-merge range) so discovery covers both the leavingOrDeparted node's own obsoleted
+ // generations and the merge-half - the adjacent pre-merge range folded into the merged range, whose
+ // shard never listed the leavingOrDeparted node. matchRange() yields the containing merged group + its
+ // current (survivor) write endpoints.
+ VersionedEndpoints.ForRange group = placement.writes.matchRange(removedReplica.range());
+ Range<Token> mergedRange = group.range();
+ Set<InetAddressAndPort> survivors = new HashSet<>(group.endpoints());
+ survivors.remove(leavingOrDepartedEndpoint);
+ futures.add(FetchShards.fetch(ksm.name, finishEpoch, mergedRange, survivors));
+ }
+ }
+
+ Set<ShardMetadata> shards = new HashSet<>();
+ try
+ {
+ for (Set<ShardMetadata> fetched : FutureCombiner.allOf(futures).get(DatabaseDescriptor.getRpcTimeout(MILLISECONDS), MILLISECONDS))
+ shards.addAll(fetched);
+ }
+ catch (InterruptedException | ExecutionException | TimeoutException e)
+ {
+ throw new RuntimeException("Failed to fetch shards to seal for leave finish", e);
+ }
+
+ seal(shards, kind == LeaveStreams.Kind.UNBOOTSTRAP ? null : leavingOrDeparted);
+ }
+
+ /*
+ * Sealing logic.
+ */
+
+ private static void seal(Set<ShardMetadata> shards)
+ {
+ seal(shards, null);
+ }
+
+ private static void seal(Set<ShardMetadata> shards, @Nullable NodeId withoutNode)
+ {
+ initiate(shards, withoutNode); // ACTIVE -> SEALING for each Shard
+ drain(shards, withoutNode); // drain in-flight local writes
+ reconcile(shards, withoutNode); // wait for logs to reconcile
+ complete(shards, withoutNode); // SEALING-> SEALED for each Shard + journal flush
+ }
+
+ private static void initiate(Set<ShardMetadata> shards, @Nullable NodeId withoutNode)
+ {
+ List<Future<Void>> futures = new ArrayList<>(shards.size());
+ for (ShardMetadata shard : shards)
+ futures.add(initiate(shard, withoutNode));
+ try
+ {
+ FutureCombiner.allOf(futures).get(DatabaseDescriptor.getRpcTimeout(MILLISECONDS), MILLISECONDS);
+ }
+ catch (InterruptedException | ExecutionException | TimeoutException e)
+ {
+ throw new RuntimeException("Failed to initiate sealing", e);
+ }
+ }
+
+ /**
+ * Fence allocation on every participant of the obsoleted shard (ACTIVE -> SEALING) by marking the shard
+ * SEALING on each.
+ * TODO (expected): this assumes every participant has already enacted the topology change that obsoleted the shard (so
+ * it no longer allocates new ids on it). The ProgressBarrier gating bootstrap()/MID_JOIN only waits for
+ * EACH_QUORUM of the affected replicas, NOT all live replicas, so a live-but-lagging participant could still
+ * be on the old shard. Bringing each participant up to the seal epoch must be done per-replica (not
+ * per-shard) and will be added separately. TLDR - add a step to guaranteed up-to-speed rest of replicas epochs.
+ */
+ private static AsyncPromise<Void> initiate(ShardMetadata shard, @Nullable NodeId withoutNode)
+ {
+ return InitSealing.initiate(shard.keyspace, shard.sinceEpoch, shard.range, toEndpoints(shard.participants, withoutNode));
+ }
+
+ private static void drain(Set<ShardMetadata> shards, @Nullable NodeId withoutNode)
+ {
+ long deadlineNanos = nanoTime() + DatabaseDescriptor.getRpcTimeout(NANOSECONDS);
+ Set<ShardMetadata> pending = new HashSet<>(shards);
+ while (true)
+ {
+ // non-blocking poll: each participant reports its drain status immediately; retry until all drained
+ pending.removeIf(shard -> drain(shard, withoutNode));
+ if (pending.isEmpty())
+ return;
+ if (nanoTime() >= deadlineNanos)
+ throw new RuntimeException("Timed out draining shards; still pending: " + pending);
+ LockSupport.parkNanos(SECONDS.toNanos(1));
+ }
+ }
+
+ /**
+ * Poll every participant of the SEALING shard once for drain status; true iff all report that no mutation
+ * id allocated before the SEALING fence is still applying locally.
+ */
+ private static boolean drain(ShardMetadata shard, @Nullable NodeId withoutNode)
+ {
+ try
+ {
+ return Drain.poll(shard.keyspace, shard.sinceEpoch, shard.range, toEndpoints(shard.participants, withoutNode))
+ .get(DatabaseDescriptor.getRpcTimeout(MILLISECONDS), MILLISECONDS);
+ }
+ catch (InterruptedException | ExecutionException | TimeoutException e)
+ {
+ throw new RuntimeException(format("Failed to poll drain for shard %s", shard), e);
+ }
+ }
+
+ /**
+ * For each of the shards, capture the witnessed union then poll each participant until it has
+ * caught up and witnessed the entire offset union itself.
+ */
+ private static void reconcile(Set<ShardMetadata> shards, @Nullable NodeId withoutNode)
+ {
+ for (ShardMetadata shard : shards)
+ reconcile(shard, withoutNode);
+ }
+
+ private static void reconcile(ShardMetadata shard, @Nullable NodeId withoutNode)
+ {
+ // capture shard's witnessed offsets once
+ Log2OffsetsMap.Mutable offsets = captureWitnessedOffsets(shard, withoutNode);
+ // poll every participant until they've each witnessed the union of offsets
+ for (InetAddressAndPort endpoint : toEndpoints(shard.participants, withoutNode))
+ pollUntilWitnesses(shard, offsets, endpoint);
+ }
+
+ private static Log2OffsetsMap.Mutable captureWitnessedOffsets(ShardMetadata shard, @Nullable NodeId withoutNode)
+ {
+ List<InetAddressAndPort> endpoints = toEndpoints(shard.participants, withoutNode);
+ try
+ {
+ return ReconcileCapture.capture(shard.keyspace, shard.sinceEpoch, shard.range, endpoints)
+ .get(DatabaseDescriptor.getRpcTimeout(MILLISECONDS), MILLISECONDS);
+ }
+ catch (InterruptedException | ExecutionException | TimeoutException e)
+ {
+ throw new RuntimeException(format("Failed to capture witnessed offsets for shard %s", shard), e);
+ }
+ }
+
+ private static void pollUntilWitnesses(ShardMetadata shard, Log2OffsetsMap.Mutable offsets, InetAddressAndPort endpoint)
+ {
+ // TODO (expected): use a longer timeout
+ long deadlineNanos = nanoTime() + DatabaseDescriptor.getRpcTimeout(NANOSECONDS);
+
+ while (true)
+ {
+ if (nanoTime() >= deadlineNanos)
+ throw new RuntimeException("Timed out reconciling shard: " + shard);
+ try
+ {
+ boolean witnessed =
+ ReconcilePoll.poll(shard.keyspace, shard.sinceEpoch, shard.range, offsets, endpoint)
+ .get(DatabaseDescriptor.getRpcTimeout(MILLISECONDS), MILLISECONDS);
+ if (witnessed)
+ return;
+ }
+ catch (InterruptedException | ExecutionException | TimeoutException e)
+ {
+ throw new RuntimeException(format("Failed to poll reconcile for shard %s", shard), e);
+ }
+
+ LockSupport.parkNanos(SECONDS.toNanos(1));
+ }
+ }
+
+ private static void complete(Set<ShardMetadata> shards, @Nullable NodeId withoutNode)
+ {
+ List<Future<Void>> futures = new ArrayList<>(shards.size());
+ for (ShardMetadata shard : shards)
+ futures.add(complete(shard, withoutNode));
+
+ try
+ {
+ FutureCombiner.allOf(futures).get(DatabaseDescriptor.getRpcTimeout(MILLISECONDS), MILLISECONDS);
+ }
+ catch (InterruptedException | ExecutionException | TimeoutException e)
+ {
+ throw new RuntimeException("Failed to complete sealing", e);
+ }
+ }
+
+ /**
+ * Promote every participant of the (drained and reconciled) shard from SEALING to SEALED.
+ */
+ private static AsyncPromise<Void> complete(ShardMetadata shard, @Nullable NodeId withoutNode)
+ {
+ return CompleteSealing.complete(shard.keyspace, shard.sinceEpoch, shard.range, toEndpoints(shard.participants, withoutNode));
+ }
+
+ /**
+ * A bootstrapping node knows nothing about the shards that currently exist,
+ * so it must collect the list of shards from the existing replicas before
+ * it can proceed with the rest of the sealing steps.
+ */
+ public static final class FetchShards
+ {
+ public static final class Request
+ {
+ final String keyspace;
+ final long beforeEpoch;
+ final Range<Token> range;
+
+ Request(String keyspace, long beforeEpoch, Range<Token> range)
+ {
+ this.keyspace = keyspace;
+ this.beforeEpoch = beforeEpoch;
+ this.range = range;
+ }
+ }
+
+ public static final class Response
+ {
+ final Set<ShardMetadata> shards;
+
+ Response(Set<ShardMetadata> shards)
+ {
+ this.shards = shards;
+ }
+ }
+
+ /**
+ * Query all replicas in a group in parallel for the shards over {@code ranges} obsoleted by an
+ * in-flight topology change (those with {@code sinceEpoch < beforeEpoch}). Resembles
+ * {@link ShardMetadataRequest#queryPeers}, but waits for every (live) replica to respond and
+ * merges their deduplicated shards rather than taking the first response.
+ */
+ public static AsyncPromise<Set<ShardMetadata>> fetch(
+ String keyspace, long beforeEpoch, Range<Token> range, Set<InetAddressAndPort> endpoints)
+ {
+ Preconditions.checkArgument(!endpoints.isEmpty());
+ AsyncPromise<Set<ShardMetadata>> promise = new AsyncPromise<>();
+ Set<ShardMetadata> merged = ConcurrentHashMap.newKeySet();
+
+ RequestCallback<Response> callback = new RequestCallback<>()
+ {
+ private final AtomicInteger remaining = new AtomicInteger(endpoints.size());
+
+ @Override
+ public void onResponse(Message<Response> msg)
+ {
+ merged.addAll(msg.payload.shards);
+ if (remaining.decrementAndGet() == 0)
+ promise.trySuccess(merged);
+ }
+
+ @Override
+ public void onFailure(InetAddressAndPort from, RequestFailure failure)
+ {
+ // happy path assumes all replicas are up; fail the whole fetch if any can't respond
+ promise.tryFailure(new RuntimeException(format("Failed to fetch shards to seal from %s for keyspace %s: %s", from, keyspace, failure)));
+ }
+
+ @Override
+ public boolean invokeOnFailure()
+ {
+ return true;
+ }
+ };
+
+ Message<Request> message = Message.out(Verb.MT_FETCH_SHARDS_REQ, new Request(keyspace, beforeEpoch, range));
+ for (InetAddressAndPort peer : endpoints)
+ MessagingService.instance().sendWithCallback(message, peer, callback);
+
+ return promise;
+ }
+
+ public static final IVerbHandler<Request> verbHandler = message ->
+ {
+ MutationTrackingService.ensureEnabled();
+ Request request = message.payload;
+ Set<ShardMetadata> shards = new HashSet<>();
+ MutationTrackingService.instance().forEachShardInKeyspace(request.keyspace, shard -> {
+ if (shard.sinceEpoch < request.beforeEpoch
+ && (request.range.contains(shard.range) || shard.range.contains(request.range))
Review Comment:
I think we want `range#intersects` here
##########
src/java/org/apache/cassandra/replication/SealingCoordinator.java:
##########
@@ -0,0 +1,1401 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.cassandra.replication;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.TimeoutException;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.concurrent.locks.LockSupport;
+
+import javax.annotation.Nonnull;
+import javax.annotation.Nullable;
+
+import com.google.common.base.Preconditions;
+
+import org.apache.cassandra.config.DatabaseDescriptor;
+import org.apache.cassandra.db.ColumnFamilyStore;
+import org.apache.cassandra.db.Keyspace;
+import org.apache.cassandra.db.TypeSizes;
+import org.apache.cassandra.dht.AbstractBounds;
+import org.apache.cassandra.dht.IPartitioner;
+import org.apache.cassandra.dht.Range;
+import org.apache.cassandra.dht.Token;
+import org.apache.cassandra.exceptions.RequestFailure;
+import org.apache.cassandra.io.UnversionedSerializer;
+import org.apache.cassandra.io.util.DataInputPlus;
+import org.apache.cassandra.io.util.DataOutputPlus;
+import org.apache.cassandra.locator.EndpointsByReplica;
+import org.apache.cassandra.locator.EndpointsForRange;
+import org.apache.cassandra.locator.InetAddressAndPort;
+import org.apache.cassandra.locator.RangesAtEndpoint;
+import org.apache.cassandra.locator.Replica;
+import org.apache.cassandra.net.IVerbHandler;
+import org.apache.cassandra.net.Message;
+import org.apache.cassandra.net.MessagingService;
+import org.apache.cassandra.net.RequestCallback;
+import org.apache.cassandra.net.Verb;
+import org.apache.cassandra.schema.KeyspaceMetadata;
+import org.apache.cassandra.tcm.ClusterMetadata;
+import org.apache.cassandra.tcm.membership.NodeId;
+import org.apache.cassandra.tcm.ownership.DataPlacement;
+import org.apache.cassandra.tcm.ownership.MovementMap;
+import org.apache.cassandra.tcm.ownership.PlacementDeltas;
+import org.apache.cassandra.tcm.ownership.VersionedEndpoints;
+import org.apache.cassandra.tcm.sequences.LeaveStreams;
+import org.apache.cassandra.utils.CollectionSerializers;
+import org.apache.cassandra.utils.FBUtilities;
+import org.apache.cassandra.utils.concurrent.AsyncPromise;
+import org.apache.cassandra.utils.concurrent.Future;
+import org.apache.cassandra.utils.concurrent.FutureCombiner;
+
+import static java.lang.String.format;
+import static java.util.concurrent.TimeUnit.MILLISECONDS;
+import static java.util.concurrent.TimeUnit.NANOSECONDS;
+import static java.util.concurrent.TimeUnit.SECONDS;
+import static org.apache.cassandra.utils.Clock.Global.nanoTime;
+
+/*
+ * Sealing {@link Shard}s once a topology change obsoletes them.
+ *
+ * Happy Path Sequence:
+ * 1. Bring every participant up to (at least) this seal's epoch and fence allocation, so that everyone
+ * stops writing to the old shards; then wait until all in-progress mutations have been done.
+ * 2. Same for tracked transfers
+ * 3. Wait until all participating replicas have reconciled the union of their witnessed ids
+ * 4. Mark the shard as fully sealed
+ * 5. Stop including the shard in mutation summaries for read requests.
+ *
+ * Unhappy Path Sequence (not yet implemented):
+ * 1. Wait until everyone who's up stops writing to the old shards and all in-progress mutations have been done
+ * 2. Same for tracked transfers
+ * 3. Wait until all *live* participating replicas have reconciled the union of their witnessed ids
+ * 4. Mark the shard as partially sealed, label it with the set of participants that were able to seal
+ * 5. When a down node comes up, have it talk to other nodes and learn about sealing, and what mutations/logs
+ * they have exchanged. Diff with what we have - in system tables and in the journal (though system table
+ * plus local replay should give us an up-to-date metadata state for the shard)
+ */
+public final class SealingCoordinator
+{
+ /*
+ * BOOTSTRAP sequence illustrated. Stages in which write placements
+ * remain unchanged, and only read placements change, are skipped
+ * here as irrelevant to sharding / sealing.
+ *
+ * There is a tracked keyspace with RF=3.
+ * A node n35 with token=35 is added in the middle of the ring.
+ *
+ * 1. Initial state at e0:
+ *
+ * (10,20] | {n20,n30,n40} | e0 | ACTIVE
+ * (20,30] | {n30,n40,n50} | e0 | ACTIVE
+ * (30,40] | {n40,n50,n60} | e0 | ACTIVE
+ *
+ * 2. PREPARE_JOIN at e1:
+ *
+ * (10,20] | {n20,n30,n40} | e0 | ACTIVE
+ * (20,30] | {n30,n40,n50} | e0 | ACTIVE
+ * (30,40] | {n40,n50,n60} | e0 | ACTIVE
+ * + (30,35] | {n40,n50,n60} | e1 | ACTIVE
+ * + (35,40] | {n40,n50,n60} | e1 | ACTIVE
+ *
+ * 3. START_JOIN at e2:
+ *
+ * (10,20] | {n20,n30,n40} | e0 | ACTIVE
+ * + (10,20] | {n20,n30,n35,n40} | e2 | ACTIVE
+ * (20,30] | {n30,n40,n50} | e0 | ACTIVE
+ * + (20,30] | {n30,n35,n40,n50} | e2 | ACTIVE
+ * (30,40] | {n40,n50,n60} | e0 | ACTIVE
+ * (30,35] | {n40,n50,n60} | e1 | ACTIVE
+ * + (30,35] | {n35,n40,n50,n60} | e2 | ACTIVE
+ * (35,40] | {n40,n50,n60} | e1 | ACTIVE
+ *
+ * 4. MID_JOIN at e3 [*]:
+ *
+ * S (10,20] | {n20,n30,n40} | e0 | SEALED
+ * (10,20] | {n20,n30,n35,n40} | e2 | ACTIVE
+ * S (20,30] | {n30,n40,n50} | e0 | SEALED
+ * (20,30] | {n30,n35,n40,n50} | e2 | ACTIVE
+ * S (30,40] | {n40,n50,n60} | e0 | SEALED
+ * S (30,35] | {n40,n50,n60} | e1 | SEALED
+ * (30,35] | {n35,n40,n50,n60} | e2 | ACTIVE
+ * (35,40] | {n40,n50,n60} | e1 | ACTIVE
+ *
+ * 5. FINISH_JOIN at e4:
+ *
+ * (10,20] | {n20,n30,n40} | e0 | SEALED
+ * S (10,20] | {n20,n30,n35,n40} | e2 | SEALED
+ * + (10,20] | {n20,n30,n35} | e4 | ACTIVE
+ * (20,30] | {n30,n40,n50} | e0 | SEALED
+ * S (20,30] | {n30,n35,n40,n50} | e2 | SEALED
+ * + (20,30] | {n30,n35,n40} | e4 | ACTIVE
+ * (30,40] | {n40,n50,n60} | e0 | SEALED
+ * (30,35] | {n40,n50,n60} | e1 | SEALED
+ * S (30,35] | {n35,n40,n50,n60} | e2 | SEALED
+ * + (30,35] | {n35,n40,n50} | e4 | ACTIVE
+ * (35,40] | {n40,n50,n60} | e1 | ACTIVE
+ *
+ * [*] At this stage, after sealing the shards, we stream the sstables.
+ * All the offsets belonging to these sealed shards get stripped from
+ * stats component of streamed sstables. Otherwise, the new node
+ * won't be able to compact them or mark them as reconciled/repaired -
+ * it is rightfully unaware of the sealed shards.
+ */
+
+ /**
+ * Seal the shards obsoleted by bootstrap before the new node starts streaming.
+ */
+ public static void sealShardsAtMidJoin(ClusterMetadata metadata, MovementMap movements)
+ {
+ seal(discoverShardsAtMidJoinOrReplace(metadata, movements));
+ }
+
+ /**
+ * Seal the intermediate shards that were created during START_JOIN.
+ * TODO (required): this MUST eventually run while the affected ranges are still locked - between the
+ * removeFromWrites placement change and the range unlock. FinishJoin/FinishReplace currently apply
+ * removeFromWrites and the unlock atomically in a single transformation, so there is no committed
+ * dropped-but-locked epoch to hook. Until then, this runs just after FINISH_JOIN/FINISH_REPLACE,
+ * when the ranges are already unlocked. Left to implement for the unhappy path (AY).
+ */
+ public static void sealShardsAtFinishJoin(ClusterMetadata metadata, PlacementDeltas finishDelta)
+ {
+ seal(discoverShardsAtFinishJoinOrReplace(metadata, finishDelta, null));
+ }
+
+ /*
+ * REPLACE sequence illustrated. Stages in which write placements
+ * remain unchanged, and only read placements change, are skipped
+ * here as irrelevant to sharding / sealing.
+ *
+ * There is a tracked keyspace with RF=3.
+ * Node n40 is dead, and is being replaced with $n40.
+ *
+ * 1. Initial state at e0:
+ *
+ * (10,20] | {n20,n30,n40} | e0 | ACTIVE
+ * (20,30] | {n30,n40,n50} | e0 | ACTIVE
+ * (30,40] | {n40,n50,n60} | e0 | ACTIVE
+ *
+ * 2. START_REPLACE at e1:
+ *
+ * (10,20] | {n20,n30,n40} | e0 | ACTIVE
+ * + (10,20] | {n20,n30,n40,$n40}| e1 | ACTIVE
+ * (20,30] | {n30,n40,n50} | e0 | ACTIVE
+ * + (20,30] | {n30,n40,$n40,n50}| e1 | ACTIVE
+ * (30,40] | {n40,n50,n60} | e0 | ACTIVE
+ * + (30,40] | {n40,$n40,n50,n60}| e1 | ACTIVE
+ *
+ * 3. MID_REPLACE at e2 [*]:
+ *
+ * S (10,20] | {n20,n30,n40} | e0 | SEALED
+ * (10,20] | {n20,n30,n40,$n40}| e1 | ACTIVE
+ * S (20,30] | {n30,n40,n50} | e0 | SEALED
+ * (20,30] | {n30,n40,$n40,n50}| e1 | ACTIVE
+ * S (30,40] | {n40,n50,n60} | e0 | SEALED
+ * (30,40] | {n40,$n40,n50,n60}| e1 | ACTIVE
+ *
+ * 4. FINISH_REPLACE at e3:
+ *
+ * (10,20] | {n20,n30,n40} | e0 | SEALED
+ * S (10,20] | {n20,n30,n40,$n40}| e1 | SEALED
+ * + (10,20] | {n20,n30,$n40} | e3 | ACTIVE
+ * (20,30] | {n30,n40,n50} | e0 | SEALED
+ * S (20,30] | {n30,n40,$n40,n50}| e1 | SEALED
+ * + (20,30] | {n30,$n40,n50} | e3 | ACTIVE
+ * (30,40] | {n40,n50,n60} | e0 | SEALED
+ * S (30,40] | {n40,$n40,n50,n60}| e1 | SEALED
+ * + (30,40] | {$n40,n50,n60} | e3 | ACTIVE
+ *
+ * [*] At this stage, after sealing the shards, we stream the sstables.
+ * All the offsets belonging to these sealed shards get stripped from
+ * stats component of streamed sstables. Otherwise, the new node
+ * won't be able to compact them or mark them as reconciled/repaired -
+ * it is rightfully unaware of the sealed shards.
+ */
+
+ /**
+ * Seal the shards obsoleted by host replacement before the replacement node starts streaming.
+ * The {@code deadNode} node cannot participate in the sealing.
+ */
+ public static void sealShardsAtMidReplace(ClusterMetadata metadata, MovementMap movements, @Nonnull NodeId deadNode)
+ {
+ seal(discoverShardsAtMidJoinOrReplace(metadata, movements), deadNode);
+ }
+
+ public static void sealShardsAtFinishReplace(ClusterMetadata metadata, PlacementDeltas finishDelta, @Nonnull NodeId deadNode)
+ {
+ seal(discoverShardsAtFinishJoinOrReplace(metadata, finishDelta, deadNode), deadNode);
+ }
+
+ /**
+ * Discover all shards that need to be sealed at MID_JOIN or MID_REPLACE.
+ */
+ private static Set<ShardMetadata> discoverShardsAtMidJoinOrReplace(ClusterMetadata metadata, MovementMap movements)
+ {
+ List<Future<Set<ShardMetadata>>> futures = new ArrayList<>();
+
+ for (KeyspaceMetadata ksm : metadata.schema.getKeyspaces())
+ {
+ if (!ksm.params.replicationType.isTracked())
+ continue;
+
+ EndpointsByReplica acquired = movements.get(ksm.params.replication);
+ if (acquired.isEmpty()) // when the keyspace is not replicated to this DC
+ continue;
+
+ DataPlacement placement = metadata.placements.get(ksm.params.replication);
+
+ for (Map.Entry<Replica, EndpointsForRange> entry : acquired.entrySet())
+ {
+ Range<Token> range = entry.getKey().range();
+ Set<InetAddressAndPort> endpoints = entry.getValue().endpoints();
+ // cutoff: the epoch this node was added to the write placement for the range;
+ // shards over the range with sinceEpoch < cutoff are obsoleted by this bootstrap or replace.
+ long beforeEpoch = placement.writes.forRange(range).lastModified().getEpoch();
+ futures.add(FetchShards.fetch(ksm.name, beforeEpoch, range, endpoints));
+ }
+ }
+
+ Set<ShardMetadata> shards = new HashSet<>();
+ try
+ {
+ for (Set<ShardMetadata> fetched : FutureCombiner.allOf(futures).get(DatabaseDescriptor.getRpcTimeout(MILLISECONDS), MILLISECONDS))
+ shards.addAll(fetched);
+ }
+ catch (InterruptedException | ExecutionException | TimeoutException e)
+ {
+ throw new RuntimeException("Failed to fetch shards to seal", e);
+ }
+ return shards;
+ }
+
+ /**
+ * Discover all shards that need to be sealed at FINISH_JOIN or FINISH_REPLACE.
+ */
+ private static Set<ShardMetadata> discoverShardsAtFinishJoinOrReplace(ClusterMetadata metadata, PlacementDeltas finishDelta, @Nullable NodeId deadNode)
+ {
+ long finishEpoch = metadata.epoch.getEpoch();
+ Set<ShardMetadata> shards = new HashSet<>();
+ for (KeyspaceMetadata ksm : metadata.schema.getKeyspaces())
+ {
+ if (!ksm.params.replicationType.isTracked())
+ continue;
+
+ DataPlacement placement = metadata.placements.get(ksm.params.replication);
+ finishDelta.get(ksm.params.replication).writes.removals.flattenValues().forEach(removed -> {
+ Set<Integer> nodeIDs = new HashSet<>();
+ for (InetAddressAndPort ep : placement.writes.forRange(removed.range()).endpoints())
+ nodeIDs.add(metadata.directory.peerId(ep).id());
+ nodeIDs.add(deadNode != null ? deadNode.id() : metadata.directory.peerId(removed.endpoint()).id());
+ Participants participants = new Participants(nodeIDs);
+
+ MutationTrackingService.instance().forEachShardInKeyspace(ksm.name, shard -> {
+ if (shard.sinceEpoch < finishEpoch
+ && shard.range.equals(removed.range())
+ && shard.participants.equals(participants)
+ && !shard.isSealed())
+ shards.add(new ShardMetadata(shard.keyspace, shard.sinceEpoch, shard.range, shard.participants));
+ });
+ });
+ }
+ return shards;
+ }
+
+ /*
+ * UNBOOTSTRAP and REMOVENODE sequences illustrated (same sequence as far as sharding is concerned).
+ * Stages in which write placements remain unchanged, and only read placements change, are skipped
+ * here as irrelevant to sharding / sealing.
+ *
+ * There is a tracked keyspace with RF=3.
+ * A node n40 with token=40 is removed in the middle of the ring.
+ *
+ * 1. Initial state at e0:
+ *
+ * (10,20] | {n20,n30,n40} | e0 | ACTIVE
+ * (20,30] | {n30,n40,n50} | e0 | ACTIVE
+ * (30,40] | {n40,n50,n60} | e0 | ACTIVE
+ * (40,50] | {n50,n60,n70} | e0 | ACTIVE
+ *
+ * 2. START_LEAVE at e1:
+ *
+ * (10,20] | {n20,n30,n40} | e0 | ACTIVE
+ * + (10,20] | {n20,n30,n40,n50} | e1 | ACTIVE
+ * (20,30] | {n30,n40,n50} | e0 | ACTIVE
+ * + (20,30] | {n30,n40,n50,n60} | e1 | ACTIVE
+ * (30,40] | {n40,n50,n60} | e0 | ACTIVE
+ * + (30,40] | {n40,n50,n60,n70} | e1 | ACTIVE
+ * (40,50] | {n50,n60,n70} | e0 | ACTIVE
+ *
+ * 3. MID_LEAVE at e2 [*]:
+ *
+ * S (10,20] | {n20,n30,n40} | e0 | SEALED
+ * (10,20] | {n20,n30,n40,n50} | e1 | ACTIVE
+ * S (20,30] | {n30,n40,n50} | e0 | SEALED
+ * (20,30] | {n30,n40,n50,n60} | e1 | ACTIVE
+ * S (30,40] | {n40,n50,n60} | e0 | SEALED
+ * (30,40] | {n40,n50,n60,n70} | e1 | ACTIVE
+ * (40,50] | {n50,n60,n70} | e0 | ACTIVE
+ *
+ * 4. FINISH_LEAVE at e3:
+ *
+ * (10,20] | {n20,n30,n40} | e0 | SEALED
+ * S (10,20] | {n20,n30,n40,n50} | e1 | SEALED
+ * + (10,20] | {n20,n30,n50} | e3 | ACTIVE
+ * (20,30] | {n30,n40,n50} | e0 | SEALED
+ * S (20,30] | {n30,n40,n50,n60} | e1 | SEALED
+ * + (20,30] | {n30,n50,n60} | e3 | ACTIVE
+ * (30,40] | {n40,n50,n60} | e0 | SEALED
+ * S (30,40] | {n40,n50,n60,n70} | e1 | SEALED
+ * S (40,50] | {n50,n60,n70} | e0 | SEALED
+ * + (30,50] | {n50,n60,n70} | e3 | ACTIVE
+ *
+ * [*] At this stage, after sealing the shards, we stream the sstables.
+ * All the offsets belonging to these sealed shards get stripped from
+ * stats component of streamed sstables. Otherwise, the new node
+ * won't be able to compact them or mark them as reconciled/repaired -
+ * it is rightfully unaware of the sealed shards.
+ */
+
+ /**
+ * Seal the shards obsoleted by START_LEAVE before streaming, for both UNBOOTSTRAP and REMOVENODE.
+ */
+ public static void sealShardsAtMidLeave(ClusterMetadata metadata, PlacementDeltas startDelta, NodeId leavingOrDeparted, LeaveStreams.Kind kind)
+ {
+ InetAddressAndPort leavingOrDepartedEndpoint = metadata.directory.endpoint(leavingOrDeparted);
+
+ List<Future<Set<ShardMetadata>>> futures = new ArrayList<>();
+ for (KeyspaceMetadata ksm : metadata.schema.getKeyspaces())
+ {
+ if (!ksm.params.replicationType.isTracked())
+ continue;
+
+ DataPlacement placement = metadata.placements.get(ksm.params.replication);
+ startDelta.get(ksm.params.replication).writes.additions.flattenValues().forEach(added -> {
+ Range<Token> range = added.range();
+ long beforeEpoch = placement.writes.forRange(range).lastModified().getEpoch();
+ Set<InetAddressAndPort> survivors = new HashSet<>(placement.writes.forRange(range).endpoints());
+ survivors.remove(leavingOrDepartedEndpoint);
+ futures.add(FetchShards.fetch(ksm.name, beforeEpoch, range, survivors));
+ });
+ }
+
+ Set<ShardMetadata> shards = new HashSet<>();
+ try
+ {
+ for (Set<ShardMetadata> fetched : FutureCombiner.allOf(futures).get(DatabaseDescriptor.getRpcTimeout(MILLISECONDS), MILLISECONDS))
+ for (ShardMetadata shard : fetched)
+ if (shard.participants.contains(leavingOrDeparted.id()))
+ shards.add(shard);
+ }
+ catch (InterruptedException | ExecutionException | TimeoutException e)
+ {
+ throw new RuntimeException("Failed to fetch shards to seal for leave", e);
+ }
+
+ seal(shards, kind == LeaveStreams.Kind.UNBOOTSTRAP ? null : leavingOrDeparted);
+ }
+
+ /**
+ * Seal the shards that need to be sealed at LEAVE_FINISH for both UNBOOTSTRAP and REMOVENODE.
+ */
+ public static void sealShardsAtFinishLeave(ClusterMetadata postFinish, PlacementDeltas finishDelta, NodeId leavingOrDeparted, LeaveStreams.Kind kind)
+ {
+ long finishEpoch = postFinish.epoch.getEpoch();
+ InetAddressAndPort leavingOrDepartedEndpoint = postFinish.directory.endpoint(leavingOrDeparted);
+
+ List<Future<Set<ShardMetadata>>> futures = new ArrayList<>();
+ for (KeyspaceMetadata ksm : postFinish.schema.getKeyspaces())
+ {
+ if (!ksm.params.replicationType.isTracked())
+ continue;
+
+ DataPlacement placement = postFinish.placements.get(ksm.params.replication);
+ RangesAtEndpoint removals = finishDelta.get(ksm.params.replication).writes.removals.get(leavingOrDepartedEndpoint);
+ for (Replica removedReplica : removals)
+ {
+ // FINISH_LEAVE merges the leavingOrDeparted node's (pre-merge) range with an adjacent one. Query the *merged*
+ // range (not the pre-merge range) so discovery covers both the leavingOrDeparted node's own obsoleted
+ // generations and the merge-half - the adjacent pre-merge range folded into the merged range, whose
+ // shard never listed the leavingOrDeparted node. matchRange() yields the containing merged group + its
+ // current (survivor) write endpoints.
+ VersionedEndpoints.ForRange group = placement.writes.matchRange(removedReplica.range());
+ Range<Token> mergedRange = group.range();
+ Set<InetAddressAndPort> survivors = new HashSet<>(group.endpoints());
+ survivors.remove(leavingOrDepartedEndpoint);
+ futures.add(FetchShards.fetch(ksm.name, finishEpoch, mergedRange, survivors));
+ }
+ }
+
+ Set<ShardMetadata> shards = new HashSet<>();
+ try
+ {
+ for (Set<ShardMetadata> fetched : FutureCombiner.allOf(futures).get(DatabaseDescriptor.getRpcTimeout(MILLISECONDS), MILLISECONDS))
+ shards.addAll(fetched);
+ }
+ catch (InterruptedException | ExecutionException | TimeoutException e)
+ {
+ throw new RuntimeException("Failed to fetch shards to seal for leave finish", e);
+ }
+
+ seal(shards, kind == LeaveStreams.Kind.UNBOOTSTRAP ? null : leavingOrDeparted);
+ }
+
+ /*
+ * Sealing logic.
+ */
+
+ private static void seal(Set<ShardMetadata> shards)
+ {
+ seal(shards, null);
+ }
+
+ private static void seal(Set<ShardMetadata> shards, @Nullable NodeId withoutNode)
+ {
+ initiate(shards, withoutNode); // ACTIVE -> SEALING for each Shard
+ drain(shards, withoutNode); // drain in-flight local writes
+ reconcile(shards, withoutNode); // wait for logs to reconcile
+ complete(shards, withoutNode); // SEALING-> SEALED for each Shard + journal flush
+ }
+
+ private static void initiate(Set<ShardMetadata> shards, @Nullable NodeId withoutNode)
+ {
+ List<Future<Void>> futures = new ArrayList<>(shards.size());
+ for (ShardMetadata shard : shards)
+ futures.add(initiate(shard, withoutNode));
+ try
+ {
+ FutureCombiner.allOf(futures).get(DatabaseDescriptor.getRpcTimeout(MILLISECONDS), MILLISECONDS);
+ }
+ catch (InterruptedException | ExecutionException | TimeoutException e)
+ {
+ throw new RuntimeException("Failed to initiate sealing", e);
+ }
+ }
+
+ /**
+ * Fence allocation on every participant of the obsoleted shard (ACTIVE -> SEALING) by marking the shard
+ * SEALING on each.
+ * TODO (expected): this assumes every participant has already enacted the topology change that obsoleted the shard (so
+ * it no longer allocates new ids on it). The ProgressBarrier gating bootstrap()/MID_JOIN only waits for
+ * EACH_QUORUM of the affected replicas, NOT all live replicas, so a live-but-lagging participant could still
+ * be on the old shard. Bringing each participant up to the seal epoch must be done per-replica (not
+ * per-shard) and will be added separately. TLDR - add a step to guaranteed up-to-speed rest of replicas epochs.
+ */
+ private static AsyncPromise<Void> initiate(ShardMetadata shard, @Nullable NodeId withoutNode)
+ {
+ return InitSealing.initiate(shard.keyspace, shard.sinceEpoch, shard.range, toEndpoints(shard.participants, withoutNode));
+ }
+
+ private static void drain(Set<ShardMetadata> shards, @Nullable NodeId withoutNode)
+ {
+ long deadlineNanos = nanoTime() + DatabaseDescriptor.getRpcTimeout(NANOSECONDS);
+ Set<ShardMetadata> pending = new HashSet<>(shards);
+ while (true)
+ {
+ // non-blocking poll: each participant reports its drain status immediately; retry until all drained
+ pending.removeIf(shard -> drain(shard, withoutNode));
+ if (pending.isEmpty())
+ return;
+ if (nanoTime() >= deadlineNanos)
+ throw new RuntimeException("Timed out draining shards; still pending: " + pending);
+ LockSupport.parkNanos(SECONDS.toNanos(1));
+ }
+ }
+
+ /**
+ * Poll every participant of the SEALING shard once for drain status; true iff all report that no mutation
+ * id allocated before the SEALING fence is still applying locally.
+ */
+ private static boolean drain(ShardMetadata shard, @Nullable NodeId withoutNode)
+ {
+ try
+ {
+ return Drain.poll(shard.keyspace, shard.sinceEpoch, shard.range, toEndpoints(shard.participants, withoutNode))
+ .get(DatabaseDescriptor.getRpcTimeout(MILLISECONDS), MILLISECONDS);
+ }
+ catch (InterruptedException | ExecutionException | TimeoutException e)
+ {
+ throw new RuntimeException(format("Failed to poll drain for shard %s", shard), e);
+ }
+ }
+
+ /**
+ * For each of the shards, capture the witnessed union then poll each participant until it has
+ * caught up and witnessed the entire offset union itself.
+ */
+ private static void reconcile(Set<ShardMetadata> shards, @Nullable NodeId withoutNode)
+ {
+ for (ShardMetadata shard : shards)
+ reconcile(shard, withoutNode);
+ }
+
+ private static void reconcile(ShardMetadata shard, @Nullable NodeId withoutNode)
+ {
+ // capture shard's witnessed offsets once
+ Log2OffsetsMap.Mutable offsets = captureWitnessedOffsets(shard, withoutNode);
+ // poll every participant until they've each witnessed the union of offsets
+ for (InetAddressAndPort endpoint : toEndpoints(shard.participants, withoutNode))
+ pollUntilWitnesses(shard, offsets, endpoint);
+ }
+
+ private static Log2OffsetsMap.Mutable captureWitnessedOffsets(ShardMetadata shard, @Nullable NodeId withoutNode)
+ {
+ List<InetAddressAndPort> endpoints = toEndpoints(shard.participants, withoutNode);
+ try
+ {
+ return ReconcileCapture.capture(shard.keyspace, shard.sinceEpoch, shard.range, endpoints)
+ .get(DatabaseDescriptor.getRpcTimeout(MILLISECONDS), MILLISECONDS);
+ }
+ catch (InterruptedException | ExecutionException | TimeoutException e)
+ {
+ throw new RuntimeException(format("Failed to capture witnessed offsets for shard %s", shard), e);
+ }
+ }
+
+ private static void pollUntilWitnesses(ShardMetadata shard, Log2OffsetsMap.Mutable offsets, InetAddressAndPort endpoint)
+ {
+ // TODO (expected): use a longer timeout
+ long deadlineNanos = nanoTime() + DatabaseDescriptor.getRpcTimeout(NANOSECONDS);
+
+ while (true)
+ {
+ if (nanoTime() >= deadlineNanos)
+ throw new RuntimeException("Timed out reconciling shard: " + shard);
+ try
+ {
+ boolean witnessed =
+ ReconcilePoll.poll(shard.keyspace, shard.sinceEpoch, shard.range, offsets, endpoint)
+ .get(DatabaseDescriptor.getRpcTimeout(MILLISECONDS), MILLISECONDS);
+ if (witnessed)
+ return;
+ }
+ catch (InterruptedException | ExecutionException | TimeoutException e)
+ {
+ throw new RuntimeException(format("Failed to poll reconcile for shard %s", shard), e);
+ }
+
+ LockSupport.parkNanos(SECONDS.toNanos(1));
+ }
+ }
+
+ private static void complete(Set<ShardMetadata> shards, @Nullable NodeId withoutNode)
+ {
+ List<Future<Void>> futures = new ArrayList<>(shards.size());
+ for (ShardMetadata shard : shards)
+ futures.add(complete(shard, withoutNode));
+
+ try
+ {
+ FutureCombiner.allOf(futures).get(DatabaseDescriptor.getRpcTimeout(MILLISECONDS), MILLISECONDS);
+ }
+ catch (InterruptedException | ExecutionException | TimeoutException e)
+ {
+ throw new RuntimeException("Failed to complete sealing", e);
+ }
+ }
+
+ /**
+ * Promote every participant of the (drained and reconciled) shard from SEALING to SEALED.
+ */
+ private static AsyncPromise<Void> complete(ShardMetadata shard, @Nullable NodeId withoutNode)
+ {
+ return CompleteSealing.complete(shard.keyspace, shard.sinceEpoch, shard.range, toEndpoints(shard.participants, withoutNode));
+ }
+
+ /**
+ * A bootstrapping node knows nothing about the shards that currently exist,
+ * so it must collect the list of shards from the existing replicas before
+ * it can proceed with the rest of the sealing steps.
+ */
+ public static final class FetchShards
+ {
+ public static final class Request
+ {
+ final String keyspace;
+ final long beforeEpoch;
+ final Range<Token> range;
+
+ Request(String keyspace, long beforeEpoch, Range<Token> range)
+ {
+ this.keyspace = keyspace;
+ this.beforeEpoch = beforeEpoch;
+ this.range = range;
+ }
+ }
+
+ public static final class Response
+ {
+ final Set<ShardMetadata> shards;
+
+ Response(Set<ShardMetadata> shards)
+ {
+ this.shards = shards;
+ }
+ }
+
+ /**
+ * Query all replicas in a group in parallel for the shards over {@code ranges} obsoleted by an
+ * in-flight topology change (those with {@code sinceEpoch < beforeEpoch}). Resembles
+ * {@link ShardMetadataRequest#queryPeers}, but waits for every (live) replica to respond and
+ * merges their deduplicated shards rather than taking the first response.
+ */
+ public static AsyncPromise<Set<ShardMetadata>> fetch(
+ String keyspace, long beforeEpoch, Range<Token> range, Set<InetAddressAndPort> endpoints)
+ {
+ Preconditions.checkArgument(!endpoints.isEmpty());
+ AsyncPromise<Set<ShardMetadata>> promise = new AsyncPromise<>();
+ Set<ShardMetadata> merged = ConcurrentHashMap.newKeySet();
+
+ RequestCallback<Response> callback = new RequestCallback<>()
+ {
+ private final AtomicInteger remaining = new AtomicInteger(endpoints.size());
+
+ @Override
+ public void onResponse(Message<Response> msg)
+ {
+ merged.addAll(msg.payload.shards);
+ if (remaining.decrementAndGet() == 0)
+ promise.trySuccess(merged);
+ }
+
+ @Override
+ public void onFailure(InetAddressAndPort from, RequestFailure failure)
+ {
+ // happy path assumes all replicas are up; fail the whole fetch if any can't respond
+ promise.tryFailure(new RuntimeException(format("Failed to fetch shards to seal from %s for keyspace %s: %s", from, keyspace, failure)));
+ }
+
+ @Override
+ public boolean invokeOnFailure()
+ {
+ return true;
+ }
+ };
+
+ Message<Request> message = Message.out(Verb.MT_FETCH_SHARDS_REQ, new Request(keyspace, beforeEpoch, range));
+ for (InetAddressAndPort peer : endpoints)
+ MessagingService.instance().sendWithCallback(message, peer, callback);
+
+ return promise;
+ }
+
+ public static final IVerbHandler<Request> verbHandler = message ->
+ {
+ MutationTrackingService.ensureEnabled();
+ Request request = message.payload;
+ Set<ShardMetadata> shards = new HashSet<>();
+ MutationTrackingService.instance().forEachShardInKeyspace(request.keyspace, shard -> {
+ if (shard.sinceEpoch < request.beforeEpoch
+ && (request.range.contains(shard.range) || shard.range.contains(request.range))
+ && !shard.isSealed())
+ shards.add(new ShardMetadata(shard.keyspace, shard.sinceEpoch, shard.range, shard.participants));
+ });
+ MessagingService.instance().send(message.responseWith(new Response(shards)), message.from());
+ };
+
+ public static final VersionedSerializer<Request> requestSerializer = new VersionedSerializer<>()
+ {
+ @Override
+ public void serialize(Request r, DataOutputPlus out, Version version) throws IOException
+ {
+ out.writeUTF(r.keyspace);
+ out.writeLong(r.beforeEpoch);
+ AbstractBounds.tokenSerializer.serialize(r.range, out, version.messagingVersion());
+ }
+
+ @Override
+ public Request deserialize(DataInputPlus in, Version version) throws IOException
+ {
+ String keyspace = in.readUTF();
+ long beforeEpoch = in.readLong();
+ Range<Token> range = (Range<Token>) AbstractBounds.tokenSerializer.deserialize(in, IPartitioner.global(), version.messagingVersion());
+ return new Request(keyspace, beforeEpoch, range);
+ }
+
+ @Override
+ public long serializedSize(Request r, Version version)
+ {
+ long size = TypeSizes.sizeof(r.keyspace);
+ size += TypeSizes.sizeof(r.beforeEpoch);
+ size += AbstractBounds.tokenSerializer.serializedSize(r.range, version.messagingVersion());
+ return size;
+ }
+ };
+
+ public static final VersionedSerializer<Response> responseSerializer = new VersionedSerializer<>()
+ {
+ @Override
+ public void serialize(Response r, DataOutputPlus out, Version version) throws IOException
+ {
+ CollectionSerializers.serializeCollection(r.shards, out, version, ShardMetadata.serializer);
+ }
+
+ @Override
+ public Response deserialize(DataInputPlus in, Version version) throws IOException
+ {
+ return new Response(CollectionSerializers.deserializeSet(in, version, ShardMetadata.serializer));
+ }
+
+ @Override
+ public long serializedSize(Response r, Version version)
+ {
+ return CollectionSerializers.serializedCollectionSize(r.shards, version, ShardMetadata.serializer);
+ }
+ };
+ }
+
+ public static final class InitSealing
+ {
+ public static final class Request
+ {
+ final String keyspace;
+ final long sinceEpoch;
+ final Range<Token> range;
+
+ Request(String keyspace, long sinceEpoch, Range<Token> range)
+ {
+ this.keyspace = keyspace;
+ this.sinceEpoch = sinceEpoch;
+ this.range = range;
+ }
+ }
+
+ // cannot use NoPayload, because its serializer cannot be wrapped inside mtEmbedded()
+ public static final class Response
+ {
+ private static final Response instance = new Response();
+ }
+
+ /**
+ * Tell every participant of an obsoleted shard to fence id allocation (mark it SEALING)
+ */
+ public static AsyncPromise<Void> initiate(
+ String keyspace, long sinceEpoch, Range<Token> range, List<InetAddressAndPort> endpoints)
+ {
+ AsyncPromise<Void> promise = new AsyncPromise<>();
+
+ RequestCallback<Response> callback = new RequestCallback<>()
+ {
+ private final AtomicInteger remaining = new AtomicInteger(endpoints.size());
+
+ @Override
+ public void onResponse(Message<Response> msg)
+ {
+ if (remaining.decrementAndGet() == 0)
+ promise.trySuccess(null);
+ }
+
+ @Override
+ public void onFailure(InetAddressAndPort from, RequestFailure failure)
+ {
+ promise.tryFailure(new RuntimeException(format("Failed to initiate sealing on %s for keyspace %s: %s", from, keyspace, failure)));
+ }
+
+ @Override
+ public boolean invokeOnFailure()
+ {
+ return true;
+ }
+ };
+
+ Message<Request> message = Message.out(Verb.MT_INIT_SEALING_REQ, new Request(keyspace, sinceEpoch, range));
+ for (InetAddressAndPort peer : endpoints)
+ MessagingService.instance().sendWithCallback(message, peer, callback);
+
+ return promise;
+ }
+
+ public static final IVerbHandler<Request> verbHandler = message -> {
+ MutationTrackingService.ensureEnabled();
+ Request request = message.payload;
+ MutationTrackingService.instance().markShardSealing(request.keyspace, request.sinceEpoch, request.range);
+ MessagingService.instance().send(message.responseWith(Response.instance), message.from());
+ };
+
+ public static final VersionedSerializer<Request> requestSerializer = new VersionedSerializer<>()
+ {
+ @Override
+ public void serialize(Request r, DataOutputPlus out, Version version) throws IOException
+ {
+ out.writeUTF(r.keyspace);
+ out.writeLong(r.sinceEpoch);
+ AbstractBounds.tokenSerializer.serialize(r.range, out, version.messagingVersion());
+ }
+
+ @Override
+ public Request deserialize(DataInputPlus in, Version version) throws IOException
+ {
+ String keyspace = in.readUTF();
+ long sinceEpoch = in.readLong();
+ Range<Token> range = (Range<Token>) AbstractBounds.tokenSerializer.deserialize(in, IPartitioner.global(), version.messagingVersion());
+ return new Request(keyspace, sinceEpoch, range);
+ }
+
+ @Override
+ public long serializedSize(Request r, Version version)
+ {
+ long size = TypeSizes.sizeof(r.keyspace);
+ size += TypeSizes.sizeof(r.sinceEpoch);
+ size += AbstractBounds.tokenSerializer.serializedSize(r.range, version.messagingVersion());
+ return size;
+ }
+ };
+
+ public static final UnversionedSerializer<Response> responseSerializer = new UnversionedSerializer<>()
+ {
+ @Override
+ public void serialize(Response r, DataOutputPlus out)
+ {
+ }
+
+ @Override
+ public Response deserialize(DataInputPlus in)
+ {
+ return Response.instance;
+ }
+
+ @Override
+ public long serializedSize(Response r)
+ {
+ return 0;
+ }
+ };
+ }
+
+ public static final class Drain
+ {
+ public static final class Request
+ {
+ final String keyspace;
+ final long sinceEpoch;
+ final Range<Token> range;
+
+ Request(String keyspace, long sinceEpoch, Range<Token> range)
+ {
+ this.keyspace = keyspace;
+ this.sinceEpoch = sinceEpoch;
+ this.range = range;
+ }
+ }
+
+ public static final class Response
+ {
+ final boolean drained;
+
+ Response(boolean drained)
+ {
+ this.drained = drained;
+ }
+ }
+
+ /**
+ * Poll every participant of a SEALING shard for whether it has drained its in-flight local
+ * writes. Succeeds with {@code true} iff every participant reports drained.
+ */
+ public static AsyncPromise<Boolean> poll(
+ String keyspace, long sinceEpoch, Range<Token> range, List<InetAddressAndPort> endpoints)
+ {
Review Comment:
```suggestion
{
Preconditions.checkArgument(!endpoints.isEmpty());
```
##########
src/java/org/apache/cassandra/replication/SealingCoordinator.java:
##########
@@ -0,0 +1,1401 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.cassandra.replication;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.TimeoutException;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.concurrent.locks.LockSupport;
+
+import javax.annotation.Nonnull;
+import javax.annotation.Nullable;
+
+import com.google.common.base.Preconditions;
+
+import org.apache.cassandra.config.DatabaseDescriptor;
+import org.apache.cassandra.db.ColumnFamilyStore;
+import org.apache.cassandra.db.Keyspace;
+import org.apache.cassandra.db.TypeSizes;
+import org.apache.cassandra.dht.AbstractBounds;
+import org.apache.cassandra.dht.IPartitioner;
+import org.apache.cassandra.dht.Range;
+import org.apache.cassandra.dht.Token;
+import org.apache.cassandra.exceptions.RequestFailure;
+import org.apache.cassandra.io.UnversionedSerializer;
+import org.apache.cassandra.io.util.DataInputPlus;
+import org.apache.cassandra.io.util.DataOutputPlus;
+import org.apache.cassandra.locator.EndpointsByReplica;
+import org.apache.cassandra.locator.EndpointsForRange;
+import org.apache.cassandra.locator.InetAddressAndPort;
+import org.apache.cassandra.locator.RangesAtEndpoint;
+import org.apache.cassandra.locator.Replica;
+import org.apache.cassandra.net.IVerbHandler;
+import org.apache.cassandra.net.Message;
+import org.apache.cassandra.net.MessagingService;
+import org.apache.cassandra.net.RequestCallback;
+import org.apache.cassandra.net.Verb;
+import org.apache.cassandra.schema.KeyspaceMetadata;
+import org.apache.cassandra.tcm.ClusterMetadata;
+import org.apache.cassandra.tcm.membership.NodeId;
+import org.apache.cassandra.tcm.ownership.DataPlacement;
+import org.apache.cassandra.tcm.ownership.MovementMap;
+import org.apache.cassandra.tcm.ownership.PlacementDeltas;
+import org.apache.cassandra.tcm.ownership.VersionedEndpoints;
+import org.apache.cassandra.tcm.sequences.LeaveStreams;
+import org.apache.cassandra.utils.CollectionSerializers;
+import org.apache.cassandra.utils.FBUtilities;
+import org.apache.cassandra.utils.concurrent.AsyncPromise;
+import org.apache.cassandra.utils.concurrent.Future;
+import org.apache.cassandra.utils.concurrent.FutureCombiner;
+
+import static java.lang.String.format;
+import static java.util.concurrent.TimeUnit.MILLISECONDS;
+import static java.util.concurrent.TimeUnit.NANOSECONDS;
+import static java.util.concurrent.TimeUnit.SECONDS;
+import static org.apache.cassandra.utils.Clock.Global.nanoTime;
+
+/*
+ * Sealing {@link Shard}s once a topology change obsoletes them.
+ *
+ * Happy Path Sequence:
+ * 1. Bring every participant up to (at least) this seal's epoch and fence allocation, so that everyone
+ * stops writing to the old shards; then wait until all in-progress mutations have been done.
+ * 2. Same for tracked transfers
+ * 3. Wait until all participating replicas have reconciled the union of their witnessed ids
+ * 4. Mark the shard as fully sealed
+ * 5. Stop including the shard in mutation summaries for read requests.
+ *
+ * Unhappy Path Sequence (not yet implemented):
+ * 1. Wait until everyone who's up stops writing to the old shards and all in-progress mutations have been done
+ * 2. Same for tracked transfers
+ * 3. Wait until all *live* participating replicas have reconciled the union of their witnessed ids
+ * 4. Mark the shard as partially sealed, label it with the set of participants that were able to seal
+ * 5. When a down node comes up, have it talk to other nodes and learn about sealing, and what mutations/logs
+ * they have exchanged. Diff with what we have - in system tables and in the journal (though system table
+ * plus local replay should give us an up-to-date metadata state for the shard)
+ */
+public final class SealingCoordinator
+{
+ /*
+ * BOOTSTRAP sequence illustrated. Stages in which write placements
+ * remain unchanged, and only read placements change, are skipped
+ * here as irrelevant to sharding / sealing.
+ *
+ * There is a tracked keyspace with RF=3.
+ * A node n35 with token=35 is added in the middle of the ring.
+ *
+ * 1. Initial state at e0:
+ *
+ * (10,20] | {n20,n30,n40} | e0 | ACTIVE
+ * (20,30] | {n30,n40,n50} | e0 | ACTIVE
+ * (30,40] | {n40,n50,n60} | e0 | ACTIVE
+ *
+ * 2. PREPARE_JOIN at e1:
+ *
+ * (10,20] | {n20,n30,n40} | e0 | ACTIVE
+ * (20,30] | {n30,n40,n50} | e0 | ACTIVE
+ * (30,40] | {n40,n50,n60} | e0 | ACTIVE
+ * + (30,35] | {n40,n50,n60} | e1 | ACTIVE
+ * + (35,40] | {n40,n50,n60} | e1 | ACTIVE
+ *
+ * 3. START_JOIN at e2:
+ *
+ * (10,20] | {n20,n30,n40} | e0 | ACTIVE
+ * + (10,20] | {n20,n30,n35,n40} | e2 | ACTIVE
+ * (20,30] | {n30,n40,n50} | e0 | ACTIVE
+ * + (20,30] | {n30,n35,n40,n50} | e2 | ACTIVE
+ * (30,40] | {n40,n50,n60} | e0 | ACTIVE
+ * (30,35] | {n40,n50,n60} | e1 | ACTIVE
+ * + (30,35] | {n35,n40,n50,n60} | e2 | ACTIVE
+ * (35,40] | {n40,n50,n60} | e1 | ACTIVE
+ *
+ * 4. MID_JOIN at e3 [*]:
+ *
+ * S (10,20] | {n20,n30,n40} | e0 | SEALED
+ * (10,20] | {n20,n30,n35,n40} | e2 | ACTIVE
+ * S (20,30] | {n30,n40,n50} | e0 | SEALED
+ * (20,30] | {n30,n35,n40,n50} | e2 | ACTIVE
+ * S (30,40] | {n40,n50,n60} | e0 | SEALED
+ * S (30,35] | {n40,n50,n60} | e1 | SEALED
+ * (30,35] | {n35,n40,n50,n60} | e2 | ACTIVE
+ * (35,40] | {n40,n50,n60} | e1 | ACTIVE
+ *
+ * 5. FINISH_JOIN at e4:
+ *
+ * (10,20] | {n20,n30,n40} | e0 | SEALED
+ * S (10,20] | {n20,n30,n35,n40} | e2 | SEALED
+ * + (10,20] | {n20,n30,n35} | e4 | ACTIVE
+ * (20,30] | {n30,n40,n50} | e0 | SEALED
+ * S (20,30] | {n30,n35,n40,n50} | e2 | SEALED
+ * + (20,30] | {n30,n35,n40} | e4 | ACTIVE
+ * (30,40] | {n40,n50,n60} | e0 | SEALED
+ * (30,35] | {n40,n50,n60} | e1 | SEALED
+ * S (30,35] | {n35,n40,n50,n60} | e2 | SEALED
+ * + (30,35] | {n35,n40,n50} | e4 | ACTIVE
+ * (35,40] | {n40,n50,n60} | e1 | ACTIVE
+ *
+ * [*] At this stage, after sealing the shards, we stream the sstables.
+ * All the offsets belonging to these sealed shards get stripped from
+ * stats component of streamed sstables. Otherwise, the new node
+ * won't be able to compact them or mark them as reconciled/repaired -
+ * it is rightfully unaware of the sealed shards.
+ */
+
+ /**
+ * Seal the shards obsoleted by bootstrap before the new node starts streaming.
+ */
+ public static void sealShardsAtMidJoin(ClusterMetadata metadata, MovementMap movements)
+ {
+ seal(discoverShardsAtMidJoinOrReplace(metadata, movements));
+ }
+
+ /**
+ * Seal the intermediate shards that were created during START_JOIN.
+ * TODO (required): this MUST eventually run while the affected ranges are still locked - between the
+ * removeFromWrites placement change and the range unlock. FinishJoin/FinishReplace currently apply
+ * removeFromWrites and the unlock atomically in a single transformation, so there is no committed
+ * dropped-but-locked epoch to hook. Until then, this runs just after FINISH_JOIN/FINISH_REPLACE,
+ * when the ranges are already unlocked. Left to implement for the unhappy path (AY).
+ */
+ public static void sealShardsAtFinishJoin(ClusterMetadata metadata, PlacementDeltas finishDelta)
+ {
+ seal(discoverShardsAtFinishJoinOrReplace(metadata, finishDelta, null));
+ }
+
+ /*
+ * REPLACE sequence illustrated. Stages in which write placements
+ * remain unchanged, and only read placements change, are skipped
+ * here as irrelevant to sharding / sealing.
+ *
+ * There is a tracked keyspace with RF=3.
+ * Node n40 is dead, and is being replaced with $n40.
+ *
+ * 1. Initial state at e0:
+ *
+ * (10,20] | {n20,n30,n40} | e0 | ACTIVE
+ * (20,30] | {n30,n40,n50} | e0 | ACTIVE
+ * (30,40] | {n40,n50,n60} | e0 | ACTIVE
+ *
+ * 2. START_REPLACE at e1:
+ *
+ * (10,20] | {n20,n30,n40} | e0 | ACTIVE
+ * + (10,20] | {n20,n30,n40,$n40}| e1 | ACTIVE
+ * (20,30] | {n30,n40,n50} | e0 | ACTIVE
+ * + (20,30] | {n30,n40,$n40,n50}| e1 | ACTIVE
+ * (30,40] | {n40,n50,n60} | e0 | ACTIVE
+ * + (30,40] | {n40,$n40,n50,n60}| e1 | ACTIVE
+ *
+ * 3. MID_REPLACE at e2 [*]:
+ *
+ * S (10,20] | {n20,n30,n40} | e0 | SEALED
+ * (10,20] | {n20,n30,n40,$n40}| e1 | ACTIVE
+ * S (20,30] | {n30,n40,n50} | e0 | SEALED
+ * (20,30] | {n30,n40,$n40,n50}| e1 | ACTIVE
+ * S (30,40] | {n40,n50,n60} | e0 | SEALED
+ * (30,40] | {n40,$n40,n50,n60}| e1 | ACTIVE
+ *
+ * 4. FINISH_REPLACE at e3:
+ *
+ * (10,20] | {n20,n30,n40} | e0 | SEALED
+ * S (10,20] | {n20,n30,n40,$n40}| e1 | SEALED
+ * + (10,20] | {n20,n30,$n40} | e3 | ACTIVE
+ * (20,30] | {n30,n40,n50} | e0 | SEALED
+ * S (20,30] | {n30,n40,$n40,n50}| e1 | SEALED
+ * + (20,30] | {n30,$n40,n50} | e3 | ACTIVE
+ * (30,40] | {n40,n50,n60} | e0 | SEALED
+ * S (30,40] | {n40,$n40,n50,n60}| e1 | SEALED
+ * + (30,40] | {$n40,n50,n60} | e3 | ACTIVE
+ *
+ * [*] At this stage, after sealing the shards, we stream the sstables.
+ * All the offsets belonging to these sealed shards get stripped from
+ * stats component of streamed sstables. Otherwise, the new node
+ * won't be able to compact them or mark them as reconciled/repaired -
+ * it is rightfully unaware of the sealed shards.
+ */
+
+ /**
+ * Seal the shards obsoleted by host replacement before the replacement node starts streaming.
+ * The {@code deadNode} node cannot participate in the sealing.
+ */
+ public static void sealShardsAtMidReplace(ClusterMetadata metadata, MovementMap movements, @Nonnull NodeId deadNode)
+ {
+ seal(discoverShardsAtMidJoinOrReplace(metadata, movements), deadNode);
+ }
+
+ public static void sealShardsAtFinishReplace(ClusterMetadata metadata, PlacementDeltas finishDelta, @Nonnull NodeId deadNode)
+ {
+ seal(discoverShardsAtFinishJoinOrReplace(metadata, finishDelta, deadNode), deadNode);
+ }
+
+ /**
+ * Discover all shards that need to be sealed at MID_JOIN or MID_REPLACE.
+ */
+ private static Set<ShardMetadata> discoverShardsAtMidJoinOrReplace(ClusterMetadata metadata, MovementMap movements)
+ {
+ List<Future<Set<ShardMetadata>>> futures = new ArrayList<>();
+
+ for (KeyspaceMetadata ksm : metadata.schema.getKeyspaces())
+ {
+ if (!ksm.params.replicationType.isTracked())
+ continue;
+
+ EndpointsByReplica acquired = movements.get(ksm.params.replication);
+ if (acquired.isEmpty()) // when the keyspace is not replicated to this DC
+ continue;
+
+ DataPlacement placement = metadata.placements.get(ksm.params.replication);
+
+ for (Map.Entry<Replica, EndpointsForRange> entry : acquired.entrySet())
+ {
+ Range<Token> range = entry.getKey().range();
+ Set<InetAddressAndPort> endpoints = entry.getValue().endpoints();
+ // cutoff: the epoch this node was added to the write placement for the range;
+ // shards over the range with sinceEpoch < cutoff are obsoleted by this bootstrap or replace.
+ long beforeEpoch = placement.writes.forRange(range).lastModified().getEpoch();
+ futures.add(FetchShards.fetch(ksm.name, beforeEpoch, range, endpoints));
+ }
+ }
+
+ Set<ShardMetadata> shards = new HashSet<>();
+ try
+ {
+ for (Set<ShardMetadata> fetched : FutureCombiner.allOf(futures).get(DatabaseDescriptor.getRpcTimeout(MILLISECONDS), MILLISECONDS))
+ shards.addAll(fetched);
+ }
+ catch (InterruptedException | ExecutionException | TimeoutException e)
+ {
+ throw new RuntimeException("Failed to fetch shards to seal", e);
+ }
+ return shards;
+ }
+
+ /**
+ * Discover all shards that need to be sealed at FINISH_JOIN or FINISH_REPLACE.
+ */
+ private static Set<ShardMetadata> discoverShardsAtFinishJoinOrReplace(ClusterMetadata metadata, PlacementDeltas finishDelta, @Nullable NodeId deadNode)
+ {
+ long finishEpoch = metadata.epoch.getEpoch();
+ Set<ShardMetadata> shards = new HashSet<>();
+ for (KeyspaceMetadata ksm : metadata.schema.getKeyspaces())
+ {
+ if (!ksm.params.replicationType.isTracked())
+ continue;
+
+ DataPlacement placement = metadata.placements.get(ksm.params.replication);
+ finishDelta.get(ksm.params.replication).writes.removals.flattenValues().forEach(removed -> {
+ Set<Integer> nodeIDs = new HashSet<>();
+ for (InetAddressAndPort ep : placement.writes.forRange(removed.range()).endpoints())
+ nodeIDs.add(metadata.directory.peerId(ep).id());
+ nodeIDs.add(deadNode != null ? deadNode.id() : metadata.directory.peerId(removed.endpoint()).id());
+ Participants participants = new Participants(nodeIDs);
+
+ MutationTrackingService.instance().forEachShardInKeyspace(ksm.name, shard -> {
+ if (shard.sinceEpoch < finishEpoch
+ && shard.range.equals(removed.range())
+ && shard.participants.equals(participants)
+ && !shard.isSealed())
+ shards.add(new ShardMetadata(shard.keyspace, shard.sinceEpoch, shard.range, shard.participants));
+ });
+ });
+ }
+ return shards;
+ }
+
+ /*
+ * UNBOOTSTRAP and REMOVENODE sequences illustrated (same sequence as far as sharding is concerned).
+ * Stages in which write placements remain unchanged, and only read placements change, are skipped
+ * here as irrelevant to sharding / sealing.
+ *
+ * There is a tracked keyspace with RF=3.
+ * A node n40 with token=40 is removed in the middle of the ring.
+ *
+ * 1. Initial state at e0:
+ *
+ * (10,20] | {n20,n30,n40} | e0 | ACTIVE
+ * (20,30] | {n30,n40,n50} | e0 | ACTIVE
+ * (30,40] | {n40,n50,n60} | e0 | ACTIVE
+ * (40,50] | {n50,n60,n70} | e0 | ACTIVE
+ *
+ * 2. START_LEAVE at e1:
+ *
+ * (10,20] | {n20,n30,n40} | e0 | ACTIVE
+ * + (10,20] | {n20,n30,n40,n50} | e1 | ACTIVE
+ * (20,30] | {n30,n40,n50} | e0 | ACTIVE
+ * + (20,30] | {n30,n40,n50,n60} | e1 | ACTIVE
+ * (30,40] | {n40,n50,n60} | e0 | ACTIVE
+ * + (30,40] | {n40,n50,n60,n70} | e1 | ACTIVE
+ * (40,50] | {n50,n60,n70} | e0 | ACTIVE
+ *
+ * 3. MID_LEAVE at e2 [*]:
+ *
+ * S (10,20] | {n20,n30,n40} | e0 | SEALED
+ * (10,20] | {n20,n30,n40,n50} | e1 | ACTIVE
+ * S (20,30] | {n30,n40,n50} | e0 | SEALED
+ * (20,30] | {n30,n40,n50,n60} | e1 | ACTIVE
+ * S (30,40] | {n40,n50,n60} | e0 | SEALED
+ * (30,40] | {n40,n50,n60,n70} | e1 | ACTIVE
+ * (40,50] | {n50,n60,n70} | e0 | ACTIVE
+ *
+ * 4. FINISH_LEAVE at e3:
+ *
+ * (10,20] | {n20,n30,n40} | e0 | SEALED
+ * S (10,20] | {n20,n30,n40,n50} | e1 | SEALED
+ * + (10,20] | {n20,n30,n50} | e3 | ACTIVE
+ * (20,30] | {n30,n40,n50} | e0 | SEALED
+ * S (20,30] | {n30,n40,n50,n60} | e1 | SEALED
+ * + (20,30] | {n30,n50,n60} | e3 | ACTIVE
+ * (30,40] | {n40,n50,n60} | e0 | SEALED
+ * S (30,40] | {n40,n50,n60,n70} | e1 | SEALED
+ * S (40,50] | {n50,n60,n70} | e0 | SEALED
+ * + (30,50] | {n50,n60,n70} | e3 | ACTIVE
+ *
+ * [*] At this stage, after sealing the shards, we stream the sstables.
+ * All the offsets belonging to these sealed shards get stripped from
+ * stats component of streamed sstables. Otherwise, the new node
+ * won't be able to compact them or mark them as reconciled/repaired -
+ * it is rightfully unaware of the sealed shards.
+ */
+
+ /**
+ * Seal the shards obsoleted by START_LEAVE before streaming, for both UNBOOTSTRAP and REMOVENODE.
+ */
+ public static void sealShardsAtMidLeave(ClusterMetadata metadata, PlacementDeltas startDelta, NodeId leavingOrDeparted, LeaveStreams.Kind kind)
+ {
+ InetAddressAndPort leavingOrDepartedEndpoint = metadata.directory.endpoint(leavingOrDeparted);
+
+ List<Future<Set<ShardMetadata>>> futures = new ArrayList<>();
+ for (KeyspaceMetadata ksm : metadata.schema.getKeyspaces())
+ {
+ if (!ksm.params.replicationType.isTracked())
+ continue;
+
+ DataPlacement placement = metadata.placements.get(ksm.params.replication);
+ startDelta.get(ksm.params.replication).writes.additions.flattenValues().forEach(added -> {
+ Range<Token> range = added.range();
+ long beforeEpoch = placement.writes.forRange(range).lastModified().getEpoch();
+ Set<InetAddressAndPort> survivors = new HashSet<>(placement.writes.forRange(range).endpoints());
+ survivors.remove(leavingOrDepartedEndpoint);
+ futures.add(FetchShards.fetch(ksm.name, beforeEpoch, range, survivors));
+ });
+ }
+
+ Set<ShardMetadata> shards = new HashSet<>();
+ try
+ {
+ for (Set<ShardMetadata> fetched : FutureCombiner.allOf(futures).get(DatabaseDescriptor.getRpcTimeout(MILLISECONDS), MILLISECONDS))
+ for (ShardMetadata shard : fetched)
+ if (shard.participants.contains(leavingOrDeparted.id()))
+ shards.add(shard);
+ }
+ catch (InterruptedException | ExecutionException | TimeoutException e)
+ {
+ throw new RuntimeException("Failed to fetch shards to seal for leave", e);
+ }
+
+ seal(shards, kind == LeaveStreams.Kind.UNBOOTSTRAP ? null : leavingOrDeparted);
+ }
+
+ /**
+ * Seal the shards that need to be sealed at LEAVE_FINISH for both UNBOOTSTRAP and REMOVENODE.
+ */
+ public static void sealShardsAtFinishLeave(ClusterMetadata postFinish, PlacementDeltas finishDelta, NodeId leavingOrDeparted, LeaveStreams.Kind kind)
+ {
+ long finishEpoch = postFinish.epoch.getEpoch();
+ InetAddressAndPort leavingOrDepartedEndpoint = postFinish.directory.endpoint(leavingOrDeparted);
+
+ List<Future<Set<ShardMetadata>>> futures = new ArrayList<>();
+ for (KeyspaceMetadata ksm : postFinish.schema.getKeyspaces())
+ {
+ if (!ksm.params.replicationType.isTracked())
+ continue;
+
+ DataPlacement placement = postFinish.placements.get(ksm.params.replication);
+ RangesAtEndpoint removals = finishDelta.get(ksm.params.replication).writes.removals.get(leavingOrDepartedEndpoint);
+ for (Replica removedReplica : removals)
+ {
+ // FINISH_LEAVE merges the leavingOrDeparted node's (pre-merge) range with an adjacent one. Query the *merged*
+ // range (not the pre-merge range) so discovery covers both the leavingOrDeparted node's own obsoleted
+ // generations and the merge-half - the adjacent pre-merge range folded into the merged range, whose
+ // shard never listed the leavingOrDeparted node. matchRange() yields the containing merged group + its
+ // current (survivor) write endpoints.
+ VersionedEndpoints.ForRange group = placement.writes.matchRange(removedReplica.range());
+ Range<Token> mergedRange = group.range();
+ Set<InetAddressAndPort> survivors = new HashSet<>(group.endpoints());
+ survivors.remove(leavingOrDepartedEndpoint);
+ futures.add(FetchShards.fetch(ksm.name, finishEpoch, mergedRange, survivors));
+ }
+ }
+
+ Set<ShardMetadata> shards = new HashSet<>();
+ try
+ {
+ for (Set<ShardMetadata> fetched : FutureCombiner.allOf(futures).get(DatabaseDescriptor.getRpcTimeout(MILLISECONDS), MILLISECONDS))
+ shards.addAll(fetched);
+ }
+ catch (InterruptedException | ExecutionException | TimeoutException e)
+ {
+ throw new RuntimeException("Failed to fetch shards to seal for leave finish", e);
+ }
+
+ seal(shards, kind == LeaveStreams.Kind.UNBOOTSTRAP ? null : leavingOrDeparted);
+ }
+
+ /*
+ * Sealing logic.
+ */
+
+ private static void seal(Set<ShardMetadata> shards)
+ {
+ seal(shards, null);
+ }
+
+ private static void seal(Set<ShardMetadata> shards, @Nullable NodeId withoutNode)
+ {
+ initiate(shards, withoutNode); // ACTIVE -> SEALING for each Shard
+ drain(shards, withoutNode); // drain in-flight local writes
+ reconcile(shards, withoutNode); // wait for logs to reconcile
+ complete(shards, withoutNode); // SEALING-> SEALED for each Shard + journal flush
+ }
+
+ private static void initiate(Set<ShardMetadata> shards, @Nullable NodeId withoutNode)
+ {
+ List<Future<Void>> futures = new ArrayList<>(shards.size());
+ for (ShardMetadata shard : shards)
+ futures.add(initiate(shard, withoutNode));
+ try
+ {
+ FutureCombiner.allOf(futures).get(DatabaseDescriptor.getRpcTimeout(MILLISECONDS), MILLISECONDS);
+ }
+ catch (InterruptedException | ExecutionException | TimeoutException e)
+ {
+ throw new RuntimeException("Failed to initiate sealing", e);
+ }
+ }
+
+ /**
+ * Fence allocation on every participant of the obsoleted shard (ACTIVE -> SEALING) by marking the shard
+ * SEALING on each.
+ * TODO (expected): this assumes every participant has already enacted the topology change that obsoleted the shard (so
+ * it no longer allocates new ids on it). The ProgressBarrier gating bootstrap()/MID_JOIN only waits for
+ * EACH_QUORUM of the affected replicas, NOT all live replicas, so a live-but-lagging participant could still
+ * be on the old shard. Bringing each participant up to the seal epoch must be done per-replica (not
+ * per-shard) and will be added separately. TLDR - add a step to guaranteed up-to-speed rest of replicas epochs.
+ */
+ private static AsyncPromise<Void> initiate(ShardMetadata shard, @Nullable NodeId withoutNode)
+ {
+ return InitSealing.initiate(shard.keyspace, shard.sinceEpoch, shard.range, toEndpoints(shard.participants, withoutNode));
+ }
+
+ private static void drain(Set<ShardMetadata> shards, @Nullable NodeId withoutNode)
+ {
+ long deadlineNanos = nanoTime() + DatabaseDescriptor.getRpcTimeout(NANOSECONDS);
+ Set<ShardMetadata> pending = new HashSet<>(shards);
+ while (true)
+ {
+ // non-blocking poll: each participant reports its drain status immediately; retry until all drained
+ pending.removeIf(shard -> drain(shard, withoutNode));
+ if (pending.isEmpty())
+ return;
+ if (nanoTime() >= deadlineNanos)
+ throw new RuntimeException("Timed out draining shards; still pending: " + pending);
+ LockSupport.parkNanos(SECONDS.toNanos(1));
+ }
+ }
+
+ /**
+ * Poll every participant of the SEALING shard once for drain status; true iff all report that no mutation
+ * id allocated before the SEALING fence is still applying locally.
+ */
+ private static boolean drain(ShardMetadata shard, @Nullable NodeId withoutNode)
+ {
+ try
+ {
+ return Drain.poll(shard.keyspace, shard.sinceEpoch, shard.range, toEndpoints(shard.participants, withoutNode))
+ .get(DatabaseDescriptor.getRpcTimeout(MILLISECONDS), MILLISECONDS);
+ }
+ catch (InterruptedException | ExecutionException | TimeoutException e)
+ {
+ throw new RuntimeException(format("Failed to poll drain for shard %s", shard), e);
+ }
+ }
+
+ /**
+ * For each of the shards, capture the witnessed union then poll each participant until it has
+ * caught up and witnessed the entire offset union itself.
+ */
+ private static void reconcile(Set<ShardMetadata> shards, @Nullable NodeId withoutNode)
+ {
+ for (ShardMetadata shard : shards)
+ reconcile(shard, withoutNode);
+ }
+
+ private static void reconcile(ShardMetadata shard, @Nullable NodeId withoutNode)
+ {
+ // capture shard's witnessed offsets once
+ Log2OffsetsMap.Mutable offsets = captureWitnessedOffsets(shard, withoutNode);
+ // poll every participant until they've each witnessed the union of offsets
+ for (InetAddressAndPort endpoint : toEndpoints(shard.participants, withoutNode))
+ pollUntilWitnesses(shard, offsets, endpoint);
+ }
+
+ private static Log2OffsetsMap.Mutable captureWitnessedOffsets(ShardMetadata shard, @Nullable NodeId withoutNode)
+ {
+ List<InetAddressAndPort> endpoints = toEndpoints(shard.participants, withoutNode);
+ try
+ {
+ return ReconcileCapture.capture(shard.keyspace, shard.sinceEpoch, shard.range, endpoints)
+ .get(DatabaseDescriptor.getRpcTimeout(MILLISECONDS), MILLISECONDS);
+ }
+ catch (InterruptedException | ExecutionException | TimeoutException e)
+ {
+ throw new RuntimeException(format("Failed to capture witnessed offsets for shard %s", shard), e);
+ }
+ }
+
+ private static void pollUntilWitnesses(ShardMetadata shard, Log2OffsetsMap.Mutable offsets, InetAddressAndPort endpoint)
+ {
+ // TODO (expected): use a longer timeout
+ long deadlineNanos = nanoTime() + DatabaseDescriptor.getRpcTimeout(NANOSECONDS);
+
+ while (true)
+ {
+ if (nanoTime() >= deadlineNanos)
+ throw new RuntimeException("Timed out reconciling shard: " + shard);
+ try
+ {
+ boolean witnessed =
+ ReconcilePoll.poll(shard.keyspace, shard.sinceEpoch, shard.range, offsets, endpoint)
+ .get(DatabaseDescriptor.getRpcTimeout(MILLISECONDS), MILLISECONDS);
+ if (witnessed)
+ return;
+ }
+ catch (InterruptedException | ExecutionException | TimeoutException e)
+ {
+ throw new RuntimeException(format("Failed to poll reconcile for shard %s", shard), e);
+ }
+
+ LockSupport.parkNanos(SECONDS.toNanos(1));
+ }
+ }
+
+ private static void complete(Set<ShardMetadata> shards, @Nullable NodeId withoutNode)
+ {
+ List<Future<Void>> futures = new ArrayList<>(shards.size());
+ for (ShardMetadata shard : shards)
+ futures.add(complete(shard, withoutNode));
+
+ try
+ {
+ FutureCombiner.allOf(futures).get(DatabaseDescriptor.getRpcTimeout(MILLISECONDS), MILLISECONDS);
+ }
+ catch (InterruptedException | ExecutionException | TimeoutException e)
+ {
+ throw new RuntimeException("Failed to complete sealing", e);
+ }
+ }
+
+ /**
+ * Promote every participant of the (drained and reconciled) shard from SEALING to SEALED.
+ */
+ private static AsyncPromise<Void> complete(ShardMetadata shard, @Nullable NodeId withoutNode)
+ {
+ return CompleteSealing.complete(shard.keyspace, shard.sinceEpoch, shard.range, toEndpoints(shard.participants, withoutNode));
+ }
+
+ /**
+ * A bootstrapping node knows nothing about the shards that currently exist,
+ * so it must collect the list of shards from the existing replicas before
+ * it can proceed with the rest of the sealing steps.
+ */
+ public static final class FetchShards
+ {
+ public static final class Request
+ {
+ final String keyspace;
+ final long beforeEpoch;
+ final Range<Token> range;
+
+ Request(String keyspace, long beforeEpoch, Range<Token> range)
+ {
+ this.keyspace = keyspace;
+ this.beforeEpoch = beforeEpoch;
+ this.range = range;
+ }
+ }
+
+ public static final class Response
+ {
+ final Set<ShardMetadata> shards;
+
+ Response(Set<ShardMetadata> shards)
+ {
+ this.shards = shards;
+ }
+ }
+
+ /**
+ * Query all replicas in a group in parallel for the shards over {@code ranges} obsoleted by an
+ * in-flight topology change (those with {@code sinceEpoch < beforeEpoch}). Resembles
+ * {@link ShardMetadataRequest#queryPeers}, but waits for every (live) replica to respond and
+ * merges their deduplicated shards rather than taking the first response.
+ */
+ public static AsyncPromise<Set<ShardMetadata>> fetch(
+ String keyspace, long beforeEpoch, Range<Token> range, Set<InetAddressAndPort> endpoints)
+ {
+ Preconditions.checkArgument(!endpoints.isEmpty());
+ AsyncPromise<Set<ShardMetadata>> promise = new AsyncPromise<>();
+ Set<ShardMetadata> merged = ConcurrentHashMap.newKeySet();
+
+ RequestCallback<Response> callback = new RequestCallback<>()
+ {
+ private final AtomicInteger remaining = new AtomicInteger(endpoints.size());
+
+ @Override
+ public void onResponse(Message<Response> msg)
+ {
+ merged.addAll(msg.payload.shards);
+ if (remaining.decrementAndGet() == 0)
+ promise.trySuccess(merged);
+ }
+
+ @Override
+ public void onFailure(InetAddressAndPort from, RequestFailure failure)
+ {
+ // happy path assumes all replicas are up; fail the whole fetch if any can't respond
+ promise.tryFailure(new RuntimeException(format("Failed to fetch shards to seal from %s for keyspace %s: %s", from, keyspace, failure)));
+ }
+
+ @Override
+ public boolean invokeOnFailure()
+ {
+ return true;
+ }
+ };
+
+ Message<Request> message = Message.out(Verb.MT_FETCH_SHARDS_REQ, new Request(keyspace, beforeEpoch, range));
+ for (InetAddressAndPort peer : endpoints)
+ MessagingService.instance().sendWithCallback(message, peer, callback);
+
+ return promise;
+ }
+
+ public static final IVerbHandler<Request> verbHandler = message ->
+ {
+ MutationTrackingService.ensureEnabled();
+ Request request = message.payload;
+ Set<ShardMetadata> shards = new HashSet<>();
+ MutationTrackingService.instance().forEachShardInKeyspace(request.keyspace, shard -> {
+ if (shard.sinceEpoch < request.beforeEpoch
+ && (request.range.contains(shard.range) || shard.range.contains(request.range))
+ && !shard.isSealed())
+ shards.add(new ShardMetadata(shard.keyspace, shard.sinceEpoch, shard.range, shard.participants));
+ });
+ MessagingService.instance().send(message.responseWith(new Response(shards)), message.from());
+ };
+
+ public static final VersionedSerializer<Request> requestSerializer = new VersionedSerializer<>()
+ {
+ @Override
+ public void serialize(Request r, DataOutputPlus out, Version version) throws IOException
+ {
+ out.writeUTF(r.keyspace);
+ out.writeLong(r.beforeEpoch);
+ AbstractBounds.tokenSerializer.serialize(r.range, out, version.messagingVersion());
+ }
+
+ @Override
+ public Request deserialize(DataInputPlus in, Version version) throws IOException
+ {
+ String keyspace = in.readUTF();
+ long beforeEpoch = in.readLong();
+ Range<Token> range = (Range<Token>) AbstractBounds.tokenSerializer.deserialize(in, IPartitioner.global(), version.messagingVersion());
+ return new Request(keyspace, beforeEpoch, range);
+ }
+
+ @Override
+ public long serializedSize(Request r, Version version)
+ {
+ long size = TypeSizes.sizeof(r.keyspace);
+ size += TypeSizes.sizeof(r.beforeEpoch);
+ size += AbstractBounds.tokenSerializer.serializedSize(r.range, version.messagingVersion());
+ return size;
+ }
+ };
+
+ public static final VersionedSerializer<Response> responseSerializer = new VersionedSerializer<>()
+ {
+ @Override
+ public void serialize(Response r, DataOutputPlus out, Version version) throws IOException
+ {
+ CollectionSerializers.serializeCollection(r.shards, out, version, ShardMetadata.serializer);
+ }
+
+ @Override
+ public Response deserialize(DataInputPlus in, Version version) throws IOException
+ {
+ return new Response(CollectionSerializers.deserializeSet(in, version, ShardMetadata.serializer));
+ }
+
+ @Override
+ public long serializedSize(Response r, Version version)
+ {
+ return CollectionSerializers.serializedCollectionSize(r.shards, version, ShardMetadata.serializer);
+ }
+ };
+ }
+
+ public static final class InitSealing
+ {
+ public static final class Request
+ {
+ final String keyspace;
+ final long sinceEpoch;
+ final Range<Token> range;
+
+ Request(String keyspace, long sinceEpoch, Range<Token> range)
+ {
+ this.keyspace = keyspace;
+ this.sinceEpoch = sinceEpoch;
+ this.range = range;
+ }
+ }
+
+ // cannot use NoPayload, because its serializer cannot be wrapped inside mtEmbedded()
+ public static final class Response
+ {
+ private static final Response instance = new Response();
+ }
+
+ /**
+ * Tell every participant of an obsoleted shard to fence id allocation (mark it SEALING)
+ */
+ public static AsyncPromise<Void> initiate(
+ String keyspace, long sinceEpoch, Range<Token> range, List<InetAddressAndPort> endpoints)
+ {
Review Comment:
```suggestion
{
Preconditions.checkArgument(!endpoints.isEmpty());
```
##########
src/java/org/apache/cassandra/replication/SealingCoordinator.java:
##########
@@ -0,0 +1,1401 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.cassandra.replication;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.TimeoutException;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.concurrent.locks.LockSupport;
+
+import javax.annotation.Nonnull;
+import javax.annotation.Nullable;
+
+import com.google.common.base.Preconditions;
+
+import org.apache.cassandra.config.DatabaseDescriptor;
+import org.apache.cassandra.db.ColumnFamilyStore;
+import org.apache.cassandra.db.Keyspace;
+import org.apache.cassandra.db.TypeSizes;
+import org.apache.cassandra.dht.AbstractBounds;
+import org.apache.cassandra.dht.IPartitioner;
+import org.apache.cassandra.dht.Range;
+import org.apache.cassandra.dht.Token;
+import org.apache.cassandra.exceptions.RequestFailure;
+import org.apache.cassandra.io.UnversionedSerializer;
+import org.apache.cassandra.io.util.DataInputPlus;
+import org.apache.cassandra.io.util.DataOutputPlus;
+import org.apache.cassandra.locator.EndpointsByReplica;
+import org.apache.cassandra.locator.EndpointsForRange;
+import org.apache.cassandra.locator.InetAddressAndPort;
+import org.apache.cassandra.locator.RangesAtEndpoint;
+import org.apache.cassandra.locator.Replica;
+import org.apache.cassandra.net.IVerbHandler;
+import org.apache.cassandra.net.Message;
+import org.apache.cassandra.net.MessagingService;
+import org.apache.cassandra.net.RequestCallback;
+import org.apache.cassandra.net.Verb;
+import org.apache.cassandra.schema.KeyspaceMetadata;
+import org.apache.cassandra.tcm.ClusterMetadata;
+import org.apache.cassandra.tcm.membership.NodeId;
+import org.apache.cassandra.tcm.ownership.DataPlacement;
+import org.apache.cassandra.tcm.ownership.MovementMap;
+import org.apache.cassandra.tcm.ownership.PlacementDeltas;
+import org.apache.cassandra.tcm.ownership.VersionedEndpoints;
+import org.apache.cassandra.tcm.sequences.LeaveStreams;
+import org.apache.cassandra.utils.CollectionSerializers;
+import org.apache.cassandra.utils.FBUtilities;
+import org.apache.cassandra.utils.concurrent.AsyncPromise;
+import org.apache.cassandra.utils.concurrent.Future;
+import org.apache.cassandra.utils.concurrent.FutureCombiner;
+
+import static java.lang.String.format;
+import static java.util.concurrent.TimeUnit.MILLISECONDS;
+import static java.util.concurrent.TimeUnit.NANOSECONDS;
+import static java.util.concurrent.TimeUnit.SECONDS;
+import static org.apache.cassandra.utils.Clock.Global.nanoTime;
+
+/*
+ * Sealing {@link Shard}s once a topology change obsoletes them.
+ *
+ * Happy Path Sequence:
+ * 1. Bring every participant up to (at least) this seal's epoch and fence allocation, so that everyone
+ * stops writing to the old shards; then wait until all in-progress mutations have been done.
+ * 2. Same for tracked transfers
+ * 3. Wait until all participating replicas have reconciled the union of their witnessed ids
+ * 4. Mark the shard as fully sealed
+ * 5. Stop including the shard in mutation summaries for read requests.
+ *
+ * Unhappy Path Sequence (not yet implemented):
+ * 1. Wait until everyone who's up stops writing to the old shards and all in-progress mutations have been done
+ * 2. Same for tracked transfers
+ * 3. Wait until all *live* participating replicas have reconciled the union of their witnessed ids
+ * 4. Mark the shard as partially sealed, label it with the set of participants that were able to seal
+ * 5. When a down node comes up, have it talk to other nodes and learn about sealing, and what mutations/logs
+ * they have exchanged. Diff with what we have - in system tables and in the journal (though system table
+ * plus local replay should give us an up-to-date metadata state for the shard)
+ */
+public final class SealingCoordinator
+{
+ /*
+ * BOOTSTRAP sequence illustrated. Stages in which write placements
+ * remain unchanged, and only read placements change, are skipped
+ * here as irrelevant to sharding / sealing.
+ *
+ * There is a tracked keyspace with RF=3.
+ * A node n35 with token=35 is added in the middle of the ring.
+ *
+ * 1. Initial state at e0:
+ *
+ * (10,20] | {n20,n30,n40} | e0 | ACTIVE
+ * (20,30] | {n30,n40,n50} | e0 | ACTIVE
+ * (30,40] | {n40,n50,n60} | e0 | ACTIVE
+ *
+ * 2. PREPARE_JOIN at e1:
+ *
+ * (10,20] | {n20,n30,n40} | e0 | ACTIVE
+ * (20,30] | {n30,n40,n50} | e0 | ACTIVE
+ * (30,40] | {n40,n50,n60} | e0 | ACTIVE
+ * + (30,35] | {n40,n50,n60} | e1 | ACTIVE
+ * + (35,40] | {n40,n50,n60} | e1 | ACTIVE
+ *
+ * 3. START_JOIN at e2:
+ *
+ * (10,20] | {n20,n30,n40} | e0 | ACTIVE
+ * + (10,20] | {n20,n30,n35,n40} | e2 | ACTIVE
+ * (20,30] | {n30,n40,n50} | e0 | ACTIVE
+ * + (20,30] | {n30,n35,n40,n50} | e2 | ACTIVE
+ * (30,40] | {n40,n50,n60} | e0 | ACTIVE
+ * (30,35] | {n40,n50,n60} | e1 | ACTIVE
+ * + (30,35] | {n35,n40,n50,n60} | e2 | ACTIVE
+ * (35,40] | {n40,n50,n60} | e1 | ACTIVE
+ *
+ * 4. MID_JOIN at e3 [*]:
+ *
+ * S (10,20] | {n20,n30,n40} | e0 | SEALED
+ * (10,20] | {n20,n30,n35,n40} | e2 | ACTIVE
+ * S (20,30] | {n30,n40,n50} | e0 | SEALED
+ * (20,30] | {n30,n35,n40,n50} | e2 | ACTIVE
+ * S (30,40] | {n40,n50,n60} | e0 | SEALED
+ * S (30,35] | {n40,n50,n60} | e1 | SEALED
+ * (30,35] | {n35,n40,n50,n60} | e2 | ACTIVE
+ * (35,40] | {n40,n50,n60} | e1 | ACTIVE
+ *
+ * 5. FINISH_JOIN at e4:
+ *
+ * (10,20] | {n20,n30,n40} | e0 | SEALED
+ * S (10,20] | {n20,n30,n35,n40} | e2 | SEALED
+ * + (10,20] | {n20,n30,n35} | e4 | ACTIVE
+ * (20,30] | {n30,n40,n50} | e0 | SEALED
+ * S (20,30] | {n30,n35,n40,n50} | e2 | SEALED
+ * + (20,30] | {n30,n35,n40} | e4 | ACTIVE
+ * (30,40] | {n40,n50,n60} | e0 | SEALED
+ * (30,35] | {n40,n50,n60} | e1 | SEALED
+ * S (30,35] | {n35,n40,n50,n60} | e2 | SEALED
+ * + (30,35] | {n35,n40,n50} | e4 | ACTIVE
+ * (35,40] | {n40,n50,n60} | e1 | ACTIVE
+ *
+ * [*] At this stage, after sealing the shards, we stream the sstables.
+ * All the offsets belonging to these sealed shards get stripped from
+ * stats component of streamed sstables. Otherwise, the new node
+ * won't be able to compact them or mark them as reconciled/repaired -
+ * it is rightfully unaware of the sealed shards.
+ */
+
+ /**
+ * Seal the shards obsoleted by bootstrap before the new node starts streaming.
+ */
+ public static void sealShardsAtMidJoin(ClusterMetadata metadata, MovementMap movements)
+ {
+ seal(discoverShardsAtMidJoinOrReplace(metadata, movements));
+ }
+
+ /**
+ * Seal the intermediate shards that were created during START_JOIN.
+ * TODO (required): this MUST eventually run while the affected ranges are still locked - between the
+ * removeFromWrites placement change and the range unlock. FinishJoin/FinishReplace currently apply
+ * removeFromWrites and the unlock atomically in a single transformation, so there is no committed
+ * dropped-but-locked epoch to hook. Until then, this runs just after FINISH_JOIN/FINISH_REPLACE,
+ * when the ranges are already unlocked. Left to implement for the unhappy path (AY).
+ */
+ public static void sealShardsAtFinishJoin(ClusterMetadata metadata, PlacementDeltas finishDelta)
+ {
+ seal(discoverShardsAtFinishJoinOrReplace(metadata, finishDelta, null));
+ }
+
+ /*
+ * REPLACE sequence illustrated. Stages in which write placements
+ * remain unchanged, and only read placements change, are skipped
+ * here as irrelevant to sharding / sealing.
+ *
+ * There is a tracked keyspace with RF=3.
+ * Node n40 is dead, and is being replaced with $n40.
+ *
+ * 1. Initial state at e0:
+ *
+ * (10,20] | {n20,n30,n40} | e0 | ACTIVE
+ * (20,30] | {n30,n40,n50} | e0 | ACTIVE
+ * (30,40] | {n40,n50,n60} | e0 | ACTIVE
+ *
+ * 2. START_REPLACE at e1:
+ *
+ * (10,20] | {n20,n30,n40} | e0 | ACTIVE
+ * + (10,20] | {n20,n30,n40,$n40}| e1 | ACTIVE
+ * (20,30] | {n30,n40,n50} | e0 | ACTIVE
+ * + (20,30] | {n30,n40,$n40,n50}| e1 | ACTIVE
+ * (30,40] | {n40,n50,n60} | e0 | ACTIVE
+ * + (30,40] | {n40,$n40,n50,n60}| e1 | ACTIVE
+ *
+ * 3. MID_REPLACE at e2 [*]:
+ *
+ * S (10,20] | {n20,n30,n40} | e0 | SEALED
+ * (10,20] | {n20,n30,n40,$n40}| e1 | ACTIVE
+ * S (20,30] | {n30,n40,n50} | e0 | SEALED
+ * (20,30] | {n30,n40,$n40,n50}| e1 | ACTIVE
+ * S (30,40] | {n40,n50,n60} | e0 | SEALED
+ * (30,40] | {n40,$n40,n50,n60}| e1 | ACTIVE
+ *
+ * 4. FINISH_REPLACE at e3:
+ *
+ * (10,20] | {n20,n30,n40} | e0 | SEALED
+ * S (10,20] | {n20,n30,n40,$n40}| e1 | SEALED
+ * + (10,20] | {n20,n30,$n40} | e3 | ACTIVE
+ * (20,30] | {n30,n40,n50} | e0 | SEALED
+ * S (20,30] | {n30,n40,$n40,n50}| e1 | SEALED
+ * + (20,30] | {n30,$n40,n50} | e3 | ACTIVE
+ * (30,40] | {n40,n50,n60} | e0 | SEALED
+ * S (30,40] | {n40,$n40,n50,n60}| e1 | SEALED
+ * + (30,40] | {$n40,n50,n60} | e3 | ACTIVE
+ *
+ * [*] At this stage, after sealing the shards, we stream the sstables.
+ * All the offsets belonging to these sealed shards get stripped from
+ * stats component of streamed sstables. Otherwise, the new node
+ * won't be able to compact them or mark them as reconciled/repaired -
+ * it is rightfully unaware of the sealed shards.
+ */
+
+ /**
+ * Seal the shards obsoleted by host replacement before the replacement node starts streaming.
+ * The {@code deadNode} node cannot participate in the sealing.
+ */
+ public static void sealShardsAtMidReplace(ClusterMetadata metadata, MovementMap movements, @Nonnull NodeId deadNode)
+ {
+ seal(discoverShardsAtMidJoinOrReplace(metadata, movements), deadNode);
+ }
+
+ public static void sealShardsAtFinishReplace(ClusterMetadata metadata, PlacementDeltas finishDelta, @Nonnull NodeId deadNode)
+ {
+ seal(discoverShardsAtFinishJoinOrReplace(metadata, finishDelta, deadNode), deadNode);
+ }
+
+ /**
+ * Discover all shards that need to be sealed at MID_JOIN or MID_REPLACE.
+ */
+ private static Set<ShardMetadata> discoverShardsAtMidJoinOrReplace(ClusterMetadata metadata, MovementMap movements)
+ {
+ List<Future<Set<ShardMetadata>>> futures = new ArrayList<>();
+
+ for (KeyspaceMetadata ksm : metadata.schema.getKeyspaces())
+ {
+ if (!ksm.params.replicationType.isTracked())
+ continue;
+
+ EndpointsByReplica acquired = movements.get(ksm.params.replication);
+ if (acquired.isEmpty()) // when the keyspace is not replicated to this DC
+ continue;
+
+ DataPlacement placement = metadata.placements.get(ksm.params.replication);
+
+ for (Map.Entry<Replica, EndpointsForRange> entry : acquired.entrySet())
+ {
+ Range<Token> range = entry.getKey().range();
+ Set<InetAddressAndPort> endpoints = entry.getValue().endpoints();
+ // cutoff: the epoch this node was added to the write placement for the range;
+ // shards over the range with sinceEpoch < cutoff are obsoleted by this bootstrap or replace.
+ long beforeEpoch = placement.writes.forRange(range).lastModified().getEpoch();
+ futures.add(FetchShards.fetch(ksm.name, beforeEpoch, range, endpoints));
+ }
+ }
+
+ Set<ShardMetadata> shards = new HashSet<>();
+ try
+ {
+ for (Set<ShardMetadata> fetched : FutureCombiner.allOf(futures).get(DatabaseDescriptor.getRpcTimeout(MILLISECONDS), MILLISECONDS))
+ shards.addAll(fetched);
+ }
+ catch (InterruptedException | ExecutionException | TimeoutException e)
+ {
+ throw new RuntimeException("Failed to fetch shards to seal", e);
+ }
+ return shards;
+ }
+
+ /**
+ * Discover all shards that need to be sealed at FINISH_JOIN or FINISH_REPLACE.
+ */
+ private static Set<ShardMetadata> discoverShardsAtFinishJoinOrReplace(ClusterMetadata metadata, PlacementDeltas finishDelta, @Nullable NodeId deadNode)
+ {
+ long finishEpoch = metadata.epoch.getEpoch();
+ Set<ShardMetadata> shards = new HashSet<>();
+ for (KeyspaceMetadata ksm : metadata.schema.getKeyspaces())
+ {
+ if (!ksm.params.replicationType.isTracked())
+ continue;
+
+ DataPlacement placement = metadata.placements.get(ksm.params.replication);
+ finishDelta.get(ksm.params.replication).writes.removals.flattenValues().forEach(removed -> {
+ Set<Integer> nodeIDs = new HashSet<>();
+ for (InetAddressAndPort ep : placement.writes.forRange(removed.range()).endpoints())
+ nodeIDs.add(metadata.directory.peerId(ep).id());
+ nodeIDs.add(deadNode != null ? deadNode.id() : metadata.directory.peerId(removed.endpoint()).id());
+ Participants participants = new Participants(nodeIDs);
+
+ MutationTrackingService.instance().forEachShardInKeyspace(ksm.name, shard -> {
+ if (shard.sinceEpoch < finishEpoch
+ && shard.range.equals(removed.range())
+ && shard.participants.equals(participants)
+ && !shard.isSealed())
+ shards.add(new ShardMetadata(shard.keyspace, shard.sinceEpoch, shard.range, shard.participants));
+ });
+ });
+ }
+ return shards;
+ }
+
+ /*
+ * UNBOOTSTRAP and REMOVENODE sequences illustrated (same sequence as far as sharding is concerned).
+ * Stages in which write placements remain unchanged, and only read placements change, are skipped
+ * here as irrelevant to sharding / sealing.
+ *
+ * There is a tracked keyspace with RF=3.
+ * A node n40 with token=40 is removed in the middle of the ring.
+ *
+ * 1. Initial state at e0:
+ *
+ * (10,20] | {n20,n30,n40} | e0 | ACTIVE
+ * (20,30] | {n30,n40,n50} | e0 | ACTIVE
+ * (30,40] | {n40,n50,n60} | e0 | ACTIVE
+ * (40,50] | {n50,n60,n70} | e0 | ACTIVE
+ *
+ * 2. START_LEAVE at e1:
+ *
+ * (10,20] | {n20,n30,n40} | e0 | ACTIVE
+ * + (10,20] | {n20,n30,n40,n50} | e1 | ACTIVE
+ * (20,30] | {n30,n40,n50} | e0 | ACTIVE
+ * + (20,30] | {n30,n40,n50,n60} | e1 | ACTIVE
+ * (30,40] | {n40,n50,n60} | e0 | ACTIVE
+ * + (30,40] | {n40,n50,n60,n70} | e1 | ACTIVE
+ * (40,50] | {n50,n60,n70} | e0 | ACTIVE
+ *
+ * 3. MID_LEAVE at e2 [*]:
+ *
+ * S (10,20] | {n20,n30,n40} | e0 | SEALED
+ * (10,20] | {n20,n30,n40,n50} | e1 | ACTIVE
+ * S (20,30] | {n30,n40,n50} | e0 | SEALED
+ * (20,30] | {n30,n40,n50,n60} | e1 | ACTIVE
+ * S (30,40] | {n40,n50,n60} | e0 | SEALED
+ * (30,40] | {n40,n50,n60,n70} | e1 | ACTIVE
+ * (40,50] | {n50,n60,n70} | e0 | ACTIVE
+ *
+ * 4. FINISH_LEAVE at e3:
+ *
+ * (10,20] | {n20,n30,n40} | e0 | SEALED
+ * S (10,20] | {n20,n30,n40,n50} | e1 | SEALED
+ * + (10,20] | {n20,n30,n50} | e3 | ACTIVE
+ * (20,30] | {n30,n40,n50} | e0 | SEALED
+ * S (20,30] | {n30,n40,n50,n60} | e1 | SEALED
+ * + (20,30] | {n30,n50,n60} | e3 | ACTIVE
+ * (30,40] | {n40,n50,n60} | e0 | SEALED
+ * S (30,40] | {n40,n50,n60,n70} | e1 | SEALED
+ * S (40,50] | {n50,n60,n70} | e0 | SEALED
+ * + (30,50] | {n50,n60,n70} | e3 | ACTIVE
+ *
+ * [*] At this stage, after sealing the shards, we stream the sstables.
+ * All the offsets belonging to these sealed shards get stripped from
+ * stats component of streamed sstables. Otherwise, the new node
+ * won't be able to compact them or mark them as reconciled/repaired -
+ * it is rightfully unaware of the sealed shards.
+ */
+
+ /**
+ * Seal the shards obsoleted by START_LEAVE before streaming, for both UNBOOTSTRAP and REMOVENODE.
+ */
+ public static void sealShardsAtMidLeave(ClusterMetadata metadata, PlacementDeltas startDelta, NodeId leavingOrDeparted, LeaveStreams.Kind kind)
+ {
+ InetAddressAndPort leavingOrDepartedEndpoint = metadata.directory.endpoint(leavingOrDeparted);
+
+ List<Future<Set<ShardMetadata>>> futures = new ArrayList<>();
+ for (KeyspaceMetadata ksm : metadata.schema.getKeyspaces())
+ {
+ if (!ksm.params.replicationType.isTracked())
+ continue;
+
+ DataPlacement placement = metadata.placements.get(ksm.params.replication);
+ startDelta.get(ksm.params.replication).writes.additions.flattenValues().forEach(added -> {
+ Range<Token> range = added.range();
+ long beforeEpoch = placement.writes.forRange(range).lastModified().getEpoch();
+ Set<InetAddressAndPort> survivors = new HashSet<>(placement.writes.forRange(range).endpoints());
+ survivors.remove(leavingOrDepartedEndpoint);
+ futures.add(FetchShards.fetch(ksm.name, beforeEpoch, range, survivors));
+ });
+ }
+
+ Set<ShardMetadata> shards = new HashSet<>();
+ try
+ {
+ for (Set<ShardMetadata> fetched : FutureCombiner.allOf(futures).get(DatabaseDescriptor.getRpcTimeout(MILLISECONDS), MILLISECONDS))
+ for (ShardMetadata shard : fetched)
+ if (shard.participants.contains(leavingOrDeparted.id()))
+ shards.add(shard);
+ }
+ catch (InterruptedException | ExecutionException | TimeoutException e)
+ {
+ throw new RuntimeException("Failed to fetch shards to seal for leave", e);
+ }
+
+ seal(shards, kind == LeaveStreams.Kind.UNBOOTSTRAP ? null : leavingOrDeparted);
+ }
+
+ /**
+ * Seal the shards that need to be sealed at LEAVE_FINISH for both UNBOOTSTRAP and REMOVENODE.
+ */
+ public static void sealShardsAtFinishLeave(ClusterMetadata postFinish, PlacementDeltas finishDelta, NodeId leavingOrDeparted, LeaveStreams.Kind kind)
+ {
+ long finishEpoch = postFinish.epoch.getEpoch();
+ InetAddressAndPort leavingOrDepartedEndpoint = postFinish.directory.endpoint(leavingOrDeparted);
+
+ List<Future<Set<ShardMetadata>>> futures = new ArrayList<>();
+ for (KeyspaceMetadata ksm : postFinish.schema.getKeyspaces())
+ {
+ if (!ksm.params.replicationType.isTracked())
+ continue;
+
+ DataPlacement placement = postFinish.placements.get(ksm.params.replication);
+ RangesAtEndpoint removals = finishDelta.get(ksm.params.replication).writes.removals.get(leavingOrDepartedEndpoint);
+ for (Replica removedReplica : removals)
+ {
+ // FINISH_LEAVE merges the leavingOrDeparted node's (pre-merge) range with an adjacent one. Query the *merged*
+ // range (not the pre-merge range) so discovery covers both the leavingOrDeparted node's own obsoleted
+ // generations and the merge-half - the adjacent pre-merge range folded into the merged range, whose
+ // shard never listed the leavingOrDeparted node. matchRange() yields the containing merged group + its
+ // current (survivor) write endpoints.
+ VersionedEndpoints.ForRange group = placement.writes.matchRange(removedReplica.range());
+ Range<Token> mergedRange = group.range();
+ Set<InetAddressAndPort> survivors = new HashSet<>(group.endpoints());
+ survivors.remove(leavingOrDepartedEndpoint);
+ futures.add(FetchShards.fetch(ksm.name, finishEpoch, mergedRange, survivors));
+ }
+ }
+
+ Set<ShardMetadata> shards = new HashSet<>();
+ try
+ {
+ for (Set<ShardMetadata> fetched : FutureCombiner.allOf(futures).get(DatabaseDescriptor.getRpcTimeout(MILLISECONDS), MILLISECONDS))
+ shards.addAll(fetched);
+ }
+ catch (InterruptedException | ExecutionException | TimeoutException e)
+ {
+ throw new RuntimeException("Failed to fetch shards to seal for leave finish", e);
+ }
+
+ seal(shards, kind == LeaveStreams.Kind.UNBOOTSTRAP ? null : leavingOrDeparted);
+ }
+
+ /*
+ * Sealing logic.
+ */
+
+ private static void seal(Set<ShardMetadata> shards)
+ {
+ seal(shards, null);
+ }
+
+ private static void seal(Set<ShardMetadata> shards, @Nullable NodeId withoutNode)
+ {
+ initiate(shards, withoutNode); // ACTIVE -> SEALING for each Shard
+ drain(shards, withoutNode); // drain in-flight local writes
+ reconcile(shards, withoutNode); // wait for logs to reconcile
+ complete(shards, withoutNode); // SEALING-> SEALED for each Shard + journal flush
+ }
+
+ private static void initiate(Set<ShardMetadata> shards, @Nullable NodeId withoutNode)
+ {
+ List<Future<Void>> futures = new ArrayList<>(shards.size());
+ for (ShardMetadata shard : shards)
+ futures.add(initiate(shard, withoutNode));
+ try
+ {
+ FutureCombiner.allOf(futures).get(DatabaseDescriptor.getRpcTimeout(MILLISECONDS), MILLISECONDS);
+ }
+ catch (InterruptedException | ExecutionException | TimeoutException e)
+ {
+ throw new RuntimeException("Failed to initiate sealing", e);
+ }
+ }
+
+ /**
+ * Fence allocation on every participant of the obsoleted shard (ACTIVE -> SEALING) by marking the shard
+ * SEALING on each.
+ * TODO (expected): this assumes every participant has already enacted the topology change that obsoleted the shard (so
+ * it no longer allocates new ids on it). The ProgressBarrier gating bootstrap()/MID_JOIN only waits for
+ * EACH_QUORUM of the affected replicas, NOT all live replicas, so a live-but-lagging participant could still
+ * be on the old shard. Bringing each participant up to the seal epoch must be done per-replica (not
+ * per-shard) and will be added separately. TLDR - add a step to guaranteed up-to-speed rest of replicas epochs.
+ */
+ private static AsyncPromise<Void> initiate(ShardMetadata shard, @Nullable NodeId withoutNode)
+ {
+ return InitSealing.initiate(shard.keyspace, shard.sinceEpoch, shard.range, toEndpoints(shard.participants, withoutNode));
+ }
+
+ private static void drain(Set<ShardMetadata> shards, @Nullable NodeId withoutNode)
+ {
+ long deadlineNanos = nanoTime() + DatabaseDescriptor.getRpcTimeout(NANOSECONDS);
+ Set<ShardMetadata> pending = new HashSet<>(shards);
+ while (true)
+ {
+ // non-blocking poll: each participant reports its drain status immediately; retry until all drained
+ pending.removeIf(shard -> drain(shard, withoutNode));
+ if (pending.isEmpty())
+ return;
+ if (nanoTime() >= deadlineNanos)
+ throw new RuntimeException("Timed out draining shards; still pending: " + pending);
+ LockSupport.parkNanos(SECONDS.toNanos(1));
+ }
+ }
+
+ /**
+ * Poll every participant of the SEALING shard once for drain status; true iff all report that no mutation
+ * id allocated before the SEALING fence is still applying locally.
+ */
+ private static boolean drain(ShardMetadata shard, @Nullable NodeId withoutNode)
+ {
+ try
+ {
+ return Drain.poll(shard.keyspace, shard.sinceEpoch, shard.range, toEndpoints(shard.participants, withoutNode))
+ .get(DatabaseDescriptor.getRpcTimeout(MILLISECONDS), MILLISECONDS);
+ }
+ catch (InterruptedException | ExecutionException | TimeoutException e)
+ {
+ throw new RuntimeException(format("Failed to poll drain for shard %s", shard), e);
+ }
+ }
+
+ /**
+ * For each of the shards, capture the witnessed union then poll each participant until it has
+ * caught up and witnessed the entire offset union itself.
+ */
+ private static void reconcile(Set<ShardMetadata> shards, @Nullable NodeId withoutNode)
+ {
+ for (ShardMetadata shard : shards)
+ reconcile(shard, withoutNode);
+ }
+
+ private static void reconcile(ShardMetadata shard, @Nullable NodeId withoutNode)
+ {
+ // capture shard's witnessed offsets once
+ Log2OffsetsMap.Mutable offsets = captureWitnessedOffsets(shard, withoutNode);
+ // poll every participant until they've each witnessed the union of offsets
+ for (InetAddressAndPort endpoint : toEndpoints(shard.participants, withoutNode))
+ pollUntilWitnesses(shard, offsets, endpoint);
+ }
+
+ private static Log2OffsetsMap.Mutable captureWitnessedOffsets(ShardMetadata shard, @Nullable NodeId withoutNode)
+ {
+ List<InetAddressAndPort> endpoints = toEndpoints(shard.participants, withoutNode);
+ try
+ {
+ return ReconcileCapture.capture(shard.keyspace, shard.sinceEpoch, shard.range, endpoints)
+ .get(DatabaseDescriptor.getRpcTimeout(MILLISECONDS), MILLISECONDS);
+ }
+ catch (InterruptedException | ExecutionException | TimeoutException e)
+ {
+ throw new RuntimeException(format("Failed to capture witnessed offsets for shard %s", shard), e);
+ }
+ }
+
+ private static void pollUntilWitnesses(ShardMetadata shard, Log2OffsetsMap.Mutable offsets, InetAddressAndPort endpoint)
+ {
+ // TODO (expected): use a longer timeout
+ long deadlineNanos = nanoTime() + DatabaseDescriptor.getRpcTimeout(NANOSECONDS);
+
+ while (true)
+ {
+ if (nanoTime() >= deadlineNanos)
+ throw new RuntimeException("Timed out reconciling shard: " + shard);
+ try
+ {
+ boolean witnessed =
+ ReconcilePoll.poll(shard.keyspace, shard.sinceEpoch, shard.range, offsets, endpoint)
+ .get(DatabaseDescriptor.getRpcTimeout(MILLISECONDS), MILLISECONDS);
+ if (witnessed)
+ return;
+ }
+ catch (InterruptedException | ExecutionException | TimeoutException e)
+ {
+ throw new RuntimeException(format("Failed to poll reconcile for shard %s", shard), e);
+ }
+
+ LockSupport.parkNanos(SECONDS.toNanos(1));
+ }
+ }
+
+ private static void complete(Set<ShardMetadata> shards, @Nullable NodeId withoutNode)
+ {
+ List<Future<Void>> futures = new ArrayList<>(shards.size());
+ for (ShardMetadata shard : shards)
+ futures.add(complete(shard, withoutNode));
+
+ try
+ {
+ FutureCombiner.allOf(futures).get(DatabaseDescriptor.getRpcTimeout(MILLISECONDS), MILLISECONDS);
+ }
+ catch (InterruptedException | ExecutionException | TimeoutException e)
+ {
+ throw new RuntimeException("Failed to complete sealing", e);
+ }
+ }
+
+ /**
+ * Promote every participant of the (drained and reconciled) shard from SEALING to SEALED.
+ */
+ private static AsyncPromise<Void> complete(ShardMetadata shard, @Nullable NodeId withoutNode)
+ {
+ return CompleteSealing.complete(shard.keyspace, shard.sinceEpoch, shard.range, toEndpoints(shard.participants, withoutNode));
+ }
+
+ /**
+ * A bootstrapping node knows nothing about the shards that currently exist,
+ * so it must collect the list of shards from the existing replicas before
+ * it can proceed with the rest of the sealing steps.
+ */
+ public static final class FetchShards
+ {
+ public static final class Request
+ {
+ final String keyspace;
+ final long beforeEpoch;
+ final Range<Token> range;
+
+ Request(String keyspace, long beforeEpoch, Range<Token> range)
+ {
+ this.keyspace = keyspace;
+ this.beforeEpoch = beforeEpoch;
+ this.range = range;
+ }
+ }
+
+ public static final class Response
+ {
+ final Set<ShardMetadata> shards;
+
+ Response(Set<ShardMetadata> shards)
+ {
+ this.shards = shards;
+ }
+ }
+
+ /**
+ * Query all replicas in a group in parallel for the shards over {@code ranges} obsoleted by an
+ * in-flight topology change (those with {@code sinceEpoch < beforeEpoch}). Resembles
+ * {@link ShardMetadataRequest#queryPeers}, but waits for every (live) replica to respond and
+ * merges their deduplicated shards rather than taking the first response.
+ */
+ public static AsyncPromise<Set<ShardMetadata>> fetch(
+ String keyspace, long beforeEpoch, Range<Token> range, Set<InetAddressAndPort> endpoints)
+ {
+ Preconditions.checkArgument(!endpoints.isEmpty());
+ AsyncPromise<Set<ShardMetadata>> promise = new AsyncPromise<>();
+ Set<ShardMetadata> merged = ConcurrentHashMap.newKeySet();
+
+ RequestCallback<Response> callback = new RequestCallback<>()
+ {
+ private final AtomicInteger remaining = new AtomicInteger(endpoints.size());
+
+ @Override
+ public void onResponse(Message<Response> msg)
+ {
+ merged.addAll(msg.payload.shards);
+ if (remaining.decrementAndGet() == 0)
+ promise.trySuccess(merged);
+ }
+
+ @Override
+ public void onFailure(InetAddressAndPort from, RequestFailure failure)
+ {
+ // happy path assumes all replicas are up; fail the whole fetch if any can't respond
+ promise.tryFailure(new RuntimeException(format("Failed to fetch shards to seal from %s for keyspace %s: %s", from, keyspace, failure)));
+ }
+
+ @Override
+ public boolean invokeOnFailure()
+ {
+ return true;
+ }
+ };
+
+ Message<Request> message = Message.out(Verb.MT_FETCH_SHARDS_REQ, new Request(keyspace, beforeEpoch, range));
+ for (InetAddressAndPort peer : endpoints)
+ MessagingService.instance().sendWithCallback(message, peer, callback);
+
+ return promise;
+ }
+
+ public static final IVerbHandler<Request> verbHandler = message ->
+ {
+ MutationTrackingService.ensureEnabled();
+ Request request = message.payload;
+ Set<ShardMetadata> shards = new HashSet<>();
+ MutationTrackingService.instance().forEachShardInKeyspace(request.keyspace, shard -> {
+ if (shard.sinceEpoch < request.beforeEpoch
+ && (request.range.contains(shard.range) || shard.range.contains(request.range))
+ && !shard.isSealed())
+ shards.add(new ShardMetadata(shard.keyspace, shard.sinceEpoch, shard.range, shard.participants));
+ });
+ MessagingService.instance().send(message.responseWith(new Response(shards)), message.from());
+ };
+
+ public static final VersionedSerializer<Request> requestSerializer = new VersionedSerializer<>()
+ {
+ @Override
+ public void serialize(Request r, DataOutputPlus out, Version version) throws IOException
+ {
+ out.writeUTF(r.keyspace);
+ out.writeLong(r.beforeEpoch);
+ AbstractBounds.tokenSerializer.serialize(r.range, out, version.messagingVersion());
+ }
+
+ @Override
+ public Request deserialize(DataInputPlus in, Version version) throws IOException
+ {
+ String keyspace = in.readUTF();
+ long beforeEpoch = in.readLong();
+ Range<Token> range = (Range<Token>) AbstractBounds.tokenSerializer.deserialize(in, IPartitioner.global(), version.messagingVersion());
+ return new Request(keyspace, beforeEpoch, range);
+ }
+
+ @Override
+ public long serializedSize(Request r, Version version)
+ {
+ long size = TypeSizes.sizeof(r.keyspace);
+ size += TypeSizes.sizeof(r.beforeEpoch);
+ size += AbstractBounds.tokenSerializer.serializedSize(r.range, version.messagingVersion());
+ return size;
+ }
+ };
+
+ public static final VersionedSerializer<Response> responseSerializer = new VersionedSerializer<>()
+ {
+ @Override
+ public void serialize(Response r, DataOutputPlus out, Version version) throws IOException
+ {
+ CollectionSerializers.serializeCollection(r.shards, out, version, ShardMetadata.serializer);
+ }
+
+ @Override
+ public Response deserialize(DataInputPlus in, Version version) throws IOException
+ {
+ return new Response(CollectionSerializers.deserializeSet(in, version, ShardMetadata.serializer));
+ }
+
+ @Override
+ public long serializedSize(Response r, Version version)
+ {
+ return CollectionSerializers.serializedCollectionSize(r.shards, version, ShardMetadata.serializer);
+ }
+ };
+ }
+
+ public static final class InitSealing
+ {
+ public static final class Request
+ {
+ final String keyspace;
+ final long sinceEpoch;
+ final Range<Token> range;
+
+ Request(String keyspace, long sinceEpoch, Range<Token> range)
+ {
+ this.keyspace = keyspace;
+ this.sinceEpoch = sinceEpoch;
+ this.range = range;
+ }
+ }
+
+ // cannot use NoPayload, because its serializer cannot be wrapped inside mtEmbedded()
+ public static final class Response
+ {
+ private static final Response instance = new Response();
+ }
+
+ /**
+ * Tell every participant of an obsoleted shard to fence id allocation (mark it SEALING)
+ */
+ public static AsyncPromise<Void> initiate(
+ String keyspace, long sinceEpoch, Range<Token> range, List<InetAddressAndPort> endpoints)
+ {
+ AsyncPromise<Void> promise = new AsyncPromise<>();
+
+ RequestCallback<Response> callback = new RequestCallback<>()
+ {
+ private final AtomicInteger remaining = new AtomicInteger(endpoints.size());
+
+ @Override
+ public void onResponse(Message<Response> msg)
+ {
+ if (remaining.decrementAndGet() == 0)
+ promise.trySuccess(null);
+ }
+
+ @Override
+ public void onFailure(InetAddressAndPort from, RequestFailure failure)
+ {
+ promise.tryFailure(new RuntimeException(format("Failed to initiate sealing on %s for keyspace %s: %s", from, keyspace, failure)));
+ }
+
+ @Override
+ public boolean invokeOnFailure()
+ {
+ return true;
+ }
+ };
+
+ Message<Request> message = Message.out(Verb.MT_INIT_SEALING_REQ, new Request(keyspace, sinceEpoch, range));
+ for (InetAddressAndPort peer : endpoints)
+ MessagingService.instance().sendWithCallback(message, peer, callback);
+
+ return promise;
+ }
+
+ public static final IVerbHandler<Request> verbHandler = message -> {
+ MutationTrackingService.ensureEnabled();
+ Request request = message.payload;
+ MutationTrackingService.instance().markShardSealing(request.keyspace, request.sinceEpoch, request.range);
+ MessagingService.instance().send(message.responseWith(Response.instance), message.from());
+ };
+
+ public static final VersionedSerializer<Request> requestSerializer = new VersionedSerializer<>()
+ {
+ @Override
+ public void serialize(Request r, DataOutputPlus out, Version version) throws IOException
+ {
+ out.writeUTF(r.keyspace);
+ out.writeLong(r.sinceEpoch);
+ AbstractBounds.tokenSerializer.serialize(r.range, out, version.messagingVersion());
+ }
+
+ @Override
+ public Request deserialize(DataInputPlus in, Version version) throws IOException
+ {
+ String keyspace = in.readUTF();
+ long sinceEpoch = in.readLong();
+ Range<Token> range = (Range<Token>) AbstractBounds.tokenSerializer.deserialize(in, IPartitioner.global(), version.messagingVersion());
+ return new Request(keyspace, sinceEpoch, range);
+ }
+
+ @Override
+ public long serializedSize(Request r, Version version)
+ {
+ long size = TypeSizes.sizeof(r.keyspace);
+ size += TypeSizes.sizeof(r.sinceEpoch);
+ size += AbstractBounds.tokenSerializer.serializedSize(r.range, version.messagingVersion());
+ return size;
+ }
+ };
+
+ public static final UnversionedSerializer<Response> responseSerializer = new UnversionedSerializer<>()
+ {
+ @Override
+ public void serialize(Response r, DataOutputPlus out)
+ {
+ }
+
+ @Override
+ public Response deserialize(DataInputPlus in)
+ {
+ return Response.instance;
+ }
+
+ @Override
+ public long serializedSize(Response r)
+ {
+ return 0;
+ }
+ };
+ }
+
+ public static final class Drain
+ {
+ public static final class Request
+ {
+ final String keyspace;
+ final long sinceEpoch;
+ final Range<Token> range;
+
+ Request(String keyspace, long sinceEpoch, Range<Token> range)
+ {
+ this.keyspace = keyspace;
+ this.sinceEpoch = sinceEpoch;
+ this.range = range;
+ }
+ }
+
+ public static final class Response
+ {
+ final boolean drained;
+
+ Response(boolean drained)
+ {
+ this.drained = drained;
+ }
+ }
+
+ /**
+ * Poll every participant of a SEALING shard for whether it has drained its in-flight local
+ * writes. Succeeds with {@code true} iff every participant reports drained.
+ */
+ public static AsyncPromise<Boolean> poll(
+ String keyspace, long sinceEpoch, Range<Token> range, List<InetAddressAndPort> endpoints)
+ {
+ AsyncPromise<Boolean> promise = new AsyncPromise<>();
+
+ RequestCallback<Response> callback = new RequestCallback<>()
+ {
+ private final AtomicInteger remaining = new AtomicInteger(endpoints.size());
+ private volatile boolean allDrained = true;
+
+ @Override
+ public void onResponse(Message<Response> msg)
+ {
+ if (!msg.payload.drained)
+ allDrained = false;
+ if (remaining.decrementAndGet() == 0)
+ promise.trySuccess(allDrained);
+ }
+
+ @Override
+ public void onFailure(InetAddressAndPort from, RequestFailure failure)
+ {
+ promise.tryFailure(new RuntimeException(format("Failed to poll drain on %s for keyspace %s: %s", from, keyspace, failure)));
+ }
+
+ @Override
+ public boolean invokeOnFailure()
+ {
+ return true;
+ }
+ };
+
+ Message<Request> message = Message.out(Verb.MT_DRAIN_REQ, new Request(keyspace, sinceEpoch, range));
+ for (InetAddressAndPort peer : endpoints)
+ MessagingService.instance().sendWithCallback(message, peer, callback);
+
+ return promise;
+ }
+
+ public static final IVerbHandler<Request> verbHandler = message -> {
+ MutationTrackingService.ensureEnabled();
+ Request request = message.payload;
+ boolean drained = MutationTrackingService.instance().isShardDrained(request.keyspace, request.sinceEpoch, request.range);
+ MessagingService.instance().send(message.responseWith(new Response(drained)), message.from());
+ };
+
+ public static final VersionedSerializer<Request> requestSerializer = new VersionedSerializer<>()
+ {
+ @Override
+ public void serialize(Request r, DataOutputPlus out, Version version) throws IOException
+ {
+ out.writeUTF(r.keyspace);
+ out.writeLong(r.sinceEpoch);
+ AbstractBounds.tokenSerializer.serialize(r.range, out, version.messagingVersion());
+ }
+
+ @Override
+ public Request deserialize(DataInputPlus in, Version version) throws IOException
+ {
+ String keyspace = in.readUTF();
+ long sinceEpoch = in.readLong();
+ Range<Token> range = (Range<Token>) AbstractBounds.tokenSerializer.deserialize(in, IPartitioner.global(), version.messagingVersion());
+ return new Request(keyspace, sinceEpoch, range);
+ }
+
+ @Override
+ public long serializedSize(Request r, Version version)
+ {
+ long size = TypeSizes.sizeof(r.keyspace);
+ size += TypeSizes.sizeof(r.sinceEpoch);
+ size += AbstractBounds.tokenSerializer.serializedSize(r.range, version.messagingVersion());
+ return size;
+ }
+ };
+
+ public static final UnversionedSerializer<Response> responseSerializer = new UnversionedSerializer<>()
+ {
+ @Override
+ public void serialize(Response r, DataOutputPlus out) throws IOException
+ {
+ out.writeBoolean(r.drained);
+ }
+
+ @Override
+ public Response deserialize(DataInputPlus in) throws IOException
+ {
+ return new Response(in.readBoolean());
+ }
+
+ @Override
+ public long serializedSize(Response r)
+ {
+ return TypeSizes.sizeof(r.drained);
+ }
+ };
+ }
+
+ public static final class ReconcileCapture
+ {
+ public static final class Request
+ {
+ final String keyspace;
+ final long sinceEpoch;
+ final Range<Token> range;
+
+ Request(String keyspace, long sinceEpoch, Range<Token> range)
+ {
+ this.keyspace = keyspace;
+ this.sinceEpoch = sinceEpoch;
+ this.range = range;
+ }
+ }
+
+ public static final class Response
+ {
+ final Log2OffsetsMap<?> witnessed;
+
+ Response(Log2OffsetsMap<?> witnessed)
+ {
+ this.witnessed = witnessed;
+ }
+ }
+
+ public static AsyncPromise<Log2OffsetsMap.Mutable> capture(
+ String keyspace, long sinceEpoch, Range<Token> range, List<InetAddressAndPort> endpoints)
+ {
Review Comment:
```suggestion
{
Preconditions.checkArgument(!endpoints.isEmpty());
```
##########
src/java/org/apache/cassandra/replication/SealingCoordinator.java:
##########
@@ -0,0 +1,1401 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.cassandra.replication;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.TimeoutException;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.concurrent.locks.LockSupport;
+
+import javax.annotation.Nonnull;
+import javax.annotation.Nullable;
+
+import com.google.common.base.Preconditions;
+
+import org.apache.cassandra.config.DatabaseDescriptor;
+import org.apache.cassandra.db.ColumnFamilyStore;
+import org.apache.cassandra.db.Keyspace;
+import org.apache.cassandra.db.TypeSizes;
+import org.apache.cassandra.dht.AbstractBounds;
+import org.apache.cassandra.dht.IPartitioner;
+import org.apache.cassandra.dht.Range;
+import org.apache.cassandra.dht.Token;
+import org.apache.cassandra.exceptions.RequestFailure;
+import org.apache.cassandra.io.UnversionedSerializer;
+import org.apache.cassandra.io.util.DataInputPlus;
+import org.apache.cassandra.io.util.DataOutputPlus;
+import org.apache.cassandra.locator.EndpointsByReplica;
+import org.apache.cassandra.locator.EndpointsForRange;
+import org.apache.cassandra.locator.InetAddressAndPort;
+import org.apache.cassandra.locator.RangesAtEndpoint;
+import org.apache.cassandra.locator.Replica;
+import org.apache.cassandra.net.IVerbHandler;
+import org.apache.cassandra.net.Message;
+import org.apache.cassandra.net.MessagingService;
+import org.apache.cassandra.net.RequestCallback;
+import org.apache.cassandra.net.Verb;
+import org.apache.cassandra.schema.KeyspaceMetadata;
+import org.apache.cassandra.tcm.ClusterMetadata;
+import org.apache.cassandra.tcm.membership.NodeId;
+import org.apache.cassandra.tcm.ownership.DataPlacement;
+import org.apache.cassandra.tcm.ownership.MovementMap;
+import org.apache.cassandra.tcm.ownership.PlacementDeltas;
+import org.apache.cassandra.tcm.ownership.VersionedEndpoints;
+import org.apache.cassandra.tcm.sequences.LeaveStreams;
+import org.apache.cassandra.utils.CollectionSerializers;
+import org.apache.cassandra.utils.FBUtilities;
+import org.apache.cassandra.utils.concurrent.AsyncPromise;
+import org.apache.cassandra.utils.concurrent.Future;
+import org.apache.cassandra.utils.concurrent.FutureCombiner;
+
+import static java.lang.String.format;
+import static java.util.concurrent.TimeUnit.MILLISECONDS;
+import static java.util.concurrent.TimeUnit.NANOSECONDS;
+import static java.util.concurrent.TimeUnit.SECONDS;
+import static org.apache.cassandra.utils.Clock.Global.nanoTime;
+
+/*
+ * Sealing {@link Shard}s once a topology change obsoletes them.
+ *
+ * Happy Path Sequence:
+ * 1. Bring every participant up to (at least) this seal's epoch and fence allocation, so that everyone
+ * stops writing to the old shards; then wait until all in-progress mutations have been done.
+ * 2. Same for tracked transfers
+ * 3. Wait until all participating replicas have reconciled the union of their witnessed ids
+ * 4. Mark the shard as fully sealed
+ * 5. Stop including the shard in mutation summaries for read requests.
+ *
+ * Unhappy Path Sequence (not yet implemented):
+ * 1. Wait until everyone who's up stops writing to the old shards and all in-progress mutations have been done
+ * 2. Same for tracked transfers
+ * 3. Wait until all *live* participating replicas have reconciled the union of their witnessed ids
+ * 4. Mark the shard as partially sealed, label it with the set of participants that were able to seal
+ * 5. When a down node comes up, have it talk to other nodes and learn about sealing, and what mutations/logs
+ * they have exchanged. Diff with what we have - in system tables and in the journal (though system table
+ * plus local replay should give us an up-to-date metadata state for the shard)
+ */
+public final class SealingCoordinator
+{
+ /*
+ * BOOTSTRAP sequence illustrated. Stages in which write placements
+ * remain unchanged, and only read placements change, are skipped
+ * here as irrelevant to sharding / sealing.
+ *
+ * There is a tracked keyspace with RF=3.
+ * A node n35 with token=35 is added in the middle of the ring.
+ *
+ * 1. Initial state at e0:
+ *
+ * (10,20] | {n20,n30,n40} | e0 | ACTIVE
+ * (20,30] | {n30,n40,n50} | e0 | ACTIVE
+ * (30,40] | {n40,n50,n60} | e0 | ACTIVE
+ *
+ * 2. PREPARE_JOIN at e1:
+ *
+ * (10,20] | {n20,n30,n40} | e0 | ACTIVE
+ * (20,30] | {n30,n40,n50} | e0 | ACTIVE
+ * (30,40] | {n40,n50,n60} | e0 | ACTIVE
+ * + (30,35] | {n40,n50,n60} | e1 | ACTIVE
+ * + (35,40] | {n40,n50,n60} | e1 | ACTIVE
+ *
+ * 3. START_JOIN at e2:
+ *
+ * (10,20] | {n20,n30,n40} | e0 | ACTIVE
+ * + (10,20] | {n20,n30,n35,n40} | e2 | ACTIVE
+ * (20,30] | {n30,n40,n50} | e0 | ACTIVE
+ * + (20,30] | {n30,n35,n40,n50} | e2 | ACTIVE
+ * (30,40] | {n40,n50,n60} | e0 | ACTIVE
+ * (30,35] | {n40,n50,n60} | e1 | ACTIVE
+ * + (30,35] | {n35,n40,n50,n60} | e2 | ACTIVE
+ * (35,40] | {n40,n50,n60} | e1 | ACTIVE
+ *
+ * 4. MID_JOIN at e3 [*]:
+ *
+ * S (10,20] | {n20,n30,n40} | e0 | SEALED
+ * (10,20] | {n20,n30,n35,n40} | e2 | ACTIVE
+ * S (20,30] | {n30,n40,n50} | e0 | SEALED
+ * (20,30] | {n30,n35,n40,n50} | e2 | ACTIVE
+ * S (30,40] | {n40,n50,n60} | e0 | SEALED
+ * S (30,35] | {n40,n50,n60} | e1 | SEALED
+ * (30,35] | {n35,n40,n50,n60} | e2 | ACTIVE
+ * (35,40] | {n40,n50,n60} | e1 | ACTIVE
+ *
+ * 5. FINISH_JOIN at e4:
+ *
+ * (10,20] | {n20,n30,n40} | e0 | SEALED
+ * S (10,20] | {n20,n30,n35,n40} | e2 | SEALED
+ * + (10,20] | {n20,n30,n35} | e4 | ACTIVE
+ * (20,30] | {n30,n40,n50} | e0 | SEALED
+ * S (20,30] | {n30,n35,n40,n50} | e2 | SEALED
+ * + (20,30] | {n30,n35,n40} | e4 | ACTIVE
+ * (30,40] | {n40,n50,n60} | e0 | SEALED
+ * (30,35] | {n40,n50,n60} | e1 | SEALED
+ * S (30,35] | {n35,n40,n50,n60} | e2 | SEALED
+ * + (30,35] | {n35,n40,n50} | e4 | ACTIVE
+ * (35,40] | {n40,n50,n60} | e1 | ACTIVE
+ *
+ * [*] At this stage, after sealing the shards, we stream the sstables.
+ * All the offsets belonging to these sealed shards get stripped from
+ * stats component of streamed sstables. Otherwise, the new node
+ * won't be able to compact them or mark them as reconciled/repaired -
+ * it is rightfully unaware of the sealed shards.
+ */
+
+ /**
+ * Seal the shards obsoleted by bootstrap before the new node starts streaming.
+ */
+ public static void sealShardsAtMidJoin(ClusterMetadata metadata, MovementMap movements)
+ {
+ seal(discoverShardsAtMidJoinOrReplace(metadata, movements));
+ }
+
+ /**
+ * Seal the intermediate shards that were created during START_JOIN.
+ * TODO (required): this MUST eventually run while the affected ranges are still locked - between the
+ * removeFromWrites placement change and the range unlock. FinishJoin/FinishReplace currently apply
+ * removeFromWrites and the unlock atomically in a single transformation, so there is no committed
+ * dropped-but-locked epoch to hook. Until then, this runs just after FINISH_JOIN/FINISH_REPLACE,
+ * when the ranges are already unlocked. Left to implement for the unhappy path (AY).
+ */
+ public static void sealShardsAtFinishJoin(ClusterMetadata metadata, PlacementDeltas finishDelta)
+ {
+ seal(discoverShardsAtFinishJoinOrReplace(metadata, finishDelta, null));
+ }
+
+ /*
+ * REPLACE sequence illustrated. Stages in which write placements
+ * remain unchanged, and only read placements change, are skipped
+ * here as irrelevant to sharding / sealing.
+ *
+ * There is a tracked keyspace with RF=3.
+ * Node n40 is dead, and is being replaced with $n40.
+ *
+ * 1. Initial state at e0:
+ *
+ * (10,20] | {n20,n30,n40} | e0 | ACTIVE
+ * (20,30] | {n30,n40,n50} | e0 | ACTIVE
+ * (30,40] | {n40,n50,n60} | e0 | ACTIVE
+ *
+ * 2. START_REPLACE at e1:
+ *
+ * (10,20] | {n20,n30,n40} | e0 | ACTIVE
+ * + (10,20] | {n20,n30,n40,$n40}| e1 | ACTIVE
+ * (20,30] | {n30,n40,n50} | e0 | ACTIVE
+ * + (20,30] | {n30,n40,$n40,n50}| e1 | ACTIVE
+ * (30,40] | {n40,n50,n60} | e0 | ACTIVE
+ * + (30,40] | {n40,$n40,n50,n60}| e1 | ACTIVE
+ *
+ * 3. MID_REPLACE at e2 [*]:
+ *
+ * S (10,20] | {n20,n30,n40} | e0 | SEALED
+ * (10,20] | {n20,n30,n40,$n40}| e1 | ACTIVE
+ * S (20,30] | {n30,n40,n50} | e0 | SEALED
+ * (20,30] | {n30,n40,$n40,n50}| e1 | ACTIVE
+ * S (30,40] | {n40,n50,n60} | e0 | SEALED
+ * (30,40] | {n40,$n40,n50,n60}| e1 | ACTIVE
+ *
+ * 4. FINISH_REPLACE at e3:
+ *
+ * (10,20] | {n20,n30,n40} | e0 | SEALED
+ * S (10,20] | {n20,n30,n40,$n40}| e1 | SEALED
+ * + (10,20] | {n20,n30,$n40} | e3 | ACTIVE
+ * (20,30] | {n30,n40,n50} | e0 | SEALED
+ * S (20,30] | {n30,n40,$n40,n50}| e1 | SEALED
+ * + (20,30] | {n30,$n40,n50} | e3 | ACTIVE
+ * (30,40] | {n40,n50,n60} | e0 | SEALED
+ * S (30,40] | {n40,$n40,n50,n60}| e1 | SEALED
+ * + (30,40] | {$n40,n50,n60} | e3 | ACTIVE
+ *
+ * [*] At this stage, after sealing the shards, we stream the sstables.
+ * All the offsets belonging to these sealed shards get stripped from
+ * stats component of streamed sstables. Otherwise, the new node
+ * won't be able to compact them or mark them as reconciled/repaired -
+ * it is rightfully unaware of the sealed shards.
+ */
+
+ /**
+ * Seal the shards obsoleted by host replacement before the replacement node starts streaming.
+ * The {@code deadNode} node cannot participate in the sealing.
+ */
+ public static void sealShardsAtMidReplace(ClusterMetadata metadata, MovementMap movements, @Nonnull NodeId deadNode)
+ {
+ seal(discoverShardsAtMidJoinOrReplace(metadata, movements), deadNode);
+ }
+
+ public static void sealShardsAtFinishReplace(ClusterMetadata metadata, PlacementDeltas finishDelta, @Nonnull NodeId deadNode)
+ {
+ seal(discoverShardsAtFinishJoinOrReplace(metadata, finishDelta, deadNode), deadNode);
+ }
+
+ /**
+ * Discover all shards that need to be sealed at MID_JOIN or MID_REPLACE.
+ */
+ private static Set<ShardMetadata> discoverShardsAtMidJoinOrReplace(ClusterMetadata metadata, MovementMap movements)
+ {
+ List<Future<Set<ShardMetadata>>> futures = new ArrayList<>();
+
+ for (KeyspaceMetadata ksm : metadata.schema.getKeyspaces())
+ {
+ if (!ksm.params.replicationType.isTracked())
+ continue;
+
+ EndpointsByReplica acquired = movements.get(ksm.params.replication);
+ if (acquired.isEmpty()) // when the keyspace is not replicated to this DC
+ continue;
+
+ DataPlacement placement = metadata.placements.get(ksm.params.replication);
+
+ for (Map.Entry<Replica, EndpointsForRange> entry : acquired.entrySet())
+ {
+ Range<Token> range = entry.getKey().range();
+ Set<InetAddressAndPort> endpoints = entry.getValue().endpoints();
+ // cutoff: the epoch this node was added to the write placement for the range;
+ // shards over the range with sinceEpoch < cutoff are obsoleted by this bootstrap or replace.
+ long beforeEpoch = placement.writes.forRange(range).lastModified().getEpoch();
+ futures.add(FetchShards.fetch(ksm.name, beforeEpoch, range, endpoints));
+ }
+ }
+
+ Set<ShardMetadata> shards = new HashSet<>();
+ try
+ {
+ for (Set<ShardMetadata> fetched : FutureCombiner.allOf(futures).get(DatabaseDescriptor.getRpcTimeout(MILLISECONDS), MILLISECONDS))
+ shards.addAll(fetched);
+ }
+ catch (InterruptedException | ExecutionException | TimeoutException e)
+ {
+ throw new RuntimeException("Failed to fetch shards to seal", e);
+ }
+ return shards;
+ }
+
+ /**
+ * Discover all shards that need to be sealed at FINISH_JOIN or FINISH_REPLACE.
+ */
+ private static Set<ShardMetadata> discoverShardsAtFinishJoinOrReplace(ClusterMetadata metadata, PlacementDeltas finishDelta, @Nullable NodeId deadNode)
+ {
+ long finishEpoch = metadata.epoch.getEpoch();
+ Set<ShardMetadata> shards = new HashSet<>();
+ for (KeyspaceMetadata ksm : metadata.schema.getKeyspaces())
+ {
+ if (!ksm.params.replicationType.isTracked())
+ continue;
+
+ DataPlacement placement = metadata.placements.get(ksm.params.replication);
+ finishDelta.get(ksm.params.replication).writes.removals.flattenValues().forEach(removed -> {
+ Set<Integer> nodeIDs = new HashSet<>();
+ for (InetAddressAndPort ep : placement.writes.forRange(removed.range()).endpoints())
+ nodeIDs.add(metadata.directory.peerId(ep).id());
+ nodeIDs.add(deadNode != null ? deadNode.id() : metadata.directory.peerId(removed.endpoint()).id());
+ Participants participants = new Participants(nodeIDs);
+
+ MutationTrackingService.instance().forEachShardInKeyspace(ksm.name, shard -> {
+ if (shard.sinceEpoch < finishEpoch
+ && shard.range.equals(removed.range())
+ && shard.participants.equals(participants)
+ && !shard.isSealed())
+ shards.add(new ShardMetadata(shard.keyspace, shard.sinceEpoch, shard.range, shard.participants));
+ });
+ });
+ }
+ return shards;
+ }
+
+ /*
+ * UNBOOTSTRAP and REMOVENODE sequences illustrated (same sequence as far as sharding is concerned).
+ * Stages in which write placements remain unchanged, and only read placements change, are skipped
+ * here as irrelevant to sharding / sealing.
+ *
+ * There is a tracked keyspace with RF=3.
+ * A node n40 with token=40 is removed in the middle of the ring.
+ *
+ * 1. Initial state at e0:
+ *
+ * (10,20] | {n20,n30,n40} | e0 | ACTIVE
+ * (20,30] | {n30,n40,n50} | e0 | ACTIVE
+ * (30,40] | {n40,n50,n60} | e0 | ACTIVE
+ * (40,50] | {n50,n60,n70} | e0 | ACTIVE
+ *
+ * 2. START_LEAVE at e1:
+ *
+ * (10,20] | {n20,n30,n40} | e0 | ACTIVE
+ * + (10,20] | {n20,n30,n40,n50} | e1 | ACTIVE
+ * (20,30] | {n30,n40,n50} | e0 | ACTIVE
+ * + (20,30] | {n30,n40,n50,n60} | e1 | ACTIVE
+ * (30,40] | {n40,n50,n60} | e0 | ACTIVE
+ * + (30,40] | {n40,n50,n60,n70} | e1 | ACTIVE
+ * (40,50] | {n50,n60,n70} | e0 | ACTIVE
+ *
+ * 3. MID_LEAVE at e2 [*]:
+ *
+ * S (10,20] | {n20,n30,n40} | e0 | SEALED
+ * (10,20] | {n20,n30,n40,n50} | e1 | ACTIVE
+ * S (20,30] | {n30,n40,n50} | e0 | SEALED
+ * (20,30] | {n30,n40,n50,n60} | e1 | ACTIVE
+ * S (30,40] | {n40,n50,n60} | e0 | SEALED
+ * (30,40] | {n40,n50,n60,n70} | e1 | ACTIVE
+ * (40,50] | {n50,n60,n70} | e0 | ACTIVE
+ *
+ * 4. FINISH_LEAVE at e3:
+ *
+ * (10,20] | {n20,n30,n40} | e0 | SEALED
+ * S (10,20] | {n20,n30,n40,n50} | e1 | SEALED
+ * + (10,20] | {n20,n30,n50} | e3 | ACTIVE
+ * (20,30] | {n30,n40,n50} | e0 | SEALED
+ * S (20,30] | {n30,n40,n50,n60} | e1 | SEALED
+ * + (20,30] | {n30,n50,n60} | e3 | ACTIVE
+ * (30,40] | {n40,n50,n60} | e0 | SEALED
+ * S (30,40] | {n40,n50,n60,n70} | e1 | SEALED
+ * S (40,50] | {n50,n60,n70} | e0 | SEALED
+ * + (30,50] | {n50,n60,n70} | e3 | ACTIVE
+ *
+ * [*] At this stage, after sealing the shards, we stream the sstables.
+ * All the offsets belonging to these sealed shards get stripped from
+ * stats component of streamed sstables. Otherwise, the new node
+ * won't be able to compact them or mark them as reconciled/repaired -
+ * it is rightfully unaware of the sealed shards.
+ */
+
+ /**
+ * Seal the shards obsoleted by START_LEAVE before streaming, for both UNBOOTSTRAP and REMOVENODE.
+ */
+ public static void sealShardsAtMidLeave(ClusterMetadata metadata, PlacementDeltas startDelta, NodeId leavingOrDeparted, LeaveStreams.Kind kind)
+ {
+ InetAddressAndPort leavingOrDepartedEndpoint = metadata.directory.endpoint(leavingOrDeparted);
+
+ List<Future<Set<ShardMetadata>>> futures = new ArrayList<>();
+ for (KeyspaceMetadata ksm : metadata.schema.getKeyspaces())
+ {
+ if (!ksm.params.replicationType.isTracked())
+ continue;
+
+ DataPlacement placement = metadata.placements.get(ksm.params.replication);
+ startDelta.get(ksm.params.replication).writes.additions.flattenValues().forEach(added -> {
+ Range<Token> range = added.range();
+ long beforeEpoch = placement.writes.forRange(range).lastModified().getEpoch();
+ Set<InetAddressAndPort> survivors = new HashSet<>(placement.writes.forRange(range).endpoints());
+ survivors.remove(leavingOrDepartedEndpoint);
+ futures.add(FetchShards.fetch(ksm.name, beforeEpoch, range, survivors));
+ });
+ }
+
+ Set<ShardMetadata> shards = new HashSet<>();
+ try
+ {
+ for (Set<ShardMetadata> fetched : FutureCombiner.allOf(futures).get(DatabaseDescriptor.getRpcTimeout(MILLISECONDS), MILLISECONDS))
+ for (ShardMetadata shard : fetched)
+ if (shard.participants.contains(leavingOrDeparted.id()))
+ shards.add(shard);
+ }
+ catch (InterruptedException | ExecutionException | TimeoutException e)
+ {
+ throw new RuntimeException("Failed to fetch shards to seal for leave", e);
+ }
+
+ seal(shards, kind == LeaveStreams.Kind.UNBOOTSTRAP ? null : leavingOrDeparted);
+ }
+
+ /**
+ * Seal the shards that need to be sealed at LEAVE_FINISH for both UNBOOTSTRAP and REMOVENODE.
+ */
+ public static void sealShardsAtFinishLeave(ClusterMetadata postFinish, PlacementDeltas finishDelta, NodeId leavingOrDeparted, LeaveStreams.Kind kind)
+ {
+ long finishEpoch = postFinish.epoch.getEpoch();
+ InetAddressAndPort leavingOrDepartedEndpoint = postFinish.directory.endpoint(leavingOrDeparted);
+
+ List<Future<Set<ShardMetadata>>> futures = new ArrayList<>();
+ for (KeyspaceMetadata ksm : postFinish.schema.getKeyspaces())
+ {
+ if (!ksm.params.replicationType.isTracked())
+ continue;
+
+ DataPlacement placement = postFinish.placements.get(ksm.params.replication);
+ RangesAtEndpoint removals = finishDelta.get(ksm.params.replication).writes.removals.get(leavingOrDepartedEndpoint);
+ for (Replica removedReplica : removals)
+ {
+ // FINISH_LEAVE merges the leavingOrDeparted node's (pre-merge) range with an adjacent one. Query the *merged*
+ // range (not the pre-merge range) so discovery covers both the leavingOrDeparted node's own obsoleted
+ // generations and the merge-half - the adjacent pre-merge range folded into the merged range, whose
+ // shard never listed the leavingOrDeparted node. matchRange() yields the containing merged group + its
+ // current (survivor) write endpoints.
+ VersionedEndpoints.ForRange group = placement.writes.matchRange(removedReplica.range());
+ Range<Token> mergedRange = group.range();
+ Set<InetAddressAndPort> survivors = new HashSet<>(group.endpoints());
+ survivors.remove(leavingOrDepartedEndpoint);
+ futures.add(FetchShards.fetch(ksm.name, finishEpoch, mergedRange, survivors));
+ }
+ }
+
+ Set<ShardMetadata> shards = new HashSet<>();
+ try
+ {
+ for (Set<ShardMetadata> fetched : FutureCombiner.allOf(futures).get(DatabaseDescriptor.getRpcTimeout(MILLISECONDS), MILLISECONDS))
+ shards.addAll(fetched);
+ }
+ catch (InterruptedException | ExecutionException | TimeoutException e)
+ {
+ throw new RuntimeException("Failed to fetch shards to seal for leave finish", e);
+ }
+
+ seal(shards, kind == LeaveStreams.Kind.UNBOOTSTRAP ? null : leavingOrDeparted);
+ }
+
+ /*
+ * Sealing logic.
+ */
+
+ private static void seal(Set<ShardMetadata> shards)
+ {
+ seal(shards, null);
+ }
+
+ private static void seal(Set<ShardMetadata> shards, @Nullable NodeId withoutNode)
+ {
+ initiate(shards, withoutNode); // ACTIVE -> SEALING for each Shard
+ drain(shards, withoutNode); // drain in-flight local writes
+ reconcile(shards, withoutNode); // wait for logs to reconcile
+ complete(shards, withoutNode); // SEALING-> SEALED for each Shard + journal flush
+ }
+
+ private static void initiate(Set<ShardMetadata> shards, @Nullable NodeId withoutNode)
+ {
+ List<Future<Void>> futures = new ArrayList<>(shards.size());
+ for (ShardMetadata shard : shards)
+ futures.add(initiate(shard, withoutNode));
+ try
+ {
+ FutureCombiner.allOf(futures).get(DatabaseDescriptor.getRpcTimeout(MILLISECONDS), MILLISECONDS);
+ }
+ catch (InterruptedException | ExecutionException | TimeoutException e)
+ {
+ throw new RuntimeException("Failed to initiate sealing", e);
+ }
+ }
+
+ /**
+ * Fence allocation on every participant of the obsoleted shard (ACTIVE -> SEALING) by marking the shard
+ * SEALING on each.
+ * TODO (expected): this assumes every participant has already enacted the topology change that obsoleted the shard (so
+ * it no longer allocates new ids on it). The ProgressBarrier gating bootstrap()/MID_JOIN only waits for
+ * EACH_QUORUM of the affected replicas, NOT all live replicas, so a live-but-lagging participant could still
+ * be on the old shard. Bringing each participant up to the seal epoch must be done per-replica (not
+ * per-shard) and will be added separately. TLDR - add a step to guaranteed up-to-speed rest of replicas epochs.
+ */
+ private static AsyncPromise<Void> initiate(ShardMetadata shard, @Nullable NodeId withoutNode)
+ {
+ return InitSealing.initiate(shard.keyspace, shard.sinceEpoch, shard.range, toEndpoints(shard.participants, withoutNode));
+ }
+
+ private static void drain(Set<ShardMetadata> shards, @Nullable NodeId withoutNode)
+ {
+ long deadlineNanos = nanoTime() + DatabaseDescriptor.getRpcTimeout(NANOSECONDS);
+ Set<ShardMetadata> pending = new HashSet<>(shards);
+ while (true)
+ {
+ // non-blocking poll: each participant reports its drain status immediately; retry until all drained
+ pending.removeIf(shard -> drain(shard, withoutNode));
+ if (pending.isEmpty())
+ return;
+ if (nanoTime() >= deadlineNanos)
+ throw new RuntimeException("Timed out draining shards; still pending: " + pending);
+ LockSupport.parkNanos(SECONDS.toNanos(1));
+ }
+ }
+
+ /**
+ * Poll every participant of the SEALING shard once for drain status; true iff all report that no mutation
+ * id allocated before the SEALING fence is still applying locally.
+ */
+ private static boolean drain(ShardMetadata shard, @Nullable NodeId withoutNode)
+ {
+ try
+ {
+ return Drain.poll(shard.keyspace, shard.sinceEpoch, shard.range, toEndpoints(shard.participants, withoutNode))
+ .get(DatabaseDescriptor.getRpcTimeout(MILLISECONDS), MILLISECONDS);
+ }
+ catch (InterruptedException | ExecutionException | TimeoutException e)
+ {
+ throw new RuntimeException(format("Failed to poll drain for shard %s", shard), e);
+ }
+ }
+
+ /**
+ * For each of the shards, capture the witnessed union then poll each participant until it has
+ * caught up and witnessed the entire offset union itself.
+ */
+ private static void reconcile(Set<ShardMetadata> shards, @Nullable NodeId withoutNode)
+ {
+ for (ShardMetadata shard : shards)
+ reconcile(shard, withoutNode);
+ }
+
+ private static void reconcile(ShardMetadata shard, @Nullable NodeId withoutNode)
+ {
+ // capture shard's witnessed offsets once
+ Log2OffsetsMap.Mutable offsets = captureWitnessedOffsets(shard, withoutNode);
+ // poll every participant until they've each witnessed the union of offsets
+ for (InetAddressAndPort endpoint : toEndpoints(shard.participants, withoutNode))
+ pollUntilWitnesses(shard, offsets, endpoint);
+ }
+
+ private static Log2OffsetsMap.Mutable captureWitnessedOffsets(ShardMetadata shard, @Nullable NodeId withoutNode)
+ {
+ List<InetAddressAndPort> endpoints = toEndpoints(shard.participants, withoutNode);
+ try
+ {
+ return ReconcileCapture.capture(shard.keyspace, shard.sinceEpoch, shard.range, endpoints)
+ .get(DatabaseDescriptor.getRpcTimeout(MILLISECONDS), MILLISECONDS);
+ }
+ catch (InterruptedException | ExecutionException | TimeoutException e)
+ {
+ throw new RuntimeException(format("Failed to capture witnessed offsets for shard %s", shard), e);
+ }
+ }
+
+ private static void pollUntilWitnesses(ShardMetadata shard, Log2OffsetsMap.Mutable offsets, InetAddressAndPort endpoint)
+ {
+ // TODO (expected): use a longer timeout
+ long deadlineNanos = nanoTime() + DatabaseDescriptor.getRpcTimeout(NANOSECONDS);
+
+ while (true)
+ {
+ if (nanoTime() >= deadlineNanos)
+ throw new RuntimeException("Timed out reconciling shard: " + shard);
+ try
+ {
+ boolean witnessed =
+ ReconcilePoll.poll(shard.keyspace, shard.sinceEpoch, shard.range, offsets, endpoint)
+ .get(DatabaseDescriptor.getRpcTimeout(MILLISECONDS), MILLISECONDS);
+ if (witnessed)
+ return;
+ }
+ catch (InterruptedException | ExecutionException | TimeoutException e)
+ {
+ throw new RuntimeException(format("Failed to poll reconcile for shard %s", shard), e);
+ }
+
+ LockSupport.parkNanos(SECONDS.toNanos(1));
+ }
+ }
+
+ private static void complete(Set<ShardMetadata> shards, @Nullable NodeId withoutNode)
+ {
+ List<Future<Void>> futures = new ArrayList<>(shards.size());
+ for (ShardMetadata shard : shards)
+ futures.add(complete(shard, withoutNode));
+
+ try
+ {
+ FutureCombiner.allOf(futures).get(DatabaseDescriptor.getRpcTimeout(MILLISECONDS), MILLISECONDS);
+ }
+ catch (InterruptedException | ExecutionException | TimeoutException e)
+ {
+ throw new RuntimeException("Failed to complete sealing", e);
+ }
+ }
+
+ /**
+ * Promote every participant of the (drained and reconciled) shard from SEALING to SEALED.
+ */
+ private static AsyncPromise<Void> complete(ShardMetadata shard, @Nullable NodeId withoutNode)
+ {
+ return CompleteSealing.complete(shard.keyspace, shard.sinceEpoch, shard.range, toEndpoints(shard.participants, withoutNode));
+ }
+
+ /**
+ * A bootstrapping node knows nothing about the shards that currently exist,
+ * so it must collect the list of shards from the existing replicas before
+ * it can proceed with the rest of the sealing steps.
+ */
+ public static final class FetchShards
+ {
+ public static final class Request
+ {
+ final String keyspace;
+ final long beforeEpoch;
+ final Range<Token> range;
+
+ Request(String keyspace, long beforeEpoch, Range<Token> range)
+ {
+ this.keyspace = keyspace;
+ this.beforeEpoch = beforeEpoch;
+ this.range = range;
+ }
+ }
+
+ public static final class Response
+ {
+ final Set<ShardMetadata> shards;
+
+ Response(Set<ShardMetadata> shards)
+ {
+ this.shards = shards;
+ }
+ }
+
+ /**
+ * Query all replicas in a group in parallel for the shards over {@code ranges} obsoleted by an
+ * in-flight topology change (those with {@code sinceEpoch < beforeEpoch}). Resembles
+ * {@link ShardMetadataRequest#queryPeers}, but waits for every (live) replica to respond and
+ * merges their deduplicated shards rather than taking the first response.
+ */
+ public static AsyncPromise<Set<ShardMetadata>> fetch(
+ String keyspace, long beforeEpoch, Range<Token> range, Set<InetAddressAndPort> endpoints)
+ {
+ Preconditions.checkArgument(!endpoints.isEmpty());
+ AsyncPromise<Set<ShardMetadata>> promise = new AsyncPromise<>();
+ Set<ShardMetadata> merged = ConcurrentHashMap.newKeySet();
+
+ RequestCallback<Response> callback = new RequestCallback<>()
+ {
+ private final AtomicInteger remaining = new AtomicInteger(endpoints.size());
+
+ @Override
+ public void onResponse(Message<Response> msg)
+ {
+ merged.addAll(msg.payload.shards);
+ if (remaining.decrementAndGet() == 0)
+ promise.trySuccess(merged);
+ }
+
+ @Override
+ public void onFailure(InetAddressAndPort from, RequestFailure failure)
+ {
+ // happy path assumes all replicas are up; fail the whole fetch if any can't respond
+ promise.tryFailure(new RuntimeException(format("Failed to fetch shards to seal from %s for keyspace %s: %s", from, keyspace, failure)));
+ }
+
+ @Override
+ public boolean invokeOnFailure()
+ {
+ return true;
+ }
+ };
+
+ Message<Request> message = Message.out(Verb.MT_FETCH_SHARDS_REQ, new Request(keyspace, beforeEpoch, range));
+ for (InetAddressAndPort peer : endpoints)
+ MessagingService.instance().sendWithCallback(message, peer, callback);
+
+ return promise;
+ }
+
+ public static final IVerbHandler<Request> verbHandler = message ->
+ {
+ MutationTrackingService.ensureEnabled();
+ Request request = message.payload;
+ Set<ShardMetadata> shards = new HashSet<>();
+ MutationTrackingService.instance().forEachShardInKeyspace(request.keyspace, shard -> {
+ if (shard.sinceEpoch < request.beforeEpoch
+ && (request.range.contains(shard.range) || shard.range.contains(request.range))
+ && !shard.isSealed())
+ shards.add(new ShardMetadata(shard.keyspace, shard.sinceEpoch, shard.range, shard.participants));
+ });
+ MessagingService.instance().send(message.responseWith(new Response(shards)), message.from());
+ };
+
+ public static final VersionedSerializer<Request> requestSerializer = new VersionedSerializer<>()
+ {
+ @Override
+ public void serialize(Request r, DataOutputPlus out, Version version) throws IOException
+ {
+ out.writeUTF(r.keyspace);
+ out.writeLong(r.beforeEpoch);
+ AbstractBounds.tokenSerializer.serialize(r.range, out, version.messagingVersion());
+ }
+
+ @Override
+ public Request deserialize(DataInputPlus in, Version version) throws IOException
+ {
+ String keyspace = in.readUTF();
+ long beforeEpoch = in.readLong();
+ Range<Token> range = (Range<Token>) AbstractBounds.tokenSerializer.deserialize(in, IPartitioner.global(), version.messagingVersion());
+ return new Request(keyspace, beforeEpoch, range);
+ }
+
+ @Override
+ public long serializedSize(Request r, Version version)
+ {
+ long size = TypeSizes.sizeof(r.keyspace);
+ size += TypeSizes.sizeof(r.beforeEpoch);
+ size += AbstractBounds.tokenSerializer.serializedSize(r.range, version.messagingVersion());
+ return size;
+ }
+ };
+
+ public static final VersionedSerializer<Response> responseSerializer = new VersionedSerializer<>()
+ {
+ @Override
+ public void serialize(Response r, DataOutputPlus out, Version version) throws IOException
+ {
+ CollectionSerializers.serializeCollection(r.shards, out, version, ShardMetadata.serializer);
+ }
+
+ @Override
+ public Response deserialize(DataInputPlus in, Version version) throws IOException
+ {
+ return new Response(CollectionSerializers.deserializeSet(in, version, ShardMetadata.serializer));
+ }
+
+ @Override
+ public long serializedSize(Response r, Version version)
+ {
+ return CollectionSerializers.serializedCollectionSize(r.shards, version, ShardMetadata.serializer);
+ }
+ };
+ }
+
+ public static final class InitSealing
+ {
+ public static final class Request
+ {
+ final String keyspace;
+ final long sinceEpoch;
+ final Range<Token> range;
+
+ Request(String keyspace, long sinceEpoch, Range<Token> range)
+ {
+ this.keyspace = keyspace;
+ this.sinceEpoch = sinceEpoch;
+ this.range = range;
+ }
+ }
+
+ // cannot use NoPayload, because its serializer cannot be wrapped inside mtEmbedded()
+ public static final class Response
+ {
+ private static final Response instance = new Response();
+ }
+
+ /**
+ * Tell every participant of an obsoleted shard to fence id allocation (mark it SEALING)
+ */
+ public static AsyncPromise<Void> initiate(
+ String keyspace, long sinceEpoch, Range<Token> range, List<InetAddressAndPort> endpoints)
+ {
+ AsyncPromise<Void> promise = new AsyncPromise<>();
+
+ RequestCallback<Response> callback = new RequestCallback<>()
+ {
+ private final AtomicInteger remaining = new AtomicInteger(endpoints.size());
+
+ @Override
+ public void onResponse(Message<Response> msg)
+ {
+ if (remaining.decrementAndGet() == 0)
+ promise.trySuccess(null);
+ }
+
+ @Override
+ public void onFailure(InetAddressAndPort from, RequestFailure failure)
+ {
+ promise.tryFailure(new RuntimeException(format("Failed to initiate sealing on %s for keyspace %s: %s", from, keyspace, failure)));
+ }
+
+ @Override
+ public boolean invokeOnFailure()
+ {
+ return true;
+ }
+ };
+
+ Message<Request> message = Message.out(Verb.MT_INIT_SEALING_REQ, new Request(keyspace, sinceEpoch, range));
+ for (InetAddressAndPort peer : endpoints)
+ MessagingService.instance().sendWithCallback(message, peer, callback);
+
+ return promise;
+ }
+
+ public static final IVerbHandler<Request> verbHandler = message -> {
+ MutationTrackingService.ensureEnabled();
+ Request request = message.payload;
+ MutationTrackingService.instance().markShardSealing(request.keyspace, request.sinceEpoch, request.range);
+ MessagingService.instance().send(message.responseWith(Response.instance), message.from());
+ };
+
+ public static final VersionedSerializer<Request> requestSerializer = new VersionedSerializer<>()
+ {
+ @Override
+ public void serialize(Request r, DataOutputPlus out, Version version) throws IOException
+ {
+ out.writeUTF(r.keyspace);
+ out.writeLong(r.sinceEpoch);
+ AbstractBounds.tokenSerializer.serialize(r.range, out, version.messagingVersion());
+ }
+
+ @Override
+ public Request deserialize(DataInputPlus in, Version version) throws IOException
+ {
+ String keyspace = in.readUTF();
+ long sinceEpoch = in.readLong();
+ Range<Token> range = (Range<Token>) AbstractBounds.tokenSerializer.deserialize(in, IPartitioner.global(), version.messagingVersion());
+ return new Request(keyspace, sinceEpoch, range);
+ }
+
+ @Override
+ public long serializedSize(Request r, Version version)
+ {
+ long size = TypeSizes.sizeof(r.keyspace);
+ size += TypeSizes.sizeof(r.sinceEpoch);
+ size += AbstractBounds.tokenSerializer.serializedSize(r.range, version.messagingVersion());
+ return size;
+ }
+ };
+
+ public static final UnversionedSerializer<Response> responseSerializer = new UnversionedSerializer<>()
+ {
+ @Override
+ public void serialize(Response r, DataOutputPlus out)
+ {
+ }
+
+ @Override
+ public Response deserialize(DataInputPlus in)
+ {
+ return Response.instance;
+ }
+
+ @Override
+ public long serializedSize(Response r)
+ {
+ return 0;
+ }
+ };
+ }
+
+ public static final class Drain
+ {
+ public static final class Request
+ {
+ final String keyspace;
+ final long sinceEpoch;
+ final Range<Token> range;
+
+ Request(String keyspace, long sinceEpoch, Range<Token> range)
+ {
+ this.keyspace = keyspace;
+ this.sinceEpoch = sinceEpoch;
+ this.range = range;
+ }
+ }
+
+ public static final class Response
+ {
+ final boolean drained;
+
+ Response(boolean drained)
+ {
+ this.drained = drained;
+ }
+ }
+
+ /**
+ * Poll every participant of a SEALING shard for whether it has drained its in-flight local
+ * writes. Succeeds with {@code true} iff every participant reports drained.
+ */
+ public static AsyncPromise<Boolean> poll(
+ String keyspace, long sinceEpoch, Range<Token> range, List<InetAddressAndPort> endpoints)
+ {
+ AsyncPromise<Boolean> promise = new AsyncPromise<>();
+
+ RequestCallback<Response> callback = new RequestCallback<>()
+ {
+ private final AtomicInteger remaining = new AtomicInteger(endpoints.size());
+ private volatile boolean allDrained = true;
+
+ @Override
+ public void onResponse(Message<Response> msg)
+ {
+ if (!msg.payload.drained)
+ allDrained = false;
+ if (remaining.decrementAndGet() == 0)
+ promise.trySuccess(allDrained);
+ }
+
+ @Override
+ public void onFailure(InetAddressAndPort from, RequestFailure failure)
+ {
+ promise.tryFailure(new RuntimeException(format("Failed to poll drain on %s for keyspace %s: %s", from, keyspace, failure)));
+ }
+
+ @Override
+ public boolean invokeOnFailure()
+ {
+ return true;
+ }
+ };
+
+ Message<Request> message = Message.out(Verb.MT_DRAIN_REQ, new Request(keyspace, sinceEpoch, range));
+ for (InetAddressAndPort peer : endpoints)
+ MessagingService.instance().sendWithCallback(message, peer, callback);
+
+ return promise;
+ }
+
+ public static final IVerbHandler<Request> verbHandler = message -> {
+ MutationTrackingService.ensureEnabled();
+ Request request = message.payload;
+ boolean drained = MutationTrackingService.instance().isShardDrained(request.keyspace, request.sinceEpoch, request.range);
+ MessagingService.instance().send(message.responseWith(new Response(drained)), message.from());
+ };
+
+ public static final VersionedSerializer<Request> requestSerializer = new VersionedSerializer<>()
+ {
+ @Override
+ public void serialize(Request r, DataOutputPlus out, Version version) throws IOException
+ {
+ out.writeUTF(r.keyspace);
+ out.writeLong(r.sinceEpoch);
+ AbstractBounds.tokenSerializer.serialize(r.range, out, version.messagingVersion());
+ }
+
+ @Override
+ public Request deserialize(DataInputPlus in, Version version) throws IOException
+ {
+ String keyspace = in.readUTF();
+ long sinceEpoch = in.readLong();
+ Range<Token> range = (Range<Token>) AbstractBounds.tokenSerializer.deserialize(in, IPartitioner.global(), version.messagingVersion());
+ return new Request(keyspace, sinceEpoch, range);
+ }
+
+ @Override
+ public long serializedSize(Request r, Version version)
+ {
+ long size = TypeSizes.sizeof(r.keyspace);
+ size += TypeSizes.sizeof(r.sinceEpoch);
+ size += AbstractBounds.tokenSerializer.serializedSize(r.range, version.messagingVersion());
+ return size;
+ }
+ };
+
+ public static final UnversionedSerializer<Response> responseSerializer = new UnversionedSerializer<>()
+ {
+ @Override
+ public void serialize(Response r, DataOutputPlus out) throws IOException
+ {
+ out.writeBoolean(r.drained);
+ }
+
+ @Override
+ public Response deserialize(DataInputPlus in) throws IOException
+ {
+ return new Response(in.readBoolean());
+ }
+
+ @Override
+ public long serializedSize(Response r)
+ {
+ return TypeSizes.sizeof(r.drained);
+ }
+ };
+ }
+
+ public static final class ReconcileCapture
+ {
+ public static final class Request
+ {
+ final String keyspace;
+ final long sinceEpoch;
+ final Range<Token> range;
+
+ Request(String keyspace, long sinceEpoch, Range<Token> range)
+ {
+ this.keyspace = keyspace;
+ this.sinceEpoch = sinceEpoch;
+ this.range = range;
+ }
+ }
+
+ public static final class Response
+ {
+ final Log2OffsetsMap<?> witnessed;
+
+ Response(Log2OffsetsMap<?> witnessed)
+ {
+ this.witnessed = witnessed;
+ }
+ }
+
+ public static AsyncPromise<Log2OffsetsMap.Mutable> capture(
+ String keyspace, long sinceEpoch, Range<Token> range, List<InetAddressAndPort> endpoints)
+ {
+ AsyncPromise<Log2OffsetsMap.Mutable> promise = new AsyncPromise<>();
+ Log2OffsetsMap.Mutable union = new Log2OffsetsMap.Mutable();
+
+ RequestCallback<Response> callback = new RequestCallback<>()
+ {
+ private final AtomicInteger remaining = new AtomicInteger(endpoints.size());
+
+ @Override
+ public void onResponse(Message<Response> msg)
+ {
+ synchronized (union)
+ {
+ union.addAll(msg.payload.witnessed);
+ }
+ if (remaining.decrementAndGet() == 0)
+ promise.trySuccess(union);
+ }
+
+ @Override
+ public void onFailure(InetAddressAndPort from, RequestFailure failure)
+ {
+ promise.tryFailure(new RuntimeException(format("Failed to capture witnessed offsets on %s for keyspace %s: %s", from, keyspace, failure)));
+ }
+
+ @Override
+ public boolean invokeOnFailure()
+ {
+ return true;
+ }
+ };
+
+ Message<Request> message = Message.out(Verb.MT_RECONCILE_CAPTURE_REQ, new Request(keyspace, sinceEpoch, range));
+ for (InetAddressAndPort peer : endpoints)
+ MessagingService.instance().sendWithCallback(message, peer, callback);
+
+ return promise;
+ }
+
+ public static final IVerbHandler<Request> verbHandler = message -> {
+ MutationTrackingService.ensureEnabled();
+ Request request = message.payload;
+ Log2OffsetsMap<?> witnessed =
+ MutationTrackingService.instance()
+ .collectLocallyWitnessedOffsets(request.keyspace, request.sinceEpoch, request.range);
+ MessagingService.instance().send(message.responseWith(new Response(witnessed)), message.from());
+ };
+
+ public static final VersionedSerializer<Request> requestSerializer = new VersionedSerializer<>()
+ {
+ @Override
+ public void serialize(Request r, DataOutputPlus out, Version version) throws IOException
+ {
+ out.writeUTF(r.keyspace);
+ out.writeLong(r.sinceEpoch);
+ AbstractBounds.tokenSerializer.serialize(r.range, out, version.messagingVersion());
+ }
+
+ @Override
+ public Request deserialize(DataInputPlus in, Version version) throws IOException
+ {
+ String keyspace = in.readUTF();
+ long sinceEpoch = in.readLong();
+ Range<Token> range = (Range<Token>) AbstractBounds.tokenSerializer.deserialize(in, IPartitioner.global(), version.messagingVersion());
+ return new Request(keyspace, sinceEpoch, range);
+ }
+
+ @Override
+ public long serializedSize(Request r, Version version)
+ {
+ long size = TypeSizes.sizeof(r.keyspace);
+ size += TypeSizes.sizeof(r.sinceEpoch);
+ size += AbstractBounds.tokenSerializer.serializedSize(r.range, version.messagingVersion());
+ return size;
+ }
+ };
+
+ public static final UnversionedSerializer<Response> responseSerializer = new UnversionedSerializer<>()
+ {
+ @Override
+ public void serialize(Response r, DataOutputPlus out) throws IOException
+ {
+ Log2OffsetsMap.serializer.serialize(r.witnessed, out);
+ }
+
+ @Override
+ public Response deserialize(DataInputPlus in) throws IOException
+ {
+ return new Response(Log2OffsetsMap.serializer.deserialize(in));
+ }
+
+ @Override
+ public long serializedSize(Response r)
+ {
+ return Log2OffsetsMap.serializer.serializedSize(r.witnessed);
+ }
+ };
+ }
+
+ public static final class ReconcilePoll
+ {
+ public static final class Request
+ {
+ final String keyspace;
+ final long sinceEpoch;
+ final Range<Token> range;
+ final Log2OffsetsMap<?> offsets;
+
+ Request(String keyspace, long sinceEpoch, Range<Token> range, Log2OffsetsMap<?> offsets)
+ {
+ this.keyspace = keyspace;
+ this.sinceEpoch = sinceEpoch;
+ this.range = range;
+ this.offsets = offsets;
+ }
+ }
+
+ public static final class Response
+ {
+ final boolean witnessed;
+
+ Response(boolean witnessed)
+ {
+ this.witnessed = witnessed;
+ }
+ }
+
+ public static AsyncPromise<Boolean> poll(
+ String keyspace, long sinceEpoch, Range<Token> range, Log2OffsetsMap.Mutable offsets, InetAddressAndPort endpoint)
+ {
+ AsyncPromise<Boolean> promise = new AsyncPromise<>();
+
+ RequestCallback<Response> callback = new RequestCallback<>()
+ {
+ @Override
+ public void onResponse(Message<Response> msg)
+ {
+ promise.trySuccess(msg.payload.witnessed);
+ }
+
+ @Override
+ public void onFailure(InetAddressAndPort from, RequestFailure failure)
+ {
+ promise.tryFailure(new RuntimeException(format("Failed to reconcile poll %s for keyspace %s: %s", from, keyspace, failure)));
+ }
+
+ @Override
+ public boolean invokeOnFailure()
+ {
+ return true;
+ }
+ };
+
+ MessagingService.instance()
+ .sendWithCallback(Message.out(Verb.MT_RECONCILE_POLL_REQ, new Request(keyspace, sinceEpoch, range, offsets)),
+ endpoint, callback);
+
+ return promise;
+ }
+
+ public static final IVerbHandler<Request> verbHandler = message -> {
+ MutationTrackingService.ensureEnabled();
+ Request request = message.payload;
+ boolean witnessed =
+ MutationTrackingService.instance().hasWitnessed(request.keyspace, request.sinceEpoch, request.range, request.offsets);
+ MessagingService.instance().send(message.responseWith(new Response(witnessed)), message.from());
+ };
+
+ public static final VersionedSerializer<Request> requestSerializer = new VersionedSerializer<>()
+ {
+ @Override
+ public void serialize(Request r, DataOutputPlus out, Version version) throws IOException
+ {
+ out.writeUTF(r.keyspace);
+ out.writeLong(r.sinceEpoch);
+ AbstractBounds.tokenSerializer.serialize(r.range, out, version.messagingVersion());
+ Log2OffsetsMap.serializer.serialize(r.offsets, out);
+ }
+
+ @Override
+ public Request deserialize(DataInputPlus in, Version version) throws IOException
+ {
+ String keyspace = in.readUTF();
+ long sinceEpoch = in.readLong();
+ Range<Token> range = (Range<Token>) AbstractBounds.tokenSerializer.deserialize(in, IPartitioner.global(), version.messagingVersion());
+ Log2OffsetsMap.Immutable offsets = Log2OffsetsMap.serializer.deserialize(in);
+ return new Request(keyspace, sinceEpoch, range, offsets);
+ }
+
+ @Override
+ public long serializedSize(Request r, Version version)
+ {
+ long size = TypeSizes.sizeof(r.keyspace);
+ size += TypeSizes.sizeof(r.sinceEpoch);
+ size += AbstractBounds.tokenSerializer.serializedSize(r.range, version.messagingVersion());
+ size += Log2OffsetsMap.serializer.serializedSize(r.offsets);
+ return size;
+ }
+ };
+
+ public static final UnversionedSerializer<Response> responseSerializer = new UnversionedSerializer<>()
+ {
+ @Override
+ public void serialize(Response r, DataOutputPlus out) throws IOException
+ {
+ out.writeBoolean(r.witnessed);
+ }
+
+ @Override
+ public Response deserialize(DataInputPlus in) throws IOException
+ {
+ return new Response(in.readBoolean());
+ }
+
+ @Override
+ public long serializedSize(Response r)
+ {
+ return TypeSizes.sizeof(r.witnessed);
+ }
+ };
+ }
+
+ public static final class CompleteSealing
+ {
+ public static final class Request
+ {
+ final String keyspace;
+ final long sinceEpoch;
+ final Range<Token> range;
+
+ Request(String keyspace, long sinceEpoch, Range<Token> range)
+ {
+ this.keyspace = keyspace;
+ this.sinceEpoch = sinceEpoch;
+ this.range = range;
+ }
+ }
+
+ // cannot use NoPayload, because its serializer cannot be wrapped inside mtEmbedded()
+ public static final class Response
+ {
+ private static final Response instance = new Response();
+ }
+
+ /**
+ * Tell every participant of a sealing shard to promote it from SEALING to SEALED.
+ */
+ public static AsyncPromise<Void> complete(
+ String keyspace, long sinceEpoch, Range<Token> range, List<InetAddressAndPort> endpoints)
+ {
Review Comment:
```suggestion
{
Preconditions.checkArgument(!endpoints.isEmpty());
```
##########
src/java/org/apache/cassandra/tcm/sequences/BootstrapAndReplace.java:
##########
@@ -290,6 +292,9 @@ public SequenceState executeNext()
}
ClusterMetadataService.instance().ensureCMSPlacement(metadata);
+ if (MutationTrackingService.isEnabled())
+ SealingCoordinator.sealShardsAtFinishReplace(metadata, finishReplace.delta(), startReplace.replaced());
Review Comment:
I think this should happen before we set the bootstrap state on the system keyspace? Otherwise a failure after the CMS commit could leave the shards in a SEALING state. I don't think there's downside to sealing before the commit. If it fails, the seal will be retried
##########
src/java/org/apache/cassandra/replication/SealingCoordinator.java:
##########
@@ -0,0 +1,1401 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.cassandra.replication;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.TimeoutException;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.concurrent.locks.LockSupport;
+
+import javax.annotation.Nonnull;
+import javax.annotation.Nullable;
+
+import com.google.common.base.Preconditions;
+
+import org.apache.cassandra.config.DatabaseDescriptor;
+import org.apache.cassandra.db.ColumnFamilyStore;
+import org.apache.cassandra.db.Keyspace;
+import org.apache.cassandra.db.TypeSizes;
+import org.apache.cassandra.dht.AbstractBounds;
+import org.apache.cassandra.dht.IPartitioner;
+import org.apache.cassandra.dht.Range;
+import org.apache.cassandra.dht.Token;
+import org.apache.cassandra.exceptions.RequestFailure;
+import org.apache.cassandra.io.UnversionedSerializer;
+import org.apache.cassandra.io.util.DataInputPlus;
+import org.apache.cassandra.io.util.DataOutputPlus;
+import org.apache.cassandra.locator.EndpointsByReplica;
+import org.apache.cassandra.locator.EndpointsForRange;
+import org.apache.cassandra.locator.InetAddressAndPort;
+import org.apache.cassandra.locator.RangesAtEndpoint;
+import org.apache.cassandra.locator.Replica;
+import org.apache.cassandra.net.IVerbHandler;
+import org.apache.cassandra.net.Message;
+import org.apache.cassandra.net.MessagingService;
+import org.apache.cassandra.net.RequestCallback;
+import org.apache.cassandra.net.Verb;
+import org.apache.cassandra.schema.KeyspaceMetadata;
+import org.apache.cassandra.tcm.ClusterMetadata;
+import org.apache.cassandra.tcm.membership.NodeId;
+import org.apache.cassandra.tcm.ownership.DataPlacement;
+import org.apache.cassandra.tcm.ownership.MovementMap;
+import org.apache.cassandra.tcm.ownership.PlacementDeltas;
+import org.apache.cassandra.tcm.ownership.VersionedEndpoints;
+import org.apache.cassandra.tcm.sequences.LeaveStreams;
+import org.apache.cassandra.utils.CollectionSerializers;
+import org.apache.cassandra.utils.FBUtilities;
+import org.apache.cassandra.utils.concurrent.AsyncPromise;
+import org.apache.cassandra.utils.concurrent.Future;
+import org.apache.cassandra.utils.concurrent.FutureCombiner;
+
+import static java.lang.String.format;
+import static java.util.concurrent.TimeUnit.MILLISECONDS;
+import static java.util.concurrent.TimeUnit.NANOSECONDS;
+import static java.util.concurrent.TimeUnit.SECONDS;
+import static org.apache.cassandra.utils.Clock.Global.nanoTime;
+
+/*
+ * Sealing {@link Shard}s once a topology change obsoletes them.
+ *
+ * Happy Path Sequence:
+ * 1. Bring every participant up to (at least) this seal's epoch and fence allocation, so that everyone
+ * stops writing to the old shards; then wait until all in-progress mutations have been done.
+ * 2. Same for tracked transfers
+ * 3. Wait until all participating replicas have reconciled the union of their witnessed ids
+ * 4. Mark the shard as fully sealed
+ * 5. Stop including the shard in mutation summaries for read requests.
+ *
+ * Unhappy Path Sequence (not yet implemented):
+ * 1. Wait until everyone who's up stops writing to the old shards and all in-progress mutations have been done
+ * 2. Same for tracked transfers
+ * 3. Wait until all *live* participating replicas have reconciled the union of their witnessed ids
+ * 4. Mark the shard as partially sealed, label it with the set of participants that were able to seal
+ * 5. When a down node comes up, have it talk to other nodes and learn about sealing, and what mutations/logs
+ * they have exchanged. Diff with what we have - in system tables and in the journal (though system table
+ * plus local replay should give us an up-to-date metadata state for the shard)
+ */
+public final class SealingCoordinator
+{
+ /*
+ * BOOTSTRAP sequence illustrated. Stages in which write placements
+ * remain unchanged, and only read placements change, are skipped
+ * here as irrelevant to sharding / sealing.
+ *
+ * There is a tracked keyspace with RF=3.
+ * A node n35 with token=35 is added in the middle of the ring.
+ *
+ * 1. Initial state at e0:
+ *
+ * (10,20] | {n20,n30,n40} | e0 | ACTIVE
+ * (20,30] | {n30,n40,n50} | e0 | ACTIVE
+ * (30,40] | {n40,n50,n60} | e0 | ACTIVE
+ *
+ * 2. PREPARE_JOIN at e1:
+ *
+ * (10,20] | {n20,n30,n40} | e0 | ACTIVE
+ * (20,30] | {n30,n40,n50} | e0 | ACTIVE
+ * (30,40] | {n40,n50,n60} | e0 | ACTIVE
+ * + (30,35] | {n40,n50,n60} | e1 | ACTIVE
+ * + (35,40] | {n40,n50,n60} | e1 | ACTIVE
+ *
+ * 3. START_JOIN at e2:
+ *
+ * (10,20] | {n20,n30,n40} | e0 | ACTIVE
+ * + (10,20] | {n20,n30,n35,n40} | e2 | ACTIVE
+ * (20,30] | {n30,n40,n50} | e0 | ACTIVE
+ * + (20,30] | {n30,n35,n40,n50} | e2 | ACTIVE
+ * (30,40] | {n40,n50,n60} | e0 | ACTIVE
+ * (30,35] | {n40,n50,n60} | e1 | ACTIVE
+ * + (30,35] | {n35,n40,n50,n60} | e2 | ACTIVE
+ * (35,40] | {n40,n50,n60} | e1 | ACTIVE
+ *
+ * 4. MID_JOIN at e3 [*]:
+ *
+ * S (10,20] | {n20,n30,n40} | e0 | SEALED
+ * (10,20] | {n20,n30,n35,n40} | e2 | ACTIVE
+ * S (20,30] | {n30,n40,n50} | e0 | SEALED
+ * (20,30] | {n30,n35,n40,n50} | e2 | ACTIVE
+ * S (30,40] | {n40,n50,n60} | e0 | SEALED
+ * S (30,35] | {n40,n50,n60} | e1 | SEALED
+ * (30,35] | {n35,n40,n50,n60} | e2 | ACTIVE
+ * (35,40] | {n40,n50,n60} | e1 | ACTIVE
+ *
+ * 5. FINISH_JOIN at e4:
+ *
+ * (10,20] | {n20,n30,n40} | e0 | SEALED
+ * S (10,20] | {n20,n30,n35,n40} | e2 | SEALED
+ * + (10,20] | {n20,n30,n35} | e4 | ACTIVE
+ * (20,30] | {n30,n40,n50} | e0 | SEALED
+ * S (20,30] | {n30,n35,n40,n50} | e2 | SEALED
+ * + (20,30] | {n30,n35,n40} | e4 | ACTIVE
+ * (30,40] | {n40,n50,n60} | e0 | SEALED
+ * (30,35] | {n40,n50,n60} | e1 | SEALED
+ * S (30,35] | {n35,n40,n50,n60} | e2 | SEALED
+ * + (30,35] | {n35,n40,n50} | e4 | ACTIVE
+ * (35,40] | {n40,n50,n60} | e1 | ACTIVE
+ *
+ * [*] At this stage, after sealing the shards, we stream the sstables.
+ * All the offsets belonging to these sealed shards get stripped from
+ * stats component of streamed sstables. Otherwise, the new node
+ * won't be able to compact them or mark them as reconciled/repaired -
+ * it is rightfully unaware of the sealed shards.
+ */
+
+ /**
+ * Seal the shards obsoleted by bootstrap before the new node starts streaming.
+ */
+ public static void sealShardsAtMidJoin(ClusterMetadata metadata, MovementMap movements)
+ {
+ seal(discoverShardsAtMidJoinOrReplace(metadata, movements));
+ }
+
+ /**
+ * Seal the intermediate shards that were created during START_JOIN.
+ * TODO (required): this MUST eventually run while the affected ranges are still locked - between the
+ * removeFromWrites placement change and the range unlock. FinishJoin/FinishReplace currently apply
+ * removeFromWrites and the unlock atomically in a single transformation, so there is no committed
+ * dropped-but-locked epoch to hook. Until then, this runs just after FINISH_JOIN/FINISH_REPLACE,
+ * when the ranges are already unlocked. Left to implement for the unhappy path (AY).
+ */
+ public static void sealShardsAtFinishJoin(ClusterMetadata metadata, PlacementDeltas finishDelta)
+ {
+ seal(discoverShardsAtFinishJoinOrReplace(metadata, finishDelta, null));
+ }
+
+ /*
+ * REPLACE sequence illustrated. Stages in which write placements
+ * remain unchanged, and only read placements change, are skipped
+ * here as irrelevant to sharding / sealing.
+ *
+ * There is a tracked keyspace with RF=3.
+ * Node n40 is dead, and is being replaced with $n40.
+ *
+ * 1. Initial state at e0:
+ *
+ * (10,20] | {n20,n30,n40} | e0 | ACTIVE
+ * (20,30] | {n30,n40,n50} | e0 | ACTIVE
+ * (30,40] | {n40,n50,n60} | e0 | ACTIVE
+ *
+ * 2. START_REPLACE at e1:
+ *
+ * (10,20] | {n20,n30,n40} | e0 | ACTIVE
+ * + (10,20] | {n20,n30,n40,$n40}| e1 | ACTIVE
+ * (20,30] | {n30,n40,n50} | e0 | ACTIVE
+ * + (20,30] | {n30,n40,$n40,n50}| e1 | ACTIVE
+ * (30,40] | {n40,n50,n60} | e0 | ACTIVE
+ * + (30,40] | {n40,$n40,n50,n60}| e1 | ACTIVE
+ *
+ * 3. MID_REPLACE at e2 [*]:
+ *
+ * S (10,20] | {n20,n30,n40} | e0 | SEALED
+ * (10,20] | {n20,n30,n40,$n40}| e1 | ACTIVE
+ * S (20,30] | {n30,n40,n50} | e0 | SEALED
+ * (20,30] | {n30,n40,$n40,n50}| e1 | ACTIVE
+ * S (30,40] | {n40,n50,n60} | e0 | SEALED
+ * (30,40] | {n40,$n40,n50,n60}| e1 | ACTIVE
+ *
+ * 4. FINISH_REPLACE at e3:
+ *
+ * (10,20] | {n20,n30,n40} | e0 | SEALED
+ * S (10,20] | {n20,n30,n40,$n40}| e1 | SEALED
+ * + (10,20] | {n20,n30,$n40} | e3 | ACTIVE
+ * (20,30] | {n30,n40,n50} | e0 | SEALED
+ * S (20,30] | {n30,n40,$n40,n50}| e1 | SEALED
+ * + (20,30] | {n30,$n40,n50} | e3 | ACTIVE
+ * (30,40] | {n40,n50,n60} | e0 | SEALED
+ * S (30,40] | {n40,$n40,n50,n60}| e1 | SEALED
+ * + (30,40] | {$n40,n50,n60} | e3 | ACTIVE
+ *
+ * [*] At this stage, after sealing the shards, we stream the sstables.
+ * All the offsets belonging to these sealed shards get stripped from
+ * stats component of streamed sstables. Otherwise, the new node
+ * won't be able to compact them or mark them as reconciled/repaired -
+ * it is rightfully unaware of the sealed shards.
+ */
+
+ /**
+ * Seal the shards obsoleted by host replacement before the replacement node starts streaming.
+ * The {@code deadNode} node cannot participate in the sealing.
+ */
+ public static void sealShardsAtMidReplace(ClusterMetadata metadata, MovementMap movements, @Nonnull NodeId deadNode)
+ {
+ seal(discoverShardsAtMidJoinOrReplace(metadata, movements), deadNode);
+ }
+
+ public static void sealShardsAtFinishReplace(ClusterMetadata metadata, PlacementDeltas finishDelta, @Nonnull NodeId deadNode)
+ {
+ seal(discoverShardsAtFinishJoinOrReplace(metadata, finishDelta, deadNode), deadNode);
+ }
+
+ /**
+ * Discover all shards that need to be sealed at MID_JOIN or MID_REPLACE.
+ */
+ private static Set<ShardMetadata> discoverShardsAtMidJoinOrReplace(ClusterMetadata metadata, MovementMap movements)
+ {
+ List<Future<Set<ShardMetadata>>> futures = new ArrayList<>();
+
+ for (KeyspaceMetadata ksm : metadata.schema.getKeyspaces())
+ {
+ if (!ksm.params.replicationType.isTracked())
+ continue;
+
+ EndpointsByReplica acquired = movements.get(ksm.params.replication);
+ if (acquired.isEmpty()) // when the keyspace is not replicated to this DC
+ continue;
+
+ DataPlacement placement = metadata.placements.get(ksm.params.replication);
+
+ for (Map.Entry<Replica, EndpointsForRange> entry : acquired.entrySet())
+ {
+ Range<Token> range = entry.getKey().range();
+ Set<InetAddressAndPort> endpoints = entry.getValue().endpoints();
+ // cutoff: the epoch this node was added to the write placement for the range;
+ // shards over the range with sinceEpoch < cutoff are obsoleted by this bootstrap or replace.
+ long beforeEpoch = placement.writes.forRange(range).lastModified().getEpoch();
+ futures.add(FetchShards.fetch(ksm.name, beforeEpoch, range, endpoints));
+ }
+ }
+
+ Set<ShardMetadata> shards = new HashSet<>();
+ try
+ {
+ for (Set<ShardMetadata> fetched : FutureCombiner.allOf(futures).get(DatabaseDescriptor.getRpcTimeout(MILLISECONDS), MILLISECONDS))
+ shards.addAll(fetched);
+ }
+ catch (InterruptedException | ExecutionException | TimeoutException e)
+ {
+ throw new RuntimeException("Failed to fetch shards to seal", e);
+ }
+ return shards;
+ }
+
+ /**
+ * Discover all shards that need to be sealed at FINISH_JOIN or FINISH_REPLACE.
+ */
+ private static Set<ShardMetadata> discoverShardsAtFinishJoinOrReplace(ClusterMetadata metadata, PlacementDeltas finishDelta, @Nullable NodeId deadNode)
+ {
+ long finishEpoch = metadata.epoch.getEpoch();
+ Set<ShardMetadata> shards = new HashSet<>();
+ for (KeyspaceMetadata ksm : metadata.schema.getKeyspaces())
+ {
+ if (!ksm.params.replicationType.isTracked())
+ continue;
+
+ DataPlacement placement = metadata.placements.get(ksm.params.replication);
+ finishDelta.get(ksm.params.replication).writes.removals.flattenValues().forEach(removed -> {
+ Set<Integer> nodeIDs = new HashSet<>();
+ for (InetAddressAndPort ep : placement.writes.forRange(removed.range()).endpoints())
+ nodeIDs.add(metadata.directory.peerId(ep).id());
+ nodeIDs.add(deadNode != null ? deadNode.id() : metadata.directory.peerId(removed.endpoint()).id());
+ Participants participants = new Participants(nodeIDs);
+
+ MutationTrackingService.instance().forEachShardInKeyspace(ksm.name, shard -> {
+ if (shard.sinceEpoch < finishEpoch
+ && shard.range.equals(removed.range())
+ && shard.participants.equals(participants)
+ && !shard.isSealed())
+ shards.add(new ShardMetadata(shard.keyspace, shard.sinceEpoch, shard.range, shard.participants));
+ });
+ });
+ }
+ return shards;
+ }
+
+ /*
+ * UNBOOTSTRAP and REMOVENODE sequences illustrated (same sequence as far as sharding is concerned).
+ * Stages in which write placements remain unchanged, and only read placements change, are skipped
+ * here as irrelevant to sharding / sealing.
+ *
+ * There is a tracked keyspace with RF=3.
+ * A node n40 with token=40 is removed in the middle of the ring.
+ *
+ * 1. Initial state at e0:
+ *
+ * (10,20] | {n20,n30,n40} | e0 | ACTIVE
+ * (20,30] | {n30,n40,n50} | e0 | ACTIVE
+ * (30,40] | {n40,n50,n60} | e0 | ACTIVE
+ * (40,50] | {n50,n60,n70} | e0 | ACTIVE
+ *
+ * 2. START_LEAVE at e1:
+ *
+ * (10,20] | {n20,n30,n40} | e0 | ACTIVE
+ * + (10,20] | {n20,n30,n40,n50} | e1 | ACTIVE
+ * (20,30] | {n30,n40,n50} | e0 | ACTIVE
+ * + (20,30] | {n30,n40,n50,n60} | e1 | ACTIVE
+ * (30,40] | {n40,n50,n60} | e0 | ACTIVE
+ * + (30,40] | {n40,n50,n60,n70} | e1 | ACTIVE
+ * (40,50] | {n50,n60,n70} | e0 | ACTIVE
+ *
+ * 3. MID_LEAVE at e2 [*]:
+ *
+ * S (10,20] | {n20,n30,n40} | e0 | SEALED
+ * (10,20] | {n20,n30,n40,n50} | e1 | ACTIVE
+ * S (20,30] | {n30,n40,n50} | e0 | SEALED
+ * (20,30] | {n30,n40,n50,n60} | e1 | ACTIVE
+ * S (30,40] | {n40,n50,n60} | e0 | SEALED
+ * (30,40] | {n40,n50,n60,n70} | e1 | ACTIVE
+ * (40,50] | {n50,n60,n70} | e0 | ACTIVE
+ *
+ * 4. FINISH_LEAVE at e3:
+ *
+ * (10,20] | {n20,n30,n40} | e0 | SEALED
+ * S (10,20] | {n20,n30,n40,n50} | e1 | SEALED
+ * + (10,20] | {n20,n30,n50} | e3 | ACTIVE
+ * (20,30] | {n30,n40,n50} | e0 | SEALED
+ * S (20,30] | {n30,n40,n50,n60} | e1 | SEALED
+ * + (20,30] | {n30,n50,n60} | e3 | ACTIVE
+ * (30,40] | {n40,n50,n60} | e0 | SEALED
+ * S (30,40] | {n40,n50,n60,n70} | e1 | SEALED
+ * S (40,50] | {n50,n60,n70} | e0 | SEALED
+ * + (30,50] | {n50,n60,n70} | e3 | ACTIVE
+ *
+ * [*] At this stage, after sealing the shards, we stream the sstables.
+ * All the offsets belonging to these sealed shards get stripped from
+ * stats component of streamed sstables. Otherwise, the new node
+ * won't be able to compact them or mark them as reconciled/repaired -
+ * it is rightfully unaware of the sealed shards.
+ */
+
+ /**
+ * Seal the shards obsoleted by START_LEAVE before streaming, for both UNBOOTSTRAP and REMOVENODE.
+ */
+ public static void sealShardsAtMidLeave(ClusterMetadata metadata, PlacementDeltas startDelta, NodeId leavingOrDeparted, LeaveStreams.Kind kind)
+ {
+ InetAddressAndPort leavingOrDepartedEndpoint = metadata.directory.endpoint(leavingOrDeparted);
+
+ List<Future<Set<ShardMetadata>>> futures = new ArrayList<>();
+ for (KeyspaceMetadata ksm : metadata.schema.getKeyspaces())
+ {
+ if (!ksm.params.replicationType.isTracked())
+ continue;
+
+ DataPlacement placement = metadata.placements.get(ksm.params.replication);
+ startDelta.get(ksm.params.replication).writes.additions.flattenValues().forEach(added -> {
+ Range<Token> range = added.range();
+ long beforeEpoch = placement.writes.forRange(range).lastModified().getEpoch();
+ Set<InetAddressAndPort> survivors = new HashSet<>(placement.writes.forRange(range).endpoints());
+ survivors.remove(leavingOrDepartedEndpoint);
+ futures.add(FetchShards.fetch(ksm.name, beforeEpoch, range, survivors));
+ });
+ }
+
+ Set<ShardMetadata> shards = new HashSet<>();
+ try
+ {
+ for (Set<ShardMetadata> fetched : FutureCombiner.allOf(futures).get(DatabaseDescriptor.getRpcTimeout(MILLISECONDS), MILLISECONDS))
+ for (ShardMetadata shard : fetched)
+ if (shard.participants.contains(leavingOrDeparted.id()))
+ shards.add(shard);
+ }
+ catch (InterruptedException | ExecutionException | TimeoutException e)
+ {
+ throw new RuntimeException("Failed to fetch shards to seal for leave", e);
+ }
+
+ seal(shards, kind == LeaveStreams.Kind.UNBOOTSTRAP ? null : leavingOrDeparted);
+ }
+
+ /**
+ * Seal the shards that need to be sealed at LEAVE_FINISH for both UNBOOTSTRAP and REMOVENODE.
+ */
+ public static void sealShardsAtFinishLeave(ClusterMetadata postFinish, PlacementDeltas finishDelta, NodeId leavingOrDeparted, LeaveStreams.Kind kind)
+ {
+ long finishEpoch = postFinish.epoch.getEpoch();
+ InetAddressAndPort leavingOrDepartedEndpoint = postFinish.directory.endpoint(leavingOrDeparted);
+
+ List<Future<Set<ShardMetadata>>> futures = new ArrayList<>();
+ for (KeyspaceMetadata ksm : postFinish.schema.getKeyspaces())
+ {
+ if (!ksm.params.replicationType.isTracked())
+ continue;
+
+ DataPlacement placement = postFinish.placements.get(ksm.params.replication);
+ RangesAtEndpoint removals = finishDelta.get(ksm.params.replication).writes.removals.get(leavingOrDepartedEndpoint);
+ for (Replica removedReplica : removals)
+ {
+ // FINISH_LEAVE merges the leavingOrDeparted node's (pre-merge) range with an adjacent one. Query the *merged*
+ // range (not the pre-merge range) so discovery covers both the leavingOrDeparted node's own obsoleted
+ // generations and the merge-half - the adjacent pre-merge range folded into the merged range, whose
+ // shard never listed the leavingOrDeparted node. matchRange() yields the containing merged group + its
+ // current (survivor) write endpoints.
+ VersionedEndpoints.ForRange group = placement.writes.matchRange(removedReplica.range());
+ Range<Token> mergedRange = group.range();
+ Set<InetAddressAndPort> survivors = new HashSet<>(group.endpoints());
+ survivors.remove(leavingOrDepartedEndpoint);
+ futures.add(FetchShards.fetch(ksm.name, finishEpoch, mergedRange, survivors));
+ }
+ }
+
+ Set<ShardMetadata> shards = new HashSet<>();
+ try
+ {
+ for (Set<ShardMetadata> fetched : FutureCombiner.allOf(futures).get(DatabaseDescriptor.getRpcTimeout(MILLISECONDS), MILLISECONDS))
+ shards.addAll(fetched);
+ }
+ catch (InterruptedException | ExecutionException | TimeoutException e)
+ {
+ throw new RuntimeException("Failed to fetch shards to seal for leave finish", e);
+ }
+
+ seal(shards, kind == LeaveStreams.Kind.UNBOOTSTRAP ? null : leavingOrDeparted);
+ }
+
+ /*
+ * Sealing logic.
+ */
+
+ private static void seal(Set<ShardMetadata> shards)
+ {
+ seal(shards, null);
+ }
+
+ private static void seal(Set<ShardMetadata> shards, @Nullable NodeId withoutNode)
+ {
+ initiate(shards, withoutNode); // ACTIVE -> SEALING for each Shard
+ drain(shards, withoutNode); // drain in-flight local writes
+ reconcile(shards, withoutNode); // wait for logs to reconcile
+ complete(shards, withoutNode); // SEALING-> SEALED for each Shard + journal flush
+ }
+
+ private static void initiate(Set<ShardMetadata> shards, @Nullable NodeId withoutNode)
+ {
+ List<Future<Void>> futures = new ArrayList<>(shards.size());
+ for (ShardMetadata shard : shards)
+ futures.add(initiate(shard, withoutNode));
+ try
+ {
+ FutureCombiner.allOf(futures).get(DatabaseDescriptor.getRpcTimeout(MILLISECONDS), MILLISECONDS);
+ }
+ catch (InterruptedException | ExecutionException | TimeoutException e)
+ {
+ throw new RuntimeException("Failed to initiate sealing", e);
+ }
+ }
+
+ /**
+ * Fence allocation on every participant of the obsoleted shard (ACTIVE -> SEALING) by marking the shard
+ * SEALING on each.
+ * TODO (expected): this assumes every participant has already enacted the topology change that obsoleted the shard (so
+ * it no longer allocates new ids on it). The ProgressBarrier gating bootstrap()/MID_JOIN only waits for
+ * EACH_QUORUM of the affected replicas, NOT all live replicas, so a live-but-lagging participant could still
+ * be on the old shard. Bringing each participant up to the seal epoch must be done per-replica (not
+ * per-shard) and will be added separately. TLDR - add a step to guaranteed up-to-speed rest of replicas epochs.
+ */
+ private static AsyncPromise<Void> initiate(ShardMetadata shard, @Nullable NodeId withoutNode)
+ {
+ return InitSealing.initiate(shard.keyspace, shard.sinceEpoch, shard.range, toEndpoints(shard.participants, withoutNode));
+ }
+
+ private static void drain(Set<ShardMetadata> shards, @Nullable NodeId withoutNode)
+ {
+ long deadlineNanos = nanoTime() + DatabaseDescriptor.getRpcTimeout(NANOSECONDS);
+ Set<ShardMetadata> pending = new HashSet<>(shards);
+ while (true)
+ {
+ // non-blocking poll: each participant reports its drain status immediately; retry until all drained
+ pending.removeIf(shard -> drain(shard, withoutNode));
+ if (pending.isEmpty())
+ return;
+ if (nanoTime() >= deadlineNanos)
+ throw new RuntimeException("Timed out draining shards; still pending: " + pending);
+ LockSupport.parkNanos(SECONDS.toNanos(1));
+ }
+ }
+
+ /**
+ * Poll every participant of the SEALING shard once for drain status; true iff all report that no mutation
+ * id allocated before the SEALING fence is still applying locally.
+ */
+ private static boolean drain(ShardMetadata shard, @Nullable NodeId withoutNode)
+ {
+ try
+ {
+ return Drain.poll(shard.keyspace, shard.sinceEpoch, shard.range, toEndpoints(shard.participants, withoutNode))
+ .get(DatabaseDescriptor.getRpcTimeout(MILLISECONDS), MILLISECONDS);
+ }
+ catch (InterruptedException | ExecutionException | TimeoutException e)
+ {
+ throw new RuntimeException(format("Failed to poll drain for shard %s", shard), e);
+ }
+ }
+
+ /**
+ * For each of the shards, capture the witnessed union then poll each participant until it has
+ * caught up and witnessed the entire offset union itself.
+ */
+ private static void reconcile(Set<ShardMetadata> shards, @Nullable NodeId withoutNode)
+ {
+ for (ShardMetadata shard : shards)
+ reconcile(shard, withoutNode);
+ }
+
+ private static void reconcile(ShardMetadata shard, @Nullable NodeId withoutNode)
+ {
+ // capture shard's witnessed offsets once
+ Log2OffsetsMap.Mutable offsets = captureWitnessedOffsets(shard, withoutNode);
+ // poll every participant until they've each witnessed the union of offsets
+ for (InetAddressAndPort endpoint : toEndpoints(shard.participants, withoutNode))
+ pollUntilWitnesses(shard, offsets, endpoint);
+ }
+
+ private static Log2OffsetsMap.Mutable captureWitnessedOffsets(ShardMetadata shard, @Nullable NodeId withoutNode)
+ {
+ List<InetAddressAndPort> endpoints = toEndpoints(shard.participants, withoutNode);
+ try
+ {
+ return ReconcileCapture.capture(shard.keyspace, shard.sinceEpoch, shard.range, endpoints)
+ .get(DatabaseDescriptor.getRpcTimeout(MILLISECONDS), MILLISECONDS);
+ }
+ catch (InterruptedException | ExecutionException | TimeoutException e)
+ {
+ throw new RuntimeException(format("Failed to capture witnessed offsets for shard %s", shard), e);
+ }
+ }
+
+ private static void pollUntilWitnesses(ShardMetadata shard, Log2OffsetsMap.Mutable offsets, InetAddressAndPort endpoint)
+ {
+ // TODO (expected): use a longer timeout
+ long deadlineNanos = nanoTime() + DatabaseDescriptor.getRpcTimeout(NANOSECONDS);
+
+ while (true)
+ {
+ if (nanoTime() >= deadlineNanos)
+ throw new RuntimeException("Timed out reconciling shard: " + shard);
+ try
+ {
+ boolean witnessed =
+ ReconcilePoll.poll(shard.keyspace, shard.sinceEpoch, shard.range, offsets, endpoint)
+ .get(DatabaseDescriptor.getRpcTimeout(MILLISECONDS), MILLISECONDS);
+ if (witnessed)
+ return;
+ }
+ catch (InterruptedException | ExecutionException | TimeoutException e)
+ {
+ throw new RuntimeException(format("Failed to poll reconcile for shard %s", shard), e);
+ }
+
+ LockSupport.parkNanos(SECONDS.toNanos(1));
+ }
+ }
+
+ private static void complete(Set<ShardMetadata> shards, @Nullable NodeId withoutNode)
+ {
+ List<Future<Void>> futures = new ArrayList<>(shards.size());
+ for (ShardMetadata shard : shards)
+ futures.add(complete(shard, withoutNode));
+
+ try
+ {
+ FutureCombiner.allOf(futures).get(DatabaseDescriptor.getRpcTimeout(MILLISECONDS), MILLISECONDS);
+ }
+ catch (InterruptedException | ExecutionException | TimeoutException e)
+ {
+ throw new RuntimeException("Failed to complete sealing", e);
+ }
+ }
+
+ /**
+ * Promote every participant of the (drained and reconciled) shard from SEALING to SEALED.
+ */
+ private static AsyncPromise<Void> complete(ShardMetadata shard, @Nullable NodeId withoutNode)
+ {
+ return CompleteSealing.complete(shard.keyspace, shard.sinceEpoch, shard.range, toEndpoints(shard.participants, withoutNode));
+ }
+
+ /**
+ * A bootstrapping node knows nothing about the shards that currently exist,
+ * so it must collect the list of shards from the existing replicas before
+ * it can proceed with the rest of the sealing steps.
+ */
+ public static final class FetchShards
+ {
+ public static final class Request
+ {
+ final String keyspace;
+ final long beforeEpoch;
+ final Range<Token> range;
+
+ Request(String keyspace, long beforeEpoch, Range<Token> range)
+ {
+ this.keyspace = keyspace;
+ this.beforeEpoch = beforeEpoch;
+ this.range = range;
+ }
+ }
+
+ public static final class Response
+ {
+ final Set<ShardMetadata> shards;
+
+ Response(Set<ShardMetadata> shards)
+ {
+ this.shards = shards;
+ }
+ }
+
+ /**
+ * Query all replicas in a group in parallel for the shards over {@code ranges} obsoleted by an
+ * in-flight topology change (those with {@code sinceEpoch < beforeEpoch}). Resembles
+ * {@link ShardMetadataRequest#queryPeers}, but waits for every (live) replica to respond and
+ * merges their deduplicated shards rather than taking the first response.
+ */
+ public static AsyncPromise<Set<ShardMetadata>> fetch(
+ String keyspace, long beforeEpoch, Range<Token> range, Set<InetAddressAndPort> endpoints)
+ {
+ Preconditions.checkArgument(!endpoints.isEmpty());
+ AsyncPromise<Set<ShardMetadata>> promise = new AsyncPromise<>();
+ Set<ShardMetadata> merged = ConcurrentHashMap.newKeySet();
+
+ RequestCallback<Response> callback = new RequestCallback<>()
+ {
+ private final AtomicInteger remaining = new AtomicInteger(endpoints.size());
+
+ @Override
+ public void onResponse(Message<Response> msg)
+ {
+ merged.addAll(msg.payload.shards);
+ if (remaining.decrementAndGet() == 0)
+ promise.trySuccess(merged);
+ }
+
+ @Override
+ public void onFailure(InetAddressAndPort from, RequestFailure failure)
+ {
+ // happy path assumes all replicas are up; fail the whole fetch if any can't respond
+ promise.tryFailure(new RuntimeException(format("Failed to fetch shards to seal from %s for keyspace %s: %s", from, keyspace, failure)));
+ }
+
+ @Override
+ public boolean invokeOnFailure()
+ {
+ return true;
+ }
+ };
+
+ Message<Request> message = Message.out(Verb.MT_FETCH_SHARDS_REQ, new Request(keyspace, beforeEpoch, range));
+ for (InetAddressAndPort peer : endpoints)
+ MessagingService.instance().sendWithCallback(message, peer, callback);
+
+ return promise;
+ }
+
+ public static final IVerbHandler<Request> verbHandler = message ->
+ {
+ MutationTrackingService.ensureEnabled();
+ Request request = message.payload;
+ Set<ShardMetadata> shards = new HashSet<>();
+ MutationTrackingService.instance().forEachShardInKeyspace(request.keyspace, shard -> {
+ if (shard.sinceEpoch < request.beforeEpoch
+ && (request.range.contains(shard.range) || shard.range.contains(request.range))
+ && !shard.isSealed())
+ shards.add(new ShardMetadata(shard.keyspace, shard.sinceEpoch, shard.range, shard.participants));
+ });
+ MessagingService.instance().send(message.responseWith(new Response(shards)), message.from());
+ };
+
+ public static final VersionedSerializer<Request> requestSerializer = new VersionedSerializer<>()
+ {
+ @Override
+ public void serialize(Request r, DataOutputPlus out, Version version) throws IOException
+ {
+ out.writeUTF(r.keyspace);
+ out.writeLong(r.beforeEpoch);
+ AbstractBounds.tokenSerializer.serialize(r.range, out, version.messagingVersion());
+ }
+
+ @Override
+ public Request deserialize(DataInputPlus in, Version version) throws IOException
+ {
+ String keyspace = in.readUTF();
+ long beforeEpoch = in.readLong();
+ Range<Token> range = (Range<Token>) AbstractBounds.tokenSerializer.deserialize(in, IPartitioner.global(), version.messagingVersion());
+ return new Request(keyspace, beforeEpoch, range);
+ }
+
+ @Override
+ public long serializedSize(Request r, Version version)
+ {
+ long size = TypeSizes.sizeof(r.keyspace);
+ size += TypeSizes.sizeof(r.beforeEpoch);
+ size += AbstractBounds.tokenSerializer.serializedSize(r.range, version.messagingVersion());
+ return size;
+ }
+ };
+
+ public static final VersionedSerializer<Response> responseSerializer = new VersionedSerializer<>()
+ {
+ @Override
+ public void serialize(Response r, DataOutputPlus out, Version version) throws IOException
+ {
+ CollectionSerializers.serializeCollection(r.shards, out, version, ShardMetadata.serializer);
+ }
+
+ @Override
+ public Response deserialize(DataInputPlus in, Version version) throws IOException
+ {
+ return new Response(CollectionSerializers.deserializeSet(in, version, ShardMetadata.serializer));
+ }
+
+ @Override
+ public long serializedSize(Response r, Version version)
+ {
+ return CollectionSerializers.serializedCollectionSize(r.shards, version, ShardMetadata.serializer);
+ }
+ };
+ }
+
+ public static final class InitSealing
+ {
+ public static final class Request
+ {
+ final String keyspace;
+ final long sinceEpoch;
+ final Range<Token> range;
+
+ Request(String keyspace, long sinceEpoch, Range<Token> range)
+ {
+ this.keyspace = keyspace;
+ this.sinceEpoch = sinceEpoch;
+ this.range = range;
+ }
+ }
+
+ // cannot use NoPayload, because its serializer cannot be wrapped inside mtEmbedded()
+ public static final class Response
+ {
+ private static final Response instance = new Response();
+ }
+
+ /**
+ * Tell every participant of an obsoleted shard to fence id allocation (mark it SEALING)
+ */
+ public static AsyncPromise<Void> initiate(
+ String keyspace, long sinceEpoch, Range<Token> range, List<InetAddressAndPort> endpoints)
+ {
+ AsyncPromise<Void> promise = new AsyncPromise<>();
+
+ RequestCallback<Response> callback = new RequestCallback<>()
+ {
+ private final AtomicInteger remaining = new AtomicInteger(endpoints.size());
+
+ @Override
+ public void onResponse(Message<Response> msg)
+ {
+ if (remaining.decrementAndGet() == 0)
+ promise.trySuccess(null);
+ }
+
+ @Override
+ public void onFailure(InetAddressAndPort from, RequestFailure failure)
+ {
+ promise.tryFailure(new RuntimeException(format("Failed to initiate sealing on %s for keyspace %s: %s", from, keyspace, failure)));
+ }
+
+ @Override
+ public boolean invokeOnFailure()
+ {
+ return true;
+ }
+ };
+
+ Message<Request> message = Message.out(Verb.MT_INIT_SEALING_REQ, new Request(keyspace, sinceEpoch, range));
+ for (InetAddressAndPort peer : endpoints)
+ MessagingService.instance().sendWithCallback(message, peer, callback);
+
+ return promise;
+ }
+
+ public static final IVerbHandler<Request> verbHandler = message -> {
+ MutationTrackingService.ensureEnabled();
+ Request request = message.payload;
+ MutationTrackingService.instance().markShardSealing(request.keyspace, request.sinceEpoch, request.range);
+ MessagingService.instance().send(message.responseWith(Response.instance), message.from());
+ };
+
+ public static final VersionedSerializer<Request> requestSerializer = new VersionedSerializer<>()
+ {
+ @Override
+ public void serialize(Request r, DataOutputPlus out, Version version) throws IOException
+ {
+ out.writeUTF(r.keyspace);
+ out.writeLong(r.sinceEpoch);
+ AbstractBounds.tokenSerializer.serialize(r.range, out, version.messagingVersion());
+ }
+
+ @Override
+ public Request deserialize(DataInputPlus in, Version version) throws IOException
+ {
+ String keyspace = in.readUTF();
+ long sinceEpoch = in.readLong();
+ Range<Token> range = (Range<Token>) AbstractBounds.tokenSerializer.deserialize(in, IPartitioner.global(), version.messagingVersion());
+ return new Request(keyspace, sinceEpoch, range);
+ }
+
+ @Override
+ public long serializedSize(Request r, Version version)
+ {
+ long size = TypeSizes.sizeof(r.keyspace);
+ size += TypeSizes.sizeof(r.sinceEpoch);
+ size += AbstractBounds.tokenSerializer.serializedSize(r.range, version.messagingVersion());
+ return size;
+ }
+ };
+
+ public static final UnversionedSerializer<Response> responseSerializer = new UnversionedSerializer<>()
+ {
+ @Override
+ public void serialize(Response r, DataOutputPlus out)
+ {
+ }
+
+ @Override
+ public Response deserialize(DataInputPlus in)
+ {
+ return Response.instance;
+ }
+
+ @Override
+ public long serializedSize(Response r)
+ {
+ return 0;
+ }
+ };
+ }
+
+ public static final class Drain
+ {
+ public static final class Request
+ {
+ final String keyspace;
+ final long sinceEpoch;
+ final Range<Token> range;
+
+ Request(String keyspace, long sinceEpoch, Range<Token> range)
+ {
+ this.keyspace = keyspace;
+ this.sinceEpoch = sinceEpoch;
+ this.range = range;
+ }
+ }
+
+ public static final class Response
+ {
+ final boolean drained;
+
+ Response(boolean drained)
+ {
+ this.drained = drained;
+ }
+ }
+
+ /**
+ * Poll every participant of a SEALING shard for whether it has drained its in-flight local
+ * writes. Succeeds with {@code true} iff every participant reports drained.
+ */
+ public static AsyncPromise<Boolean> poll(
+ String keyspace, long sinceEpoch, Range<Token> range, List<InetAddressAndPort> endpoints)
+ {
+ AsyncPromise<Boolean> promise = new AsyncPromise<>();
+
+ RequestCallback<Response> callback = new RequestCallback<>()
+ {
+ private final AtomicInteger remaining = new AtomicInteger(endpoints.size());
+ private volatile boolean allDrained = true;
+
+ @Override
+ public void onResponse(Message<Response> msg)
+ {
+ if (!msg.payload.drained)
+ allDrained = false;
+ if (remaining.decrementAndGet() == 0)
+ promise.trySuccess(allDrained);
+ }
+
+ @Override
+ public void onFailure(InetAddressAndPort from, RequestFailure failure)
+ {
+ promise.tryFailure(new RuntimeException(format("Failed to poll drain on %s for keyspace %s: %s", from, keyspace, failure)));
+ }
+
+ @Override
+ public boolean invokeOnFailure()
+ {
+ return true;
+ }
+ };
+
+ Message<Request> message = Message.out(Verb.MT_DRAIN_REQ, new Request(keyspace, sinceEpoch, range));
+ for (InetAddressAndPort peer : endpoints)
+ MessagingService.instance().sendWithCallback(message, peer, callback);
+
+ return promise;
+ }
+
+ public static final IVerbHandler<Request> verbHandler = message -> {
+ MutationTrackingService.ensureEnabled();
+ Request request = message.payload;
+ boolean drained = MutationTrackingService.instance().isShardDrained(request.keyspace, request.sinceEpoch, request.range);
+ MessagingService.instance().send(message.responseWith(new Response(drained)), message.from());
+ };
+
+ public static final VersionedSerializer<Request> requestSerializer = new VersionedSerializer<>()
+ {
+ @Override
+ public void serialize(Request r, DataOutputPlus out, Version version) throws IOException
+ {
+ out.writeUTF(r.keyspace);
+ out.writeLong(r.sinceEpoch);
+ AbstractBounds.tokenSerializer.serialize(r.range, out, version.messagingVersion());
+ }
+
+ @Override
+ public Request deserialize(DataInputPlus in, Version version) throws IOException
+ {
+ String keyspace = in.readUTF();
+ long sinceEpoch = in.readLong();
+ Range<Token> range = (Range<Token>) AbstractBounds.tokenSerializer.deserialize(in, IPartitioner.global(), version.messagingVersion());
+ return new Request(keyspace, sinceEpoch, range);
+ }
+
+ @Override
+ public long serializedSize(Request r, Version version)
+ {
+ long size = TypeSizes.sizeof(r.keyspace);
+ size += TypeSizes.sizeof(r.sinceEpoch);
+ size += AbstractBounds.tokenSerializer.serializedSize(r.range, version.messagingVersion());
+ return size;
+ }
+ };
+
+ public static final UnversionedSerializer<Response> responseSerializer = new UnversionedSerializer<>()
+ {
+ @Override
+ public void serialize(Response r, DataOutputPlus out) throws IOException
+ {
+ out.writeBoolean(r.drained);
+ }
+
+ @Override
+ public Response deserialize(DataInputPlus in) throws IOException
+ {
+ return new Response(in.readBoolean());
+ }
+
+ @Override
+ public long serializedSize(Response r)
+ {
+ return TypeSizes.sizeof(r.drained);
+ }
+ };
+ }
+
+ public static final class ReconcileCapture
+ {
+ public static final class Request
+ {
+ final String keyspace;
+ final long sinceEpoch;
+ final Range<Token> range;
+
+ Request(String keyspace, long sinceEpoch, Range<Token> range)
+ {
+ this.keyspace = keyspace;
+ this.sinceEpoch = sinceEpoch;
+ this.range = range;
+ }
+ }
+
+ public static final class Response
+ {
+ final Log2OffsetsMap<?> witnessed;
+
+ Response(Log2OffsetsMap<?> witnessed)
+ {
+ this.witnessed = witnessed;
+ }
+ }
+
+ public static AsyncPromise<Log2OffsetsMap.Mutable> capture(
+ String keyspace, long sinceEpoch, Range<Token> range, List<InetAddressAndPort> endpoints)
+ {
+ AsyncPromise<Log2OffsetsMap.Mutable> promise = new AsyncPromise<>();
+ Log2OffsetsMap.Mutable union = new Log2OffsetsMap.Mutable();
+
+ RequestCallback<Response> callback = new RequestCallback<>()
+ {
+ private final AtomicInteger remaining = new AtomicInteger(endpoints.size());
+
+ @Override
+ public void onResponse(Message<Response> msg)
+ {
+ synchronized (union)
+ {
+ union.addAll(msg.payload.witnessed);
+ }
+ if (remaining.decrementAndGet() == 0)
+ promise.trySuccess(union);
+ }
+
+ @Override
+ public void onFailure(InetAddressAndPort from, RequestFailure failure)
+ {
+ promise.tryFailure(new RuntimeException(format("Failed to capture witnessed offsets on %s for keyspace %s: %s", from, keyspace, failure)));
+ }
+
+ @Override
+ public boolean invokeOnFailure()
+ {
+ return true;
+ }
+ };
+
+ Message<Request> message = Message.out(Verb.MT_RECONCILE_CAPTURE_REQ, new Request(keyspace, sinceEpoch, range));
+ for (InetAddressAndPort peer : endpoints)
+ MessagingService.instance().sendWithCallback(message, peer, callback);
+
+ return promise;
+ }
+
+ public static final IVerbHandler<Request> verbHandler = message -> {
+ MutationTrackingService.ensureEnabled();
+ Request request = message.payload;
+ Log2OffsetsMap<?> witnessed =
+ MutationTrackingService.instance()
+ .collectLocallyWitnessedOffsets(request.keyspace, request.sinceEpoch, request.range);
+ MessagingService.instance().send(message.responseWith(new Response(witnessed)), message.from());
+ };
+
+ public static final VersionedSerializer<Request> requestSerializer = new VersionedSerializer<>()
+ {
+ @Override
+ public void serialize(Request r, DataOutputPlus out, Version version) throws IOException
+ {
+ out.writeUTF(r.keyspace);
+ out.writeLong(r.sinceEpoch);
+ AbstractBounds.tokenSerializer.serialize(r.range, out, version.messagingVersion());
+ }
+
+ @Override
+ public Request deserialize(DataInputPlus in, Version version) throws IOException
+ {
+ String keyspace = in.readUTF();
+ long sinceEpoch = in.readLong();
+ Range<Token> range = (Range<Token>) AbstractBounds.tokenSerializer.deserialize(in, IPartitioner.global(), version.messagingVersion());
+ return new Request(keyspace, sinceEpoch, range);
+ }
+
+ @Override
+ public long serializedSize(Request r, Version version)
+ {
+ long size = TypeSizes.sizeof(r.keyspace);
+ size += TypeSizes.sizeof(r.sinceEpoch);
+ size += AbstractBounds.tokenSerializer.serializedSize(r.range, version.messagingVersion());
+ return size;
+ }
+ };
+
+ public static final UnversionedSerializer<Response> responseSerializer = new UnversionedSerializer<>()
+ {
+ @Override
+ public void serialize(Response r, DataOutputPlus out) throws IOException
+ {
+ Log2OffsetsMap.serializer.serialize(r.witnessed, out);
+ }
+
+ @Override
+ public Response deserialize(DataInputPlus in) throws IOException
+ {
+ return new Response(Log2OffsetsMap.serializer.deserialize(in));
+ }
+
+ @Override
+ public long serializedSize(Response r)
+ {
+ return Log2OffsetsMap.serializer.serializedSize(r.witnessed);
+ }
+ };
+ }
+
+ public static final class ReconcilePoll
+ {
+ public static final class Request
+ {
+ final String keyspace;
+ final long sinceEpoch;
+ final Range<Token> range;
+ final Log2OffsetsMap<?> offsets;
+
+ Request(String keyspace, long sinceEpoch, Range<Token> range, Log2OffsetsMap<?> offsets)
+ {
+ this.keyspace = keyspace;
+ this.sinceEpoch = sinceEpoch;
+ this.range = range;
+ this.offsets = offsets;
+ }
+ }
+
+ public static final class Response
+ {
+ final boolean witnessed;
+
+ Response(boolean witnessed)
+ {
+ this.witnessed = witnessed;
+ }
+ }
+
+ public static AsyncPromise<Boolean> poll(
+ String keyspace, long sinceEpoch, Range<Token> range, Log2OffsetsMap.Mutable offsets, InetAddressAndPort endpoint)
+ {
Review Comment:
```suggestion
{
Preconditions.checkArgument(!endpoints.isEmpty());
```
##########
src/java/org/apache/cassandra/replication/SealingCoordinator.java:
##########
@@ -0,0 +1,1401 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.cassandra.replication;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.TimeoutException;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.concurrent.locks.LockSupport;
+
+import javax.annotation.Nonnull;
+import javax.annotation.Nullable;
+
+import com.google.common.base.Preconditions;
+
+import org.apache.cassandra.config.DatabaseDescriptor;
+import org.apache.cassandra.db.ColumnFamilyStore;
+import org.apache.cassandra.db.Keyspace;
+import org.apache.cassandra.db.TypeSizes;
+import org.apache.cassandra.dht.AbstractBounds;
+import org.apache.cassandra.dht.IPartitioner;
+import org.apache.cassandra.dht.Range;
+import org.apache.cassandra.dht.Token;
+import org.apache.cassandra.exceptions.RequestFailure;
+import org.apache.cassandra.io.UnversionedSerializer;
+import org.apache.cassandra.io.util.DataInputPlus;
+import org.apache.cassandra.io.util.DataOutputPlus;
+import org.apache.cassandra.locator.EndpointsByReplica;
+import org.apache.cassandra.locator.EndpointsForRange;
+import org.apache.cassandra.locator.InetAddressAndPort;
+import org.apache.cassandra.locator.RangesAtEndpoint;
+import org.apache.cassandra.locator.Replica;
+import org.apache.cassandra.net.IVerbHandler;
+import org.apache.cassandra.net.Message;
+import org.apache.cassandra.net.MessagingService;
+import org.apache.cassandra.net.RequestCallback;
+import org.apache.cassandra.net.Verb;
+import org.apache.cassandra.schema.KeyspaceMetadata;
+import org.apache.cassandra.tcm.ClusterMetadata;
+import org.apache.cassandra.tcm.membership.NodeId;
+import org.apache.cassandra.tcm.ownership.DataPlacement;
+import org.apache.cassandra.tcm.ownership.MovementMap;
+import org.apache.cassandra.tcm.ownership.PlacementDeltas;
+import org.apache.cassandra.tcm.ownership.VersionedEndpoints;
+import org.apache.cassandra.tcm.sequences.LeaveStreams;
+import org.apache.cassandra.utils.CollectionSerializers;
+import org.apache.cassandra.utils.FBUtilities;
+import org.apache.cassandra.utils.concurrent.AsyncPromise;
+import org.apache.cassandra.utils.concurrent.Future;
+import org.apache.cassandra.utils.concurrent.FutureCombiner;
+
+import static java.lang.String.format;
+import static java.util.concurrent.TimeUnit.MILLISECONDS;
+import static java.util.concurrent.TimeUnit.NANOSECONDS;
+import static java.util.concurrent.TimeUnit.SECONDS;
+import static org.apache.cassandra.utils.Clock.Global.nanoTime;
+
+/*
+ * Sealing {@link Shard}s once a topology change obsoletes them.
+ *
+ * Happy Path Sequence:
+ * 1. Bring every participant up to (at least) this seal's epoch and fence allocation, so that everyone
+ * stops writing to the old shards; then wait until all in-progress mutations have been done.
+ * 2. Same for tracked transfers
+ * 3. Wait until all participating replicas have reconciled the union of their witnessed ids
+ * 4. Mark the shard as fully sealed
+ * 5. Stop including the shard in mutation summaries for read requests.
+ *
+ * Unhappy Path Sequence (not yet implemented):
+ * 1. Wait until everyone who's up stops writing to the old shards and all in-progress mutations have been done
+ * 2. Same for tracked transfers
+ * 3. Wait until all *live* participating replicas have reconciled the union of their witnessed ids
+ * 4. Mark the shard as partially sealed, label it with the set of participants that were able to seal
+ * 5. When a down node comes up, have it talk to other nodes and learn about sealing, and what mutations/logs
+ * they have exchanged. Diff with what we have - in system tables and in the journal (though system table
+ * plus local replay should give us an up-to-date metadata state for the shard)
+ */
+public final class SealingCoordinator
+{
+ /*
+ * BOOTSTRAP sequence illustrated. Stages in which write placements
+ * remain unchanged, and only read placements change, are skipped
+ * here as irrelevant to sharding / sealing.
+ *
+ * There is a tracked keyspace with RF=3.
+ * A node n35 with token=35 is added in the middle of the ring.
+ *
+ * 1. Initial state at e0:
+ *
+ * (10,20] | {n20,n30,n40} | e0 | ACTIVE
+ * (20,30] | {n30,n40,n50} | e0 | ACTIVE
+ * (30,40] | {n40,n50,n60} | e0 | ACTIVE
+ *
+ * 2. PREPARE_JOIN at e1:
+ *
+ * (10,20] | {n20,n30,n40} | e0 | ACTIVE
+ * (20,30] | {n30,n40,n50} | e0 | ACTIVE
+ * (30,40] | {n40,n50,n60} | e0 | ACTIVE
+ * + (30,35] | {n40,n50,n60} | e1 | ACTIVE
+ * + (35,40] | {n40,n50,n60} | e1 | ACTIVE
+ *
+ * 3. START_JOIN at e2:
+ *
+ * (10,20] | {n20,n30,n40} | e0 | ACTIVE
+ * + (10,20] | {n20,n30,n35,n40} | e2 | ACTIVE
+ * (20,30] | {n30,n40,n50} | e0 | ACTIVE
+ * + (20,30] | {n30,n35,n40,n50} | e2 | ACTIVE
+ * (30,40] | {n40,n50,n60} | e0 | ACTIVE
+ * (30,35] | {n40,n50,n60} | e1 | ACTIVE
+ * + (30,35] | {n35,n40,n50,n60} | e2 | ACTIVE
+ * (35,40] | {n40,n50,n60} | e1 | ACTIVE
+ *
+ * 4. MID_JOIN at e3 [*]:
+ *
+ * S (10,20] | {n20,n30,n40} | e0 | SEALED
+ * (10,20] | {n20,n30,n35,n40} | e2 | ACTIVE
+ * S (20,30] | {n30,n40,n50} | e0 | SEALED
+ * (20,30] | {n30,n35,n40,n50} | e2 | ACTIVE
+ * S (30,40] | {n40,n50,n60} | e0 | SEALED
+ * S (30,35] | {n40,n50,n60} | e1 | SEALED
+ * (30,35] | {n35,n40,n50,n60} | e2 | ACTIVE
+ * (35,40] | {n40,n50,n60} | e1 | ACTIVE
+ *
+ * 5. FINISH_JOIN at e4:
+ *
+ * (10,20] | {n20,n30,n40} | e0 | SEALED
+ * S (10,20] | {n20,n30,n35,n40} | e2 | SEALED
+ * + (10,20] | {n20,n30,n35} | e4 | ACTIVE
+ * (20,30] | {n30,n40,n50} | e0 | SEALED
+ * S (20,30] | {n30,n35,n40,n50} | e2 | SEALED
+ * + (20,30] | {n30,n35,n40} | e4 | ACTIVE
+ * (30,40] | {n40,n50,n60} | e0 | SEALED
+ * (30,35] | {n40,n50,n60} | e1 | SEALED
+ * S (30,35] | {n35,n40,n50,n60} | e2 | SEALED
+ * + (30,35] | {n35,n40,n50} | e4 | ACTIVE
+ * (35,40] | {n40,n50,n60} | e1 | ACTIVE
+ *
+ * [*] At this stage, after sealing the shards, we stream the sstables.
+ * All the offsets belonging to these sealed shards get stripped from
+ * stats component of streamed sstables. Otherwise, the new node
+ * won't be able to compact them or mark them as reconciled/repaired -
+ * it is rightfully unaware of the sealed shards.
+ */
+
+ /**
+ * Seal the shards obsoleted by bootstrap before the new node starts streaming.
+ */
+ public static void sealShardsAtMidJoin(ClusterMetadata metadata, MovementMap movements)
+ {
+ seal(discoverShardsAtMidJoinOrReplace(metadata, movements));
+ }
+
+ /**
+ * Seal the intermediate shards that were created during START_JOIN.
+ * TODO (required): this MUST eventually run while the affected ranges are still locked - between the
+ * removeFromWrites placement change and the range unlock. FinishJoin/FinishReplace currently apply
+ * removeFromWrites and the unlock atomically in a single transformation, so there is no committed
+ * dropped-but-locked epoch to hook. Until then, this runs just after FINISH_JOIN/FINISH_REPLACE,
+ * when the ranges are already unlocked. Left to implement for the unhappy path (AY).
+ */
+ public static void sealShardsAtFinishJoin(ClusterMetadata metadata, PlacementDeltas finishDelta)
+ {
+ seal(discoverShardsAtFinishJoinOrReplace(metadata, finishDelta, null));
+ }
+
+ /*
+ * REPLACE sequence illustrated. Stages in which write placements
+ * remain unchanged, and only read placements change, are skipped
+ * here as irrelevant to sharding / sealing.
+ *
+ * There is a tracked keyspace with RF=3.
+ * Node n40 is dead, and is being replaced with $n40.
+ *
+ * 1. Initial state at e0:
+ *
+ * (10,20] | {n20,n30,n40} | e0 | ACTIVE
+ * (20,30] | {n30,n40,n50} | e0 | ACTIVE
+ * (30,40] | {n40,n50,n60} | e0 | ACTIVE
+ *
+ * 2. START_REPLACE at e1:
+ *
+ * (10,20] | {n20,n30,n40} | e0 | ACTIVE
+ * + (10,20] | {n20,n30,n40,$n40}| e1 | ACTIVE
+ * (20,30] | {n30,n40,n50} | e0 | ACTIVE
+ * + (20,30] | {n30,n40,$n40,n50}| e1 | ACTIVE
+ * (30,40] | {n40,n50,n60} | e0 | ACTIVE
+ * + (30,40] | {n40,$n40,n50,n60}| e1 | ACTIVE
+ *
+ * 3. MID_REPLACE at e2 [*]:
+ *
+ * S (10,20] | {n20,n30,n40} | e0 | SEALED
+ * (10,20] | {n20,n30,n40,$n40}| e1 | ACTIVE
+ * S (20,30] | {n30,n40,n50} | e0 | SEALED
+ * (20,30] | {n30,n40,$n40,n50}| e1 | ACTIVE
+ * S (30,40] | {n40,n50,n60} | e0 | SEALED
+ * (30,40] | {n40,$n40,n50,n60}| e1 | ACTIVE
+ *
+ * 4. FINISH_REPLACE at e3:
+ *
+ * (10,20] | {n20,n30,n40} | e0 | SEALED
+ * S (10,20] | {n20,n30,n40,$n40}| e1 | SEALED
+ * + (10,20] | {n20,n30,$n40} | e3 | ACTIVE
+ * (20,30] | {n30,n40,n50} | e0 | SEALED
+ * S (20,30] | {n30,n40,$n40,n50}| e1 | SEALED
+ * + (20,30] | {n30,$n40,n50} | e3 | ACTIVE
+ * (30,40] | {n40,n50,n60} | e0 | SEALED
+ * S (30,40] | {n40,$n40,n50,n60}| e1 | SEALED
+ * + (30,40] | {$n40,n50,n60} | e3 | ACTIVE
+ *
+ * [*] At this stage, after sealing the shards, we stream the sstables.
+ * All the offsets belonging to these sealed shards get stripped from
+ * stats component of streamed sstables. Otherwise, the new node
+ * won't be able to compact them or mark them as reconciled/repaired -
+ * it is rightfully unaware of the sealed shards.
+ */
+
+ /**
+ * Seal the shards obsoleted by host replacement before the replacement node starts streaming.
+ * The {@code deadNode} node cannot participate in the sealing.
+ */
+ public static void sealShardsAtMidReplace(ClusterMetadata metadata, MovementMap movements, @Nonnull NodeId deadNode)
+ {
+ seal(discoverShardsAtMidJoinOrReplace(metadata, movements), deadNode);
+ }
+
+ public static void sealShardsAtFinishReplace(ClusterMetadata metadata, PlacementDeltas finishDelta, @Nonnull NodeId deadNode)
+ {
+ seal(discoverShardsAtFinishJoinOrReplace(metadata, finishDelta, deadNode), deadNode);
+ }
+
+ /**
+ * Discover all shards that need to be sealed at MID_JOIN or MID_REPLACE.
+ */
+ private static Set<ShardMetadata> discoverShardsAtMidJoinOrReplace(ClusterMetadata metadata, MovementMap movements)
+ {
+ List<Future<Set<ShardMetadata>>> futures = new ArrayList<>();
+
+ for (KeyspaceMetadata ksm : metadata.schema.getKeyspaces())
+ {
+ if (!ksm.params.replicationType.isTracked())
+ continue;
+
+ EndpointsByReplica acquired = movements.get(ksm.params.replication);
+ if (acquired.isEmpty()) // when the keyspace is not replicated to this DC
+ continue;
+
+ DataPlacement placement = metadata.placements.get(ksm.params.replication);
+
+ for (Map.Entry<Replica, EndpointsForRange> entry : acquired.entrySet())
+ {
+ Range<Token> range = entry.getKey().range();
+ Set<InetAddressAndPort> endpoints = entry.getValue().endpoints();
+ // cutoff: the epoch this node was added to the write placement for the range;
+ // shards over the range with sinceEpoch < cutoff are obsoleted by this bootstrap or replace.
+ long beforeEpoch = placement.writes.forRange(range).lastModified().getEpoch();
+ futures.add(FetchShards.fetch(ksm.name, beforeEpoch, range, endpoints));
+ }
+ }
+
+ Set<ShardMetadata> shards = new HashSet<>();
+ try
+ {
+ for (Set<ShardMetadata> fetched : FutureCombiner.allOf(futures).get(DatabaseDescriptor.getRpcTimeout(MILLISECONDS), MILLISECONDS))
+ shards.addAll(fetched);
+ }
+ catch (InterruptedException | ExecutionException | TimeoutException e)
+ {
+ throw new RuntimeException("Failed to fetch shards to seal", e);
+ }
+ return shards;
+ }
+
+ /**
+ * Discover all shards that need to be sealed at FINISH_JOIN or FINISH_REPLACE.
+ */
+ private static Set<ShardMetadata> discoverShardsAtFinishJoinOrReplace(ClusterMetadata metadata, PlacementDeltas finishDelta, @Nullable NodeId deadNode)
+ {
+ long finishEpoch = metadata.epoch.getEpoch();
+ Set<ShardMetadata> shards = new HashSet<>();
+ for (KeyspaceMetadata ksm : metadata.schema.getKeyspaces())
+ {
+ if (!ksm.params.replicationType.isTracked())
+ continue;
+
+ DataPlacement placement = metadata.placements.get(ksm.params.replication);
+ finishDelta.get(ksm.params.replication).writes.removals.flattenValues().forEach(removed -> {
+ Set<Integer> nodeIDs = new HashSet<>();
+ for (InetAddressAndPort ep : placement.writes.forRange(removed.range()).endpoints())
+ nodeIDs.add(metadata.directory.peerId(ep).id());
+ nodeIDs.add(deadNode != null ? deadNode.id() : metadata.directory.peerId(removed.endpoint()).id());
+ Participants participants = new Participants(nodeIDs);
+
+ MutationTrackingService.instance().forEachShardInKeyspace(ksm.name, shard -> {
+ if (shard.sinceEpoch < finishEpoch
+ && shard.range.equals(removed.range())
+ && shard.participants.equals(participants)
+ && !shard.isSealed())
+ shards.add(new ShardMetadata(shard.keyspace, shard.sinceEpoch, shard.range, shard.participants));
+ });
+ });
+ }
+ return shards;
+ }
+
+ /*
+ * UNBOOTSTRAP and REMOVENODE sequences illustrated (same sequence as far as sharding is concerned).
+ * Stages in which write placements remain unchanged, and only read placements change, are skipped
+ * here as irrelevant to sharding / sealing.
+ *
+ * There is a tracked keyspace with RF=3.
+ * A node n40 with token=40 is removed in the middle of the ring.
+ *
+ * 1. Initial state at e0:
+ *
+ * (10,20] | {n20,n30,n40} | e0 | ACTIVE
+ * (20,30] | {n30,n40,n50} | e0 | ACTIVE
+ * (30,40] | {n40,n50,n60} | e0 | ACTIVE
+ * (40,50] | {n50,n60,n70} | e0 | ACTIVE
+ *
+ * 2. START_LEAVE at e1:
+ *
+ * (10,20] | {n20,n30,n40} | e0 | ACTIVE
+ * + (10,20] | {n20,n30,n40,n50} | e1 | ACTIVE
+ * (20,30] | {n30,n40,n50} | e0 | ACTIVE
+ * + (20,30] | {n30,n40,n50,n60} | e1 | ACTIVE
+ * (30,40] | {n40,n50,n60} | e0 | ACTIVE
+ * + (30,40] | {n40,n50,n60,n70} | e1 | ACTIVE
+ * (40,50] | {n50,n60,n70} | e0 | ACTIVE
+ *
+ * 3. MID_LEAVE at e2 [*]:
+ *
+ * S (10,20] | {n20,n30,n40} | e0 | SEALED
+ * (10,20] | {n20,n30,n40,n50} | e1 | ACTIVE
+ * S (20,30] | {n30,n40,n50} | e0 | SEALED
+ * (20,30] | {n30,n40,n50,n60} | e1 | ACTIVE
+ * S (30,40] | {n40,n50,n60} | e0 | SEALED
+ * (30,40] | {n40,n50,n60,n70} | e1 | ACTIVE
+ * (40,50] | {n50,n60,n70} | e0 | ACTIVE
+ *
+ * 4. FINISH_LEAVE at e3:
+ *
+ * (10,20] | {n20,n30,n40} | e0 | SEALED
+ * S (10,20] | {n20,n30,n40,n50} | e1 | SEALED
+ * + (10,20] | {n20,n30,n50} | e3 | ACTIVE
+ * (20,30] | {n30,n40,n50} | e0 | SEALED
+ * S (20,30] | {n30,n40,n50,n60} | e1 | SEALED
+ * + (20,30] | {n30,n50,n60} | e3 | ACTIVE
+ * (30,40] | {n40,n50,n60} | e0 | SEALED
+ * S (30,40] | {n40,n50,n60,n70} | e1 | SEALED
+ * S (40,50] | {n50,n60,n70} | e0 | SEALED
+ * + (30,50] | {n50,n60,n70} | e3 | ACTIVE
+ *
+ * [*] At this stage, after sealing the shards, we stream the sstables.
+ * All the offsets belonging to these sealed shards get stripped from
+ * stats component of streamed sstables. Otherwise, the new node
+ * won't be able to compact them or mark them as reconciled/repaired -
+ * it is rightfully unaware of the sealed shards.
+ */
+
+ /**
+ * Seal the shards obsoleted by START_LEAVE before streaming, for both UNBOOTSTRAP and REMOVENODE.
+ */
+ public static void sealShardsAtMidLeave(ClusterMetadata metadata, PlacementDeltas startDelta, NodeId leavingOrDeparted, LeaveStreams.Kind kind)
+ {
+ InetAddressAndPort leavingOrDepartedEndpoint = metadata.directory.endpoint(leavingOrDeparted);
+
+ List<Future<Set<ShardMetadata>>> futures = new ArrayList<>();
+ for (KeyspaceMetadata ksm : metadata.schema.getKeyspaces())
+ {
+ if (!ksm.params.replicationType.isTracked())
+ continue;
+
+ DataPlacement placement = metadata.placements.get(ksm.params.replication);
+ startDelta.get(ksm.params.replication).writes.additions.flattenValues().forEach(added -> {
+ Range<Token> range = added.range();
+ long beforeEpoch = placement.writes.forRange(range).lastModified().getEpoch();
+ Set<InetAddressAndPort> survivors = new HashSet<>(placement.writes.forRange(range).endpoints());
+ survivors.remove(leavingOrDepartedEndpoint);
+ futures.add(FetchShards.fetch(ksm.name, beforeEpoch, range, survivors));
+ });
+ }
+
+ Set<ShardMetadata> shards = new HashSet<>();
+ try
+ {
+ for (Set<ShardMetadata> fetched : FutureCombiner.allOf(futures).get(DatabaseDescriptor.getRpcTimeout(MILLISECONDS), MILLISECONDS))
+ for (ShardMetadata shard : fetched)
+ if (shard.participants.contains(leavingOrDeparted.id()))
+ shards.add(shard);
+ }
+ catch (InterruptedException | ExecutionException | TimeoutException e)
+ {
+ throw new RuntimeException("Failed to fetch shards to seal for leave", e);
+ }
+
+ seal(shards, kind == LeaveStreams.Kind.UNBOOTSTRAP ? null : leavingOrDeparted);
+ }
+
+ /**
+ * Seal the shards that need to be sealed at LEAVE_FINISH for both UNBOOTSTRAP and REMOVENODE.
+ */
+ public static void sealShardsAtFinishLeave(ClusterMetadata postFinish, PlacementDeltas finishDelta, NodeId leavingOrDeparted, LeaveStreams.Kind kind)
+ {
+ long finishEpoch = postFinish.epoch.getEpoch();
+ InetAddressAndPort leavingOrDepartedEndpoint = postFinish.directory.endpoint(leavingOrDeparted);
+
+ List<Future<Set<ShardMetadata>>> futures = new ArrayList<>();
+ for (KeyspaceMetadata ksm : postFinish.schema.getKeyspaces())
+ {
+ if (!ksm.params.replicationType.isTracked())
+ continue;
+
+ DataPlacement placement = postFinish.placements.get(ksm.params.replication);
+ RangesAtEndpoint removals = finishDelta.get(ksm.params.replication).writes.removals.get(leavingOrDepartedEndpoint);
+ for (Replica removedReplica : removals)
+ {
+ // FINISH_LEAVE merges the leavingOrDeparted node's (pre-merge) range with an adjacent one. Query the *merged*
+ // range (not the pre-merge range) so discovery covers both the leavingOrDeparted node's own obsoleted
+ // generations and the merge-half - the adjacent pre-merge range folded into the merged range, whose
+ // shard never listed the leavingOrDeparted node. matchRange() yields the containing merged group + its
+ // current (survivor) write endpoints.
+ VersionedEndpoints.ForRange group = placement.writes.matchRange(removedReplica.range());
+ Range<Token> mergedRange = group.range();
+ Set<InetAddressAndPort> survivors = new HashSet<>(group.endpoints());
+ survivors.remove(leavingOrDepartedEndpoint);
+ futures.add(FetchShards.fetch(ksm.name, finishEpoch, mergedRange, survivors));
+ }
+ }
+
+ Set<ShardMetadata> shards = new HashSet<>();
+ try
+ {
+ for (Set<ShardMetadata> fetched : FutureCombiner.allOf(futures).get(DatabaseDescriptor.getRpcTimeout(MILLISECONDS), MILLISECONDS))
+ shards.addAll(fetched);
+ }
+ catch (InterruptedException | ExecutionException | TimeoutException e)
+ {
+ throw new RuntimeException("Failed to fetch shards to seal for leave finish", e);
+ }
+
+ seal(shards, kind == LeaveStreams.Kind.UNBOOTSTRAP ? null : leavingOrDeparted);
+ }
+
+ /*
+ * Sealing logic.
+ */
+
+ private static void seal(Set<ShardMetadata> shards)
+ {
+ seal(shards, null);
+ }
+
+ private static void seal(Set<ShardMetadata> shards, @Nullable NodeId withoutNode)
+ {
+ initiate(shards, withoutNode); // ACTIVE -> SEALING for each Shard
+ drain(shards, withoutNode); // drain in-flight local writes
+ reconcile(shards, withoutNode); // wait for logs to reconcile
+ complete(shards, withoutNode); // SEALING-> SEALED for each Shard + journal flush
+ }
+
+ private static void initiate(Set<ShardMetadata> shards, @Nullable NodeId withoutNode)
+ {
+ List<Future<Void>> futures = new ArrayList<>(shards.size());
+ for (ShardMetadata shard : shards)
+ futures.add(initiate(shard, withoutNode));
+ try
+ {
+ FutureCombiner.allOf(futures).get(DatabaseDescriptor.getRpcTimeout(MILLISECONDS), MILLISECONDS);
+ }
+ catch (InterruptedException | ExecutionException | TimeoutException e)
+ {
+ throw new RuntimeException("Failed to initiate sealing", e);
+ }
+ }
+
+ /**
+ * Fence allocation on every participant of the obsoleted shard (ACTIVE -> SEALING) by marking the shard
+ * SEALING on each.
+ * TODO (expected): this assumes every participant has already enacted the topology change that obsoleted the shard (so
+ * it no longer allocates new ids on it). The ProgressBarrier gating bootstrap()/MID_JOIN only waits for
+ * EACH_QUORUM of the affected replicas, NOT all live replicas, so a live-but-lagging participant could still
+ * be on the old shard. Bringing each participant up to the seal epoch must be done per-replica (not
+ * per-shard) and will be added separately. TLDR - add a step to guaranteed up-to-speed rest of replicas epochs.
+ */
+ private static AsyncPromise<Void> initiate(ShardMetadata shard, @Nullable NodeId withoutNode)
+ {
+ return InitSealing.initiate(shard.keyspace, shard.sinceEpoch, shard.range, toEndpoints(shard.participants, withoutNode));
+ }
+
+ private static void drain(Set<ShardMetadata> shards, @Nullable NodeId withoutNode)
+ {
+ long deadlineNanos = nanoTime() + DatabaseDescriptor.getRpcTimeout(NANOSECONDS);
+ Set<ShardMetadata> pending = new HashSet<>(shards);
+ while (true)
+ {
+ // non-blocking poll: each participant reports its drain status immediately; retry until all drained
+ pending.removeIf(shard -> drain(shard, withoutNode));
+ if (pending.isEmpty())
+ return;
+ if (nanoTime() >= deadlineNanos)
+ throw new RuntimeException("Timed out draining shards; still pending: " + pending);
+ LockSupport.parkNanos(SECONDS.toNanos(1));
+ }
+ }
+
+ /**
+ * Poll every participant of the SEALING shard once for drain status; true iff all report that no mutation
+ * id allocated before the SEALING fence is still applying locally.
+ */
+ private static boolean drain(ShardMetadata shard, @Nullable NodeId withoutNode)
+ {
+ try
+ {
+ return Drain.poll(shard.keyspace, shard.sinceEpoch, shard.range, toEndpoints(shard.participants, withoutNode))
+ .get(DatabaseDescriptor.getRpcTimeout(MILLISECONDS), MILLISECONDS);
+ }
+ catch (InterruptedException | ExecutionException | TimeoutException e)
+ {
+ throw new RuntimeException(format("Failed to poll drain for shard %s", shard), e);
+ }
+ }
+
+ /**
+ * For each of the shards, capture the witnessed union then poll each participant until it has
+ * caught up and witnessed the entire offset union itself.
+ */
+ private static void reconcile(Set<ShardMetadata> shards, @Nullable NodeId withoutNode)
+ {
+ for (ShardMetadata shard : shards)
+ reconcile(shard, withoutNode);
+ }
+
+ private static void reconcile(ShardMetadata shard, @Nullable NodeId withoutNode)
Review Comment:
This seems like (roughly) a reimplimentation of `MutationTrackingSyncCoordinator` with the addition of the sinceEpoch filter. Could we just extend and use that instead of implementing here?
##########
src/java/org/apache/cassandra/replication/ActivationRequest.java:
##########
@@ -61,6 +61,7 @@ public class ActivationRequest
public final ShortMutationId transferId;
public final NodeId coordinatorId;
public final String keyspace;
+ public final long sinceEpoch;
Review Comment:
toString/equals/hashCode need to be updated to include this
##########
src/java/org/apache/cassandra/replication/MutationTrackingService.java:
##########
@@ -841,6 +844,44 @@ private KeyspaceShards getOrCreateShards(String keyspace)
return keyspaceShards.computeIfAbsent(keyspace, ignore -> KeyspaceShards.make(ksm, csm, this::nextLogId, this::onNewLog));
}
+ /**
+ * Find an existing shard matching the response's (epoch, range), or create a new one from the response metadata.
+ * TODO (expected): validate if I need to persist the newly created shard, if it was just created here
+ * TODO (expected): validate if this should be called with a shard lock everywhere it's called
+ * TODO (expected): log2Shard map updates even on failed CAS. Need to clean up there(?); Register callbacks?
+ */
+ @Nonnull
+ private Shard getOrCreateShard(
+ String keyspace, long sinceEpoch, Range<Token> range, Participants participants)
+ {
+ int localNodeId = ClusterMetadata.current().myNodeId().id();
+
+ if (!participants.contains(localNodeId))
+ throw new IllegalStateException("A shard looked up that this node doesn't participate in");
+
+ // unlikely, but possible to race here, hence the CAS loop
+ while (true)
+ {
+ KeyspaceShards current = getOrCreateShards(keyspace);
+ Shard shard = current.get(range, sinceEpoch);
+ if (shard != null)
+ return shard;
+ shard = new Shard(localNodeId, keyspace, sinceEpoch, range, participants, this::nextLogId, this::onNewLog);
Review Comment:
this is going to create a new log, so any cas retries will leave dangling logs
##########
src/java/org/apache/cassandra/replication/SealingCoordinator.java:
##########
@@ -0,0 +1,1401 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.cassandra.replication;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.TimeoutException;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.concurrent.locks.LockSupport;
+
+import javax.annotation.Nonnull;
+import javax.annotation.Nullable;
+
+import com.google.common.base.Preconditions;
+
+import org.apache.cassandra.config.DatabaseDescriptor;
+import org.apache.cassandra.db.ColumnFamilyStore;
+import org.apache.cassandra.db.Keyspace;
+import org.apache.cassandra.db.TypeSizes;
+import org.apache.cassandra.dht.AbstractBounds;
+import org.apache.cassandra.dht.IPartitioner;
+import org.apache.cassandra.dht.Range;
+import org.apache.cassandra.dht.Token;
+import org.apache.cassandra.exceptions.RequestFailure;
+import org.apache.cassandra.io.UnversionedSerializer;
+import org.apache.cassandra.io.util.DataInputPlus;
+import org.apache.cassandra.io.util.DataOutputPlus;
+import org.apache.cassandra.locator.EndpointsByReplica;
+import org.apache.cassandra.locator.EndpointsForRange;
+import org.apache.cassandra.locator.InetAddressAndPort;
+import org.apache.cassandra.locator.RangesAtEndpoint;
+import org.apache.cassandra.locator.Replica;
+import org.apache.cassandra.net.IVerbHandler;
+import org.apache.cassandra.net.Message;
+import org.apache.cassandra.net.MessagingService;
+import org.apache.cassandra.net.RequestCallback;
+import org.apache.cassandra.net.Verb;
+import org.apache.cassandra.schema.KeyspaceMetadata;
+import org.apache.cassandra.tcm.ClusterMetadata;
+import org.apache.cassandra.tcm.membership.NodeId;
+import org.apache.cassandra.tcm.ownership.DataPlacement;
+import org.apache.cassandra.tcm.ownership.MovementMap;
+import org.apache.cassandra.tcm.ownership.PlacementDeltas;
+import org.apache.cassandra.tcm.ownership.VersionedEndpoints;
+import org.apache.cassandra.tcm.sequences.LeaveStreams;
+import org.apache.cassandra.utils.CollectionSerializers;
+import org.apache.cassandra.utils.FBUtilities;
+import org.apache.cassandra.utils.concurrent.AsyncPromise;
+import org.apache.cassandra.utils.concurrent.Future;
+import org.apache.cassandra.utils.concurrent.FutureCombiner;
+
+import static java.lang.String.format;
+import static java.util.concurrent.TimeUnit.MILLISECONDS;
+import static java.util.concurrent.TimeUnit.NANOSECONDS;
+import static java.util.concurrent.TimeUnit.SECONDS;
+import static org.apache.cassandra.utils.Clock.Global.nanoTime;
+
+/*
+ * Sealing {@link Shard}s once a topology change obsoletes them.
+ *
+ * Happy Path Sequence:
+ * 1. Bring every participant up to (at least) this seal's epoch and fence allocation, so that everyone
+ * stops writing to the old shards; then wait until all in-progress mutations have been done.
+ * 2. Same for tracked transfers
+ * 3. Wait until all participating replicas have reconciled the union of their witnessed ids
+ * 4. Mark the shard as fully sealed
+ * 5. Stop including the shard in mutation summaries for read requests.
+ *
+ * Unhappy Path Sequence (not yet implemented):
+ * 1. Wait until everyone who's up stops writing to the old shards and all in-progress mutations have been done
+ * 2. Same for tracked transfers
+ * 3. Wait until all *live* participating replicas have reconciled the union of their witnessed ids
+ * 4. Mark the shard as partially sealed, label it with the set of participants that were able to seal
+ * 5. When a down node comes up, have it talk to other nodes and learn about sealing, and what mutations/logs
+ * they have exchanged. Diff with what we have - in system tables and in the journal (though system table
+ * plus local replay should give us an up-to-date metadata state for the shard)
+ */
+public final class SealingCoordinator
+{
+ /*
+ * BOOTSTRAP sequence illustrated. Stages in which write placements
+ * remain unchanged, and only read placements change, are skipped
+ * here as irrelevant to sharding / sealing.
+ *
+ * There is a tracked keyspace with RF=3.
+ * A node n35 with token=35 is added in the middle of the ring.
+ *
+ * 1. Initial state at e0:
+ *
+ * (10,20] | {n20,n30,n40} | e0 | ACTIVE
+ * (20,30] | {n30,n40,n50} | e0 | ACTIVE
+ * (30,40] | {n40,n50,n60} | e0 | ACTIVE
+ *
+ * 2. PREPARE_JOIN at e1:
+ *
+ * (10,20] | {n20,n30,n40} | e0 | ACTIVE
+ * (20,30] | {n30,n40,n50} | e0 | ACTIVE
+ * (30,40] | {n40,n50,n60} | e0 | ACTIVE
+ * + (30,35] | {n40,n50,n60} | e1 | ACTIVE
+ * + (35,40] | {n40,n50,n60} | e1 | ACTIVE
+ *
+ * 3. START_JOIN at e2:
+ *
+ * (10,20] | {n20,n30,n40} | e0 | ACTIVE
+ * + (10,20] | {n20,n30,n35,n40} | e2 | ACTIVE
+ * (20,30] | {n30,n40,n50} | e0 | ACTIVE
+ * + (20,30] | {n30,n35,n40,n50} | e2 | ACTIVE
+ * (30,40] | {n40,n50,n60} | e0 | ACTIVE
+ * (30,35] | {n40,n50,n60} | e1 | ACTIVE
+ * + (30,35] | {n35,n40,n50,n60} | e2 | ACTIVE
+ * (35,40] | {n40,n50,n60} | e1 | ACTIVE
+ *
+ * 4. MID_JOIN at e3 [*]:
+ *
+ * S (10,20] | {n20,n30,n40} | e0 | SEALED
+ * (10,20] | {n20,n30,n35,n40} | e2 | ACTIVE
+ * S (20,30] | {n30,n40,n50} | e0 | SEALED
+ * (20,30] | {n30,n35,n40,n50} | e2 | ACTIVE
+ * S (30,40] | {n40,n50,n60} | e0 | SEALED
+ * S (30,35] | {n40,n50,n60} | e1 | SEALED
+ * (30,35] | {n35,n40,n50,n60} | e2 | ACTIVE
+ * (35,40] | {n40,n50,n60} | e1 | ACTIVE
+ *
+ * 5. FINISH_JOIN at e4:
+ *
+ * (10,20] | {n20,n30,n40} | e0 | SEALED
+ * S (10,20] | {n20,n30,n35,n40} | e2 | SEALED
+ * + (10,20] | {n20,n30,n35} | e4 | ACTIVE
+ * (20,30] | {n30,n40,n50} | e0 | SEALED
+ * S (20,30] | {n30,n35,n40,n50} | e2 | SEALED
+ * + (20,30] | {n30,n35,n40} | e4 | ACTIVE
+ * (30,40] | {n40,n50,n60} | e0 | SEALED
+ * (30,35] | {n40,n50,n60} | e1 | SEALED
+ * S (30,35] | {n35,n40,n50,n60} | e2 | SEALED
+ * + (30,35] | {n35,n40,n50} | e4 | ACTIVE
+ * (35,40] | {n40,n50,n60} | e1 | ACTIVE
+ *
+ * [*] At this stage, after sealing the shards, we stream the sstables.
+ * All the offsets belonging to these sealed shards get stripped from
+ * stats component of streamed sstables. Otherwise, the new node
+ * won't be able to compact them or mark them as reconciled/repaired -
+ * it is rightfully unaware of the sealed shards.
+ */
+
+ /**
+ * Seal the shards obsoleted by bootstrap before the new node starts streaming.
+ */
+ public static void sealShardsAtMidJoin(ClusterMetadata metadata, MovementMap movements)
+ {
+ seal(discoverShardsAtMidJoinOrReplace(metadata, movements));
+ }
+
+ /**
+ * Seal the intermediate shards that were created during START_JOIN.
+ * TODO (required): this MUST eventually run while the affected ranges are still locked - between the
+ * removeFromWrites placement change and the range unlock. FinishJoin/FinishReplace currently apply
+ * removeFromWrites and the unlock atomically in a single transformation, so there is no committed
+ * dropped-but-locked epoch to hook. Until then, this runs just after FINISH_JOIN/FINISH_REPLACE,
+ * when the ranges are already unlocked. Left to implement for the unhappy path (AY).
+ */
+ public static void sealShardsAtFinishJoin(ClusterMetadata metadata, PlacementDeltas finishDelta)
+ {
+ seal(discoverShardsAtFinishJoinOrReplace(metadata, finishDelta, null));
+ }
+
+ /*
+ * REPLACE sequence illustrated. Stages in which write placements
+ * remain unchanged, and only read placements change, are skipped
+ * here as irrelevant to sharding / sealing.
+ *
+ * There is a tracked keyspace with RF=3.
+ * Node n40 is dead, and is being replaced with $n40.
+ *
+ * 1. Initial state at e0:
+ *
+ * (10,20] | {n20,n30,n40} | e0 | ACTIVE
+ * (20,30] | {n30,n40,n50} | e0 | ACTIVE
+ * (30,40] | {n40,n50,n60} | e0 | ACTIVE
+ *
+ * 2. START_REPLACE at e1:
+ *
+ * (10,20] | {n20,n30,n40} | e0 | ACTIVE
+ * + (10,20] | {n20,n30,n40,$n40}| e1 | ACTIVE
+ * (20,30] | {n30,n40,n50} | e0 | ACTIVE
+ * + (20,30] | {n30,n40,$n40,n50}| e1 | ACTIVE
+ * (30,40] | {n40,n50,n60} | e0 | ACTIVE
+ * + (30,40] | {n40,$n40,n50,n60}| e1 | ACTIVE
+ *
+ * 3. MID_REPLACE at e2 [*]:
+ *
+ * S (10,20] | {n20,n30,n40} | e0 | SEALED
+ * (10,20] | {n20,n30,n40,$n40}| e1 | ACTIVE
+ * S (20,30] | {n30,n40,n50} | e0 | SEALED
+ * (20,30] | {n30,n40,$n40,n50}| e1 | ACTIVE
+ * S (30,40] | {n40,n50,n60} | e0 | SEALED
+ * (30,40] | {n40,$n40,n50,n60}| e1 | ACTIVE
+ *
+ * 4. FINISH_REPLACE at e3:
+ *
+ * (10,20] | {n20,n30,n40} | e0 | SEALED
+ * S (10,20] | {n20,n30,n40,$n40}| e1 | SEALED
+ * + (10,20] | {n20,n30,$n40} | e3 | ACTIVE
+ * (20,30] | {n30,n40,n50} | e0 | SEALED
+ * S (20,30] | {n30,n40,$n40,n50}| e1 | SEALED
+ * + (20,30] | {n30,$n40,n50} | e3 | ACTIVE
+ * (30,40] | {n40,n50,n60} | e0 | SEALED
+ * S (30,40] | {n40,$n40,n50,n60}| e1 | SEALED
+ * + (30,40] | {$n40,n50,n60} | e3 | ACTIVE
+ *
+ * [*] At this stage, after sealing the shards, we stream the sstables.
+ * All the offsets belonging to these sealed shards get stripped from
+ * stats component of streamed sstables. Otherwise, the new node
+ * won't be able to compact them or mark them as reconciled/repaired -
+ * it is rightfully unaware of the sealed shards.
+ */
+
+ /**
+ * Seal the shards obsoleted by host replacement before the replacement node starts streaming.
+ * The {@code deadNode} node cannot participate in the sealing.
+ */
+ public static void sealShardsAtMidReplace(ClusterMetadata metadata, MovementMap movements, @Nonnull NodeId deadNode)
+ {
+ seal(discoverShardsAtMidJoinOrReplace(metadata, movements), deadNode);
+ }
+
+ public static void sealShardsAtFinishReplace(ClusterMetadata metadata, PlacementDeltas finishDelta, @Nonnull NodeId deadNode)
+ {
+ seal(discoverShardsAtFinishJoinOrReplace(metadata, finishDelta, deadNode), deadNode);
+ }
+
+ /**
+ * Discover all shards that need to be sealed at MID_JOIN or MID_REPLACE.
+ */
+ private static Set<ShardMetadata> discoverShardsAtMidJoinOrReplace(ClusterMetadata metadata, MovementMap movements)
+ {
+ List<Future<Set<ShardMetadata>>> futures = new ArrayList<>();
+
+ for (KeyspaceMetadata ksm : metadata.schema.getKeyspaces())
+ {
+ if (!ksm.params.replicationType.isTracked())
+ continue;
+
+ EndpointsByReplica acquired = movements.get(ksm.params.replication);
+ if (acquired.isEmpty()) // when the keyspace is not replicated to this DC
+ continue;
+
+ DataPlacement placement = metadata.placements.get(ksm.params.replication);
+
+ for (Map.Entry<Replica, EndpointsForRange> entry : acquired.entrySet())
+ {
+ Range<Token> range = entry.getKey().range();
+ Set<InetAddressAndPort> endpoints = entry.getValue().endpoints();
+ // cutoff: the epoch this node was added to the write placement for the range;
+ // shards over the range with sinceEpoch < cutoff are obsoleted by this bootstrap or replace.
+ long beforeEpoch = placement.writes.forRange(range).lastModified().getEpoch();
+ futures.add(FetchShards.fetch(ksm.name, beforeEpoch, range, endpoints));
+ }
+ }
+
+ Set<ShardMetadata> shards = new HashSet<>();
+ try
+ {
+ for (Set<ShardMetadata> fetched : FutureCombiner.allOf(futures).get(DatabaseDescriptor.getRpcTimeout(MILLISECONDS), MILLISECONDS))
+ shards.addAll(fetched);
+ }
+ catch (InterruptedException | ExecutionException | TimeoutException e)
+ {
+ throw new RuntimeException("Failed to fetch shards to seal", e);
+ }
+ return shards;
+ }
+
+ /**
+ * Discover all shards that need to be sealed at FINISH_JOIN or FINISH_REPLACE.
+ */
+ private static Set<ShardMetadata> discoverShardsAtFinishJoinOrReplace(ClusterMetadata metadata, PlacementDeltas finishDelta, @Nullable NodeId deadNode)
+ {
+ long finishEpoch = metadata.epoch.getEpoch();
+ Set<ShardMetadata> shards = new HashSet<>();
+ for (KeyspaceMetadata ksm : metadata.schema.getKeyspaces())
+ {
+ if (!ksm.params.replicationType.isTracked())
+ continue;
+
+ DataPlacement placement = metadata.placements.get(ksm.params.replication);
+ finishDelta.get(ksm.params.replication).writes.removals.flattenValues().forEach(removed -> {
+ Set<Integer> nodeIDs = new HashSet<>();
+ for (InetAddressAndPort ep : placement.writes.forRange(removed.range()).endpoints())
+ nodeIDs.add(metadata.directory.peerId(ep).id());
+ nodeIDs.add(deadNode != null ? deadNode.id() : metadata.directory.peerId(removed.endpoint()).id());
+ Participants participants = new Participants(nodeIDs);
+
+ MutationTrackingService.instance().forEachShardInKeyspace(ksm.name, shard -> {
+ if (shard.sinceEpoch < finishEpoch
+ && shard.range.equals(removed.range())
Review Comment:
could we use `range#intersects` here? I know that range should be fine, but intersects is more robust against future changes and/or edge cases
##########
src/java/org/apache/cassandra/tcm/sequences/BootstrapAndJoin.java:
##########
@@ -300,10 +302,15 @@ public SequenceState executeNext()
return continuable();
}
ClusterMetadataService.instance().ensureCMSPlacement(metadata);
+
+ if (MutationTrackingService.isEnabled())
+ SealingCoordinator.sealShardsAtFinishJoin(metadata, finishJoin.delta());
Review Comment:
Same concern about dangling SEALING shards as in BootstrapAndReplace
##########
src/java/org/apache/cassandra/tcm/sequences/UnbootstrapAndLeave.java:
##########
@@ -211,16 +213,23 @@ public SequenceState executeNext()
}
break;
case FINISH_LEAVE:
+ ClusterMetadata postFinish;
try
{
- ClusterMetadataService.instance().commit(finishLeave);
+ postFinish = ClusterMetadataService.instance().commit(finishLeave);
StorageService.instance.clearTransientMode();
}
catch (Throwable t)
{
JVMStabilityInspector.inspectThrowable(t);
return continuable();
}
+
+ // Seal the shards obsoleted by FINISH_LEAVE: the departed node's over-replicated generations created
+ // during START_LEAVE, plus the merge-half folded into the departed node's range by the merge.
+ if (MutationTrackingService.isEnabled())
+ if (streams.kind() == LeaveStreams.Kind.UNBOOTSTRAP || streams.kind() == LeaveStreams.Kind.REMOVENODE)
+ SealingCoordinator.sealShardsAtFinishLeave(postFinish, finishLeave.delta(), finishLeave.nodeId(), streams.kind());
Review Comment:
Same concern about dangling SEALING shards as in BootstrapAndReplace
##########
src/java/org/apache/cassandra/replication/MutationTrackingService.java:
##########
@@ -483,52 +398,105 @@ public void updateReplicatedOffsets(String keyspace, Range<Token> range, List<?
}
}
- public void recordFullyReconciledOffsets(ReconciledLogSnapshot reconciledSnapshot)
+ public boolean startWriting(Mutation mutation)
{
+ Preconditions.checkArgument(!mutation.id().isNone());
+ // resolve the shard first - might trigger a blocking network call
+ Shard shard = getOrCreateShardForMutation(mutation);
shardLock.readLock().lock();
try
{
- reconciledSnapshot.forEach((keyspace, keyspaceOffsets) -> {
- KeyspaceShards ksShards = getOrCreateShards(keyspace);
- if (ksShards != null)
- ksShards.recordFullyReconciledOffsets(keyspaceOffsets);
- });
+ return shard.startWriting(mutation);
}
finally
{
shardLock.readLock().unlock();
}
}
- public boolean startWriting(Mutation mutation)
+ public void finishWriting(Mutation mutation)
{
+ Preconditions.checkArgument(!mutation.id().isNone());
shardLock.readLock().lock();
try
{
- Preconditions.checkArgument(!mutation.id().isNone());
- return getOrCreateShards(mutation.getKeyspaceName()).startWriting(mutation);
+ getShard(mutation.id().asLogId()).finishWriting(mutation);
+ incomingMutations.invokeListeners(mutation.id());
}
finally
{
shardLock.readLock().unlock();
}
}
- public void finishWriting(Mutation mutation)
+ /**
+ * Must be called exactly once per {@code nextId()} invocation.
+ */
+ public void completeLocalWrite(MutationId id)
+ {
+ Preconditions.checkArgument(!id.isNone());
+ Shard shard = getShardNullable(id.asLogId());
+ if (null == shard)
+ throw new IllegalStateException(format("Shard for log %s was not found in log2ShardMap", id.asLogId()));
+ shard.completeLocalWrite();
+ }
+
+ /**
+ * Check the log-to-shard index first; if the log ID is locally unknown, query all peers for the shard metadata,
+ * then find or create the matching shard locally under the read lock.
+ * <p>
+ * The returned shard is safe to write to even if a topology change interleaves between this
+ * call and the subsequent write, because shard recreation carries existing shards forward rather
+ * than discarding them (see {@link KeyspaceShards#withNewShards}).
+ */
+ @Nonnull
+ private Shard getOrCreateShardForMutation(Mutation mutation)
{
+ CoordinatorLogId logId = mutation.id().asLogId();
+
+ Shard shard = getShardNullable(logId);
+ if (null != shard)
+ return shard;
+
+ ShardMetadata metadata = queryPeersForShardMetadata(logId, mutation.getKeyspaceName());
shardLock.readLock().lock();
try
{
- Preconditions.checkArgument(!mutation.id().isNone());
- getOrCreateShards(mutation.getKeyspaceName()).finishWriting(mutation);
- incomingMutations.invokeListeners(mutation.id());
+ return getOrCreateShard(metadata.keyspace, metadata.sinceEpoch, metadata.range, metadata.participants);
}
finally
{
shardLock.readLock().unlock();
}
}
+ /**
+ * When we see a new coordinator log id for the first time, we may not know what shard to place it into.
+ * Queries all peers for the shard metadata, finds or creates the matching shard locally, and returns it.
+ * Throws if resolution fails entirely.
+ */
+ private ShardMetadata queryPeersForShardMetadata(CoordinatorLogId logId, String keyspace)
Review Comment:
Also - this is kind of what we discussed earlier today, but I'm not saying _this_ is the way to do it, just that we should refine this a bit, and this is one way to do it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CEP-45: Topology Change Support for Mutation Tracking [cassandra]
Posted by "iamaleksey (via GitHub)" <gi...@apache.org>.
iamaleksey commented on code in PR #4910:
URL: https://github.com/apache/cassandra/pull/4910#discussion_r3796904812
##########
src/java/org/apache/cassandra/tcm/sequences/BootstrapAndReplace.java:
##########
@@ -290,6 +292,9 @@ public SequenceState executeNext()
}
ClusterMetadataService.instance().ensureCMSPlacement(metadata);
+ if (MutationTrackingService.isEnabled())
+ SealingCoordinator.sealShardsAtFinishReplace(metadata, finishReplace.delta(), startReplace.replaced());
Review Comment:
Not quite. Until `FINISH_REPLACE` is committed, there won't be a new epoch, and there won't be a new set of shards created by `MTS`. You'll seal the most recent shards without a successor present, and all new writes will be failing to allocate new mutation ids.
The correct fix is to introduce an additional TCM state right after we seal the old shards (but when the successor shards have already been instantiated). There is a related TODO for `sealShardsAtFinishJoin()`, that applies here as well:
```
/**
* Seal the intermediate shards that were created during START_JOIN.
* TODO (required): this MUST eventually run while the affected ranges are still locked - between the
* removeFromWrites placement change and the range unlock. FinishJoin/FinishReplace currently apply
* removeFromWrites and the unlock atomically in a single transformation, so there is no committed
* dropped-but-locked epoch to hook. Until then, this runs just after FINISH_JOIN/FINISH_REPLACE,
* when the ranges are already unlocked. Left to implement for the unhappy path (AY).
*/
```
Was just initially left for the unhappy path work (as failing at this point, right now, is going to make the cluster unhappy, of course).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CEP-45: Topology Change Support for Mutation Tracking [cassandra]
Posted by "bdeggleston (via GitHub)" <gi...@apache.org>.
bdeggleston commented on code in PR #4910:
URL: https://github.com/apache/cassandra/pull/4910#discussion_r3797877507
##########
src/java/org/apache/cassandra/tcm/sequences/BootstrapAndReplace.java:
##########
@@ -290,6 +292,9 @@ public SequenceState executeNext()
}
ClusterMetadataService.instance().ensureCMSPlacement(metadata);
+ if (MutationTrackingService.isEnabled())
+ SealingCoordinator.sealShardsAtFinishReplace(metadata, finishReplace.delta(), startReplace.replaced());
Review Comment:
'implicit sealing' is a partially finished thought. We create a new shard set as appropriate on a TCM change, I wonder if we could start the sealing process in the background instead of starting and waiting for it in the TCM sequence like this. We do need to wait for it to actually complete though before, so maybe it's not really practical and an additional TCM step is the best solution anyway
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CEP-45: Topology Change Support for Mutation Tracking [cassandra]
Posted by "iamaleksey (via GitHub)" <gi...@apache.org>.
iamaleksey commented on code in PR #4910:
URL: https://github.com/apache/cassandra/pull/4910#discussion_r3803506753
##########
src/java/org/apache/cassandra/tcm/sequences/BootstrapAndReplace.java:
##########
@@ -290,6 +292,9 @@ public SequenceState executeNext()
}
ClusterMetadataService.instance().ensureCMSPlacement(metadata);
+ if (MutationTrackingService.isEnabled())
+ SealingCoordinator.sealShardsAtFinishReplace(metadata, finishReplace.delta(), startReplace.replaced());
Review Comment:
I think from topology change perspective it's conceptually most similar to streaming - which also needs to complete before we transition to the next phase, and TCM sequences are where it's both initiated and blocked on.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CEP-45: Topology Change Support for Mutation Tracking [cassandra]
Posted by "bdeggleston (via GitHub)" <gi...@apache.org>.
bdeggleston commented on code in PR #4910:
URL: https://github.com/apache/cassandra/pull/4910#discussion_r3797106588
##########
src/java/org/apache/cassandra/tcm/sequences/BootstrapAndReplace.java:
##########
@@ -290,6 +292,9 @@ public SequenceState executeNext()
}
ClusterMetadataService.instance().ensureCMSPlacement(metadata);
+ if (MutationTrackingService.isEnabled())
+ SealingCoordinator.sealShardsAtFinishReplace(metadata, finishReplace.delta(), startReplace.replaced());
Review Comment:
What if we made CMS changes like `FINISH_REPLACE` implicitly sealing? Here we're worried about cms and sealing states getting out of sync. That would at least solve half the problem, creating new shards at the appropriate epoch. If you left for this for the unhappy path though, maybe I should just take it as part of partial sealing. WDYT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CEP-45: Topology Change Support for Mutation Tracking [cassandra]
Posted by "iamaleksey (via GitHub)" <gi...@apache.org>.
iamaleksey commented on code in PR #4910:
URL: https://github.com/apache/cassandra/pull/4910#discussion_r4072694621
##########
src/java/org/apache/cassandra/tcm/sequences/BootstrapAndReplace.java:
##########
@@ -290,6 +292,9 @@ public SequenceState executeNext()
}
ClusterMetadataService.instance().ensureCMSPlacement(metadata);
+ if (MutationTrackingService.isEnabled())
+ SealingCoordinator.sealShardsAtFinishReplace(metadata, finishReplace.delta(), startReplace.replaced());
Review Comment:
Solved by adding the additional TCM step to the sequences.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org