You are viewing a plain text version of this content. The canonical link for it is here.
Posted to pr@cassandra.apache.org by "ifesdjeen (via GitHub)" <gi...@apache.org> on 2025/01/28 18:11:58 UTC
[PR] CASSANDRA-20245: Fix problems and race conditions with topology fetching [cassandra]
ifesdjeen opened a new pull request, #3842:
URL: https://github.com/apache/cassandra/pull/3842
- preclude TCM of being behind Accord if newer epoch is reported via withEpoch/fetchTopologyInternal
- improve topology discovery during first boot and replay
- fix races between config service TCM listener reporting topologies, and fetched topologies during
Patch by Alex Petrov, reviewed by Ariel Weisberg for CASSANDRA-20245
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-20245: Fix problems and race conditions with topology fetching [cassandra]
Posted by "dcapwell (via GitHub)" <gi...@apache.org>.
dcapwell commented on code in PR #3842:
URL: https://github.com/apache/cassandra/pull/3842#discussion_r1934243353
##########
src/java/org/apache/cassandra/service/accord/AccordService.java:
##########
@@ -410,48 +413,67 @@ public synchronized void startup()
}
/**
- * Queries peers to discover min epoch
+ * Queries peers to discover min epoch, and then fetches all topologies between min and current epochs
*/
- private long fetchMinEpoch()
+ private List<Topology> fetchTopologies(long minEpoch, ClusterMetadata metadata) throws ExecutionException, InterruptedException
{
- ClusterMetadata metadata = ClusterMetadata.current();
- Map<InetAddressAndPort, Set<TokenRange>> peers = new HashMap<>();
- for (KeyspaceMetadata keyspace : metadata.schema.getKeyspaces())
+ if (configService.maxEpoch() >= metadata.epoch.getEpoch())
{
- List<TableMetadata> tables = keyspace.tables.stream().filter(TableMetadata::requiresAccordSupport).collect(Collectors.toList());
- if (tables.isEmpty())
- continue;
- DataPlacement current = metadata.placements.get(keyspace.params.replication);
- DataPlacement settled = metadata.writePlacementAllSettled(keyspace);
- Sets.SetView<InetAddressAndPort> alive = Sets.intersection(settled.writes.byEndpoint().keySet(), current.writes.byEndpoint().keySet());
- InetAddressAndPort self = FBUtilities.getBroadcastAddressAndPort();
- settled.writes.forEach((range, group) -> {
- if (group.endpoints().contains(self))
- {
- for (InetAddressAndPort peer : group.endpoints())
- {
- if (peer.equals(self) || !alive.contains(peer)) continue;
- for (TableMetadata table : tables)
- peers.computeIfAbsent(peer, i -> new HashSet<>()).add(AccordTopology.fullRange(table.id));
- }
- }
- });
+ logger.info("Accord epoch {} matches TCM. All topologies are known locally", metadata.epoch);
+ return Collections.singletonList(AccordTopology.createAccordTopology(metadata));
}
+
+ Set<InetAddressAndPort> peers = new HashSet<>();
+ peers.addAll(metadata.directory.allAddresses());
+ peers.remove(FBUtilities.getBroadcastAddressAndPort());
+
+ // No peers: single node cluster or first node to boot
if (peers.isEmpty())
- return -1;
+ return Collections.singletonList(AccordTopology.createAccordTopology(metadata));;
+
+ // Bootstrap, fetch min epoch
+ if (minEpoch == 0)
+ {
+ long fetched = findMinEpoch(SharedContext.Global.instance, peers);
+ // No other node has advanced epoch just yet
+ if (fetched == 0)
+ return Collections.singletonList(AccordTopology.createAccordTopology(metadata));
+
+ minEpoch = fetched;
+ }
+
+ long maxEpoch = metadata.epoch.getEpoch();
+
+ // If we are behind minEpoch, catch up to at least minEpoch
+ if (metadata.epoch.getEpoch() < minEpoch)
+ {
+ minEpoch = metadata.epoch.getEpoch();
+ maxEpoch = minEpoch;
+ }
+
+ List<Future<Topology>> futures = new ArrayList<>();
+ logger.info("Discovered min epoch of {}. Proceeding to fetch epochs up to {}.", minEpoch, maxEpoch);
+
+ for (long epoch = minEpoch; epoch <= maxEpoch; epoch++)
+ futures.add(FetchTopology.fetch(SharedContext.Global.instance, peers, epoch));
Review Comment:
is there a reason to not make this a batch? if you know `N` you know `N - 1`, so rather than do many concurrent fetches we could just ask for the full batch in 1 request?
thinking about the current bootstrap case, you can startup with 50 epochs just in benchmarks, so we have 50 concurrent fetches, and they will go to the same node at the same time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-20245: Fix problems and race conditions with topology fetching [cassandra]
Posted by "ifesdjeen (via GitHub)" <gi...@apache.org>.
ifesdjeen commented on code in PR #3842:
URL: https://github.com/apache/cassandra/pull/3842#discussion_r1934403392
##########
src/java/org/apache/cassandra/service/accord/AccordService.java:
##########
@@ -410,48 +413,67 @@ public synchronized void startup()
}
/**
- * Queries peers to discover min epoch
+ * Queries peers to discover min epoch, and then fetches all topologies between min and current epochs
*/
- private long fetchMinEpoch()
+ private List<Topology> fetchTopologies(long minEpoch, ClusterMetadata metadata) throws ExecutionException, InterruptedException
{
- ClusterMetadata metadata = ClusterMetadata.current();
- Map<InetAddressAndPort, Set<TokenRange>> peers = new HashMap<>();
- for (KeyspaceMetadata keyspace : metadata.schema.getKeyspaces())
+ if (configService.maxEpoch() >= metadata.epoch.getEpoch())
{
- List<TableMetadata> tables = keyspace.tables.stream().filter(TableMetadata::requiresAccordSupport).collect(Collectors.toList());
- if (tables.isEmpty())
- continue;
- DataPlacement current = metadata.placements.get(keyspace.params.replication);
- DataPlacement settled = metadata.writePlacementAllSettled(keyspace);
- Sets.SetView<InetAddressAndPort> alive = Sets.intersection(settled.writes.byEndpoint().keySet(), current.writes.byEndpoint().keySet());
- InetAddressAndPort self = FBUtilities.getBroadcastAddressAndPort();
- settled.writes.forEach((range, group) -> {
- if (group.endpoints().contains(self))
- {
- for (InetAddressAndPort peer : group.endpoints())
- {
- if (peer.equals(self) || !alive.contains(peer)) continue;
- for (TableMetadata table : tables)
- peers.computeIfAbsent(peer, i -> new HashSet<>()).add(AccordTopology.fullRange(table.id));
- }
- }
- });
+ logger.info("Accord epoch {} matches TCM. All topologies are known locally", metadata.epoch);
+ return Collections.singletonList(AccordTopology.createAccordTopology(metadata));
}
+
+ Set<InetAddressAndPort> peers = new HashSet<>();
+ peers.addAll(metadata.directory.allAddresses());
+ peers.remove(FBUtilities.getBroadcastAddressAndPort());
+
+ // No peers: single node cluster or first node to boot
if (peers.isEmpty())
- return -1;
+ return Collections.singletonList(AccordTopology.createAccordTopology(metadata));;
+
+ // Bootstrap, fetch min epoch
+ if (minEpoch == 0)
+ {
+ long fetched = findMinEpoch(SharedContext.Global.instance, peers);
+ // No other node has advanced epoch just yet
+ if (fetched == 0)
+ return Collections.singletonList(AccordTopology.createAccordTopology(metadata));
+
+ minEpoch = fetched;
+ }
+
+ long maxEpoch = metadata.epoch.getEpoch();
+
+ // If we are behind minEpoch, catch up to at least minEpoch
+ if (metadata.epoch.getEpoch() < minEpoch)
+ {
+ minEpoch = metadata.epoch.getEpoch();
+ maxEpoch = minEpoch;
+ }
+
+ List<Future<Topology>> futures = new ArrayList<>();
+ logger.info("Discovered min epoch of {}. Proceeding to fetch epochs up to {}.", minEpoch, maxEpoch);
+
+ for (long epoch = minEpoch; epoch <= maxEpoch; epoch++)
+ futures.add(FetchTopology.fetch(SharedContext.Global.instance, peers, epoch));
Review Comment:
Talked in chat, and agreed with you and Ariel to make a follow-up patch shortly to switch to batching here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-20245: Fix problems and race conditions with topology fetching [cassandra]
Posted by "dcapwell (via GitHub)" <gi...@apache.org>.
dcapwell commented on code in PR #3842:
URL: https://github.com/apache/cassandra/pull/3842#discussion_r1937731916
##########
src/java/org/apache/cassandra/config/AccordSpec.java:
##########
@@ -204,6 +205,14 @@ public MinEpochRetrySpec()
}
}
+ public static class FetchRetrySpec extends RetrySpec
Review Comment:
`RetrySpec` has the following constructor
```
public RetrySpec(MaxAttempt maxAttempts, LongMillisecondsBound baseSleepTime, LongMillisecondsBound maxSleepTime)
```
Nothing stops us from adding
```
public RetrySpec(MaxAttempt maxAttempts)
```
or a `create` method... I think i did things this way mostly because of repair but nothing is wrong with `RetrySpec fetchRetry = RetrySpec.create(100);` IMO
I am 100% fine with this class (assuming the ref test is fixed), I am also fine with creating new methods in `RetrySpec` to get the same behavior
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-20245: Fix problems and race conditions with topology fetching [cassandra]
Posted by "dcapwell (via GitHub)" <gi...@apache.org>.
dcapwell commented on code in PR #3842:
URL: https://github.com/apache/cassandra/pull/3842#discussion_r1934238285
##########
src/java/org/apache/cassandra/service/accord/AccordService.java:
##########
@@ -410,48 +413,67 @@ public synchronized void startup()
}
/**
- * Queries peers to discover min epoch
+ * Queries peers to discover min epoch, and then fetches all topologies between min and current epochs
*/
- private long fetchMinEpoch()
+ private List<Topology> fetchTopologies(long minEpoch, ClusterMetadata metadata) throws ExecutionException, InterruptedException
{
- ClusterMetadata metadata = ClusterMetadata.current();
- Map<InetAddressAndPort, Set<TokenRange>> peers = new HashMap<>();
- for (KeyspaceMetadata keyspace : metadata.schema.getKeyspaces())
+ if (configService.maxEpoch() >= metadata.epoch.getEpoch())
{
- List<TableMetadata> tables = keyspace.tables.stream().filter(TableMetadata::requiresAccordSupport).collect(Collectors.toList());
- if (tables.isEmpty())
- continue;
- DataPlacement current = metadata.placements.get(keyspace.params.replication);
- DataPlacement settled = metadata.writePlacementAllSettled(keyspace);
- Sets.SetView<InetAddressAndPort> alive = Sets.intersection(settled.writes.byEndpoint().keySet(), current.writes.byEndpoint().keySet());
- InetAddressAndPort self = FBUtilities.getBroadcastAddressAndPort();
- settled.writes.forEach((range, group) -> {
- if (group.endpoints().contains(self))
- {
- for (InetAddressAndPort peer : group.endpoints())
- {
- if (peer.equals(self) || !alive.contains(peer)) continue;
- for (TableMetadata table : tables)
- peers.computeIfAbsent(peer, i -> new HashSet<>()).add(AccordTopology.fullRange(table.id));
- }
- }
- });
+ logger.info("Accord epoch {} matches TCM. All topologies are known locally", metadata.epoch);
+ return Collections.singletonList(AccordTopology.createAccordTopology(metadata));
}
+
+ Set<InetAddressAndPort> peers = new HashSet<>();
+ peers.addAll(metadata.directory.allAddresses());
+ peers.remove(FBUtilities.getBroadcastAddressAndPort());
+
+ // No peers: single node cluster or first node to boot
if (peers.isEmpty())
- return -1;
+ return Collections.singletonList(AccordTopology.createAccordTopology(metadata));;
Review Comment:
```suggestion
return Collections.singletonList(AccordTopology.createAccordTopology(metadata));
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-20245: Fix problems and race conditions with topology fetching [cassandra]
Posted by "dcapwell (via GitHub)" <gi...@apache.org>.
dcapwell commented on code in PR #3842:
URL: https://github.com/apache/cassandra/pull/3842#discussion_r1934267347
##########
test/distributed/org/apache/cassandra/distributed/test/accord/AccordDropKeyspaceTest.java:
##########
@@ -35,8 +34,7 @@ public void dropKeyspace() throws IOException
int steps = 5;
try (Cluster cluster = Cluster.build(3)
.withoutVNodes()
- .withConfig(c -> c.with(Feature.values())
Review Comment:
I am ok with `GOSSIP, NETWORK, NATIVE_PROTOCOL` or `values()`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-20245: Fix problems and race conditions with topology fetching [cassandra]
Posted by "dcapwell (via GitHub)" <gi...@apache.org>.
dcapwell commented on code in PR #3842:
URL: https://github.com/apache/cassandra/pull/3842#discussion_r1936394993
##########
src/java/org/apache/cassandra/service/accord/FetchTopology.java:
##########
@@ -122,19 +114,26 @@ public Response(long epoch, Topology topology)
public static final IVerbHandler<FetchTopology> handler = message -> {
long epoch = message.payload.epoch;
Topology topology = AccordService.instance().topology().maybeGlobalForEpoch(epoch);
- if (topology == null)
- MessagingService.instance().respond(Response.UNKNOWN, message);
- else
+ if (topology != null)
MessagingService.instance().respond(new Response(epoch, topology), message);
+ else
+ throw new IllegalStateException("Unknown topology: " + epoch);
Review Comment:
said this in another thread, but given the refactor its not showing up in the file, so reposing.
I do not think this is a good idea. `UNKNOWN` failure is truly unknown... it could be a NPE, it could be we ran *before* we were ready... it could be anything... its unknown... I do not think its a good idea to conflate that with "I don't know this epoch", because the handling is different...
if `UNKNOWN` exception is returned, retrying on the same node makes sense, it could be an ephemeral issue.
If the node doesn't know about the epoch, calling them again isn't the best idea... we should try another node, and if no node knows the epoch we are in a bad state...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-20245: Fix problems and race conditions with topology fetching [cassandra]
Posted by "dcapwell (via GitHub)" <gi...@apache.org>.
dcapwell commented on code in PR #3842:
URL: https://github.com/apache/cassandra/pull/3842#discussion_r1934269317
##########
src/java/org/apache/cassandra/service/accord/FetchTopology.java:
##########
@@ -123,18 +116,20 @@ public Response(long epoch, Topology topology)
long epoch = message.payload.epoch;
Topology topology = AccordService.instance().topology().maybeGlobalForEpoch(epoch);
if (topology == null)
- MessagingService.instance().respond(Response.UNKNOWN, message);
+ MessagingService.instance().respond(Response.unkonwn(epoch), message);
Review Comment:
i couldn't find any tests for this logic. Since this is a verb handle we should get back an `UNKNOWN` exception; that could be a NPE or an unknown epoch... its truly unknown.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-20245: Fix problems and race conditions with topology fetching [cassandra]
Posted by "ifesdjeen (via GitHub)" <gi...@apache.org>.
ifesdjeen commented on code in PR #3842:
URL: https://github.com/apache/cassandra/pull/3842#discussion_r1934317012
##########
src/java/org/apache/cassandra/service/accord/FetchTopology.java:
##########
@@ -37,6 +41,7 @@
public class FetchTopology
{
+ private static final Logger log = LoggerFactory.getLogger(FetchTopology.class);
Review Comment:
Sure, I wouldn't mind. However for this particular instance I removed it as it's unused.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-20245: Fix problems and race conditions with topology fetching [cassandra]
Posted by "dcapwell (via GitHub)" <gi...@apache.org>.
dcapwell commented on code in PR #3842:
URL: https://github.com/apache/cassandra/pull/3842#discussion_r1936383814
##########
src/java/org/apache/cassandra/config/AccordSpec.java:
##########
@@ -194,15 +194,17 @@ public enum TransactionalRangeMigration
public boolean ephemeralReadEnabled = true;
public boolean state_cache_listener_jfr_enabled = true;
public final JournalSpec journal = new JournalSpec();
- public final MinEpochRetrySpec minEpochSyncRetry = new MinEpochRetrySpec();
-
- public static class MinEpochRetrySpec extends RetrySpec
- {
- public MinEpochRetrySpec()
+ public final RetrySpec minEpochSyncRetry = new RetrySpec() {
Review Comment:
`"org.apache.cassandra.config.Config$1",` stuff like this is likely to break `DatabaseDescriptorRefTest`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-20245: Fix problems and race conditions with topology fetching [cassandra]
Posted by "ifesdjeen (via GitHub)" <gi...@apache.org>.
ifesdjeen commented on code in PR #3842:
URL: https://github.com/apache/cassandra/pull/3842#discussion_r1934327383
##########
src/java/org/apache/cassandra/service/accord/AccordService.java:
##########
@@ -377,24 +371,33 @@ public synchronized void startup()
node.commandStores().restoreShardStateUnsafe(topology -> configService.reportTopology(topology, true, true));
configService.start();
- long minEpoch = fetchMinEpoch();
- if (minEpoch >= 0)
+ try
{
- for (long epoch = minEpoch; epoch <= metadata.epoch.getEpoch(); epoch++)
- node.configService().fetchTopologyForEpoch(epoch);
+ // Fetch topologies up to current
+ List<Topology> topologies = fetchTopologies(0, metadata);
+ for (Topology topology : topologies)
+ configService.reportTopology(topology);
- try
- {
- epochReady(metadata.epoch).get(DatabaseDescriptor.getTransactionTimeout(MILLISECONDS), MILLISECONDS);
- }
- catch (InterruptedException e)
- {
- throw new UncheckedInterruptedException(e);
- }
- catch (ExecutionException | TimeoutException e)
+ ClusterMetadataService.instance().log().addListener(configService.listener);
+ ClusterMetadata next = ClusterMetadata.current();
+
+ // if metadata was updated before we were able to add a listener, fetch remaining topologies
+ if (metadata != next)
Review Comment:
This is a very common pattern, both with TCM and outside it, especially in Accord, so I assumed this is the way we commonly do it.
But I don't mind comparing epochs either, switched to `if (!metadata.epoch.is(next.epoch))`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-20245: Fix problems and race conditions with topology fetching [cassandra]
Posted by "dcapwell (via GitHub)" <gi...@apache.org>.
dcapwell commented on code in PR #3842:
URL: https://github.com/apache/cassandra/pull/3842#discussion_r1934232268
##########
src/java/org/apache/cassandra/service/accord/AccordConfigurationService.java:
##########
@@ -420,14 +424,17 @@ void maybeReportMetadata(ClusterMetadata metadata)
@Override
protected void fetchTopologyInternal(long epoch)
{
+ if (ClusterMetadata.current().epoch.getEpoch() < epoch)
Review Comment:
I think you need to rebase, Ariel added this and moved this function to run on `Stage.ACCORD_MIGRATION`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-20245: Fix problems and race conditions with topology fetching [cassandra]
Posted by "ifesdjeen (via GitHub)" <gi...@apache.org>.
ifesdjeen commented on code in PR #3842:
URL: https://github.com/apache/cassandra/pull/3842#discussion_r1937018098
##########
src/java/org/apache/cassandra/service/accord/FetchTopology.java:
##########
@@ -122,19 +114,26 @@ public Response(long epoch, Topology topology)
public static final IVerbHandler<FetchTopology> handler = message -> {
long epoch = message.payload.epoch;
Topology topology = AccordService.instance().topology().maybeGlobalForEpoch(epoch);
- if (topology == null)
- MessagingService.instance().respond(Response.UNKNOWN, message);
- else
+ if (topology != null)
MessagingService.instance().respond(new Response(epoch, topology), message);
+ else
+ throw new IllegalStateException("Unknown topology: " + epoch);
Review Comment:
Switched to specialized UNKNOWN_TOPLOGY failure response.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-20245: Fix problems and race conditions with topology fetching [cassandra]
Posted by "aweisberg (via GitHub)" <gi...@apache.org>.
aweisberg commented on code in PR #3842:
URL: https://github.com/apache/cassandra/pull/3842#discussion_r1932793214
##########
src/java/org/apache/cassandra/service/accord/AccordConfigurationService.java:
##########
@@ -407,7 +407,11 @@ void maybeReportMetadata(ClusterMetadata metadata)
long epoch = metadata.epoch.getEpoch();
synchronized (epochs)
{
- if (epochs.maxEpoch() == 0)
+ long maxEpoch = epochs.maxEpoch();
+ if (maxEpoch >= epoch)
Review Comment:
Does Accord knowing about the epoch guarantee that TCM has already loaded it? We don't want to skip the TCM loading step by not indirectly calling `fetchTopologyInternal` or something to ensure TCM loaded it.
##########
src/java/org/apache/cassandra/service/accord/AccordService.java:
##########
@@ -410,48 +413,67 @@ public synchronized void startup()
}
/**
- * Queries peers to discover min epoch
+ * Queries peers to discover min epoch, and then fetches all topologies between min and current epochs
*/
- private long fetchMinEpoch()
+ private List<Topology> fetchTopologies(long minEpoch, ClusterMetadata metadata) throws ExecutionException, InterruptedException
{
- ClusterMetadata metadata = ClusterMetadata.current();
- Map<InetAddressAndPort, Set<TokenRange>> peers = new HashMap<>();
- for (KeyspaceMetadata keyspace : metadata.schema.getKeyspaces())
+ if (configService.maxEpoch() >= metadata.epoch.getEpoch())
{
- List<TableMetadata> tables = keyspace.tables.stream().filter(TableMetadata::requiresAccordSupport).collect(Collectors.toList());
- if (tables.isEmpty())
- continue;
- DataPlacement current = metadata.placements.get(keyspace.params.replication);
- DataPlacement settled = metadata.writePlacementAllSettled(keyspace);
- Sets.SetView<InetAddressAndPort> alive = Sets.intersection(settled.writes.byEndpoint().keySet(), current.writes.byEndpoint().keySet());
- InetAddressAndPort self = FBUtilities.getBroadcastAddressAndPort();
- settled.writes.forEach((range, group) -> {
- if (group.endpoints().contains(self))
- {
- for (InetAddressAndPort peer : group.endpoints())
- {
- if (peer.equals(self) || !alive.contains(peer)) continue;
- for (TableMetadata table : tables)
- peers.computeIfAbsent(peer, i -> new HashSet<>()).add(AccordTopology.fullRange(table.id));
- }
- }
- });
+ logger.info("Accord epoch {} matches TCM. All topologies are known locally", metadata.epoch);
+ return Collections.singletonList(AccordTopology.createAccordTopology(metadata));
}
+
+ Set<InetAddressAndPort> peers = new HashSet<>();
+ peers.addAll(metadata.directory.allAddresses());
+ peers.remove(FBUtilities.getBroadcastAddressAndPort());
+
+ // No peers: single node cluster or first node to boot
if (peers.isEmpty())
- return -1;
+ return Collections.singletonList(AccordTopology.createAccordTopology(metadata));;
+
+ // Bootstrap, fetch min epoch
+ if (minEpoch == 0)
+ {
+ long fetched = findMinEpoch(SharedContext.Global.instance, peers);
+ // No other node has advanced epoch just yet
+ if (fetched == 0)
+ return Collections.singletonList(AccordTopology.createAccordTopology(metadata));
+
+ minEpoch = fetched;
+ }
+
+ long maxEpoch = metadata.epoch.getEpoch();
+
+ // If we are behind minEpoch, catch up to at least minEpoch
+ if (metadata.epoch.getEpoch() < minEpoch)
+ {
+ minEpoch = metadata.epoch.getEpoch();
+ maxEpoch = minEpoch;
+ }
+
+ List<Future<Topology>> futures = new ArrayList<>();
+ logger.info("Discovered min epoch of {}. Proceeding to fetch epochs up to {}.", minEpoch, maxEpoch);
+
+ for (long epoch = minEpoch; epoch <= maxEpoch; epoch++)
+ futures.add(FetchTopology.fetch(SharedContext.Global.instance, peers, epoch));
+
+ FBUtilities.waitOnFutures(futures);
+ List<Topology> topologies = new ArrayList<>(futures.size());
+ for (Future<Topology> future : futures)
+ topologies.add(future.get());
- Long minEpoch = findMinEpoch(SharedContext.Global.instance, peers);
- if (minEpoch == null)
- return -1;
- return minEpoch;
+ return topologies;
}
@VisibleForTesting
- static Long findMinEpoch(SharedContext context, Map<InetAddressAndPort, Set<TokenRange>> peers)
+ static long findMinEpoch(SharedContext context, Set<InetAddressAndPort> peers)
{
try
{
- return FetchMinEpoch.fetch(context, peers).get();
+ Long result = FetchMinEpoch.fetch(context, peers).get();
+ if (result == null)
+ return 0L;
+ return result.longValue();
Review Comment:
Won't this unbox automatically?
##########
src/java/org/apache/cassandra/service/accord/FetchTopology.java:
##########
@@ -123,18 +116,20 @@ public Response(long epoch, Topology topology)
long epoch = message.payload.epoch;
Topology topology = AccordService.instance().topology().maybeGlobalForEpoch(epoch);
if (topology == null)
- MessagingService.instance().respond(Response.UNKNOWN, message);
+ MessagingService.instance().respond(Response.unkonwn(epoch), message);
else
MessagingService.instance().respond(new Response(epoch, topology), message);
};
+ private static final Logger logger = LoggerFactory.getLogger(FetchTopology.class);
Review Comment:
`logger` goes at the top? Also unused.
##########
src/java/org/apache/cassandra/service/accord/FetchMinEpoch.java:
##########
@@ -145,21 +108,22 @@ public static Future<Long> fetch(SharedContext context, Map<InetAddressAndPort,
}
@VisibleForTesting
- static Future<Long> fetch(SharedContext context, InetAddressAndPort to, Set<TokenRange> value)
+ static Future<Long> fetch(SharedContext context, InetAddressAndPort to)
{
- FetchMinEpoch req = new FetchMinEpoch(value);
- return context.messaging().<FetchMinEpoch, FetchMinEpoch.Response>sendWithRetries(Backoff.NO_OP.INSTANCE,
- MessageDelivery.ImmediateRetryScheduler.instance,
- Verb.ACCORD_FETCH_MIN_EPOCH_REQ, req,
+ Backoff backoff = Backoff.fromConfig(context, DatabaseDescriptor.getAccord().minEpochSyncRetry);
+ return context.messaging().<FetchMinEpoch, FetchMinEpoch.Response>sendWithRetries(backoff,
+ context.optionalTasks()::schedule,
+ Verb.ACCORD_FETCH_MIN_EPOCH_REQ,
+ FetchMinEpoch.instance,
Iterators.cycle(to),
- RetryPredicate.times(DatabaseDescriptor.getAccord().minEpochSyncRetry.maxAttempts.value),
+ RetryPredicate.ALWAYS_RETRY,
Review Comment:
Does this mean that fetching the min epoch requires all nodes to be up to complete? Just looking at how this is accumulated by the caller of `fetch` which combines all the futures and can't complete until every future completes which means any down node would stop this from working?
##########
src/java/org/apache/cassandra/service/accord/AccordService.java:
##########
@@ -377,24 +371,33 @@ public synchronized void startup()
node.commandStores().restoreShardStateUnsafe(topology -> configService.reportTopology(topology, true, true));
configService.start();
- long minEpoch = fetchMinEpoch();
- if (minEpoch >= 0)
+ try
{
- for (long epoch = minEpoch; epoch <= metadata.epoch.getEpoch(); epoch++)
- node.configService().fetchTopologyForEpoch(epoch);
+ // Fetch topologies up to current
+ List<Topology> topologies = fetchTopologies(0, metadata);
Review Comment:
Maybe make 0 a constant indicating that it is actually supposed to find the `minEpoch` to fetch?
##########
src/java/org/apache/cassandra/service/accord/AccordConfigurationService.java:
##########
@@ -420,14 +424,17 @@ void maybeReportMetadata(ClusterMetadata metadata)
@Override
protected void fetchTopologyInternal(long epoch)
{
+ if (ClusterMetadata.current().epoch.getEpoch() < epoch)
+ ClusterMetadataService.instance().fetchLogFromCMS(Epoch.create(epoch));
+
try
{
Set<InetAddressAndPort> peers = new HashSet<>(ClusterMetadata.current().directory.allJoinedEndpoints());
peers.remove(FBUtilities.getBroadcastAddressAndPort());
if (peers.isEmpty())
return;
Topology topology;
- while ((topology =FetchTopology.fetch(SharedContext.Global.instance, peers, epoch).get()) == null) {}
+ while ((topology = FetchTopology.fetch(SharedContext.Global.instance, peers, epoch).get()) == null) {}
Review Comment:
Why do we need to run `FetchTopology` here if we would indirectly get the topology from ` ClusterMetadataService.instance().fetchLogFromCMS(Epoch.create(epoch));`
##########
src/java/org/apache/cassandra/service/accord/AccordService.java:
##########
@@ -410,48 +413,67 @@ public synchronized void startup()
}
/**
- * Queries peers to discover min epoch
+ * Queries peers to discover min epoch, and then fetches all topologies between min and current epochs
*/
- private long fetchMinEpoch()
+ private List<Topology> fetchTopologies(long minEpoch, ClusterMetadata metadata) throws ExecutionException, InterruptedException
{
- ClusterMetadata metadata = ClusterMetadata.current();
- Map<InetAddressAndPort, Set<TokenRange>> peers = new HashMap<>();
- for (KeyspaceMetadata keyspace : metadata.schema.getKeyspaces())
+ if (configService.maxEpoch() >= metadata.epoch.getEpoch())
{
- List<TableMetadata> tables = keyspace.tables.stream().filter(TableMetadata::requiresAccordSupport).collect(Collectors.toList());
- if (tables.isEmpty())
- continue;
- DataPlacement current = metadata.placements.get(keyspace.params.replication);
- DataPlacement settled = metadata.writePlacementAllSettled(keyspace);
- Sets.SetView<InetAddressAndPort> alive = Sets.intersection(settled.writes.byEndpoint().keySet(), current.writes.byEndpoint().keySet());
- InetAddressAndPort self = FBUtilities.getBroadcastAddressAndPort();
- settled.writes.forEach((range, group) -> {
- if (group.endpoints().contains(self))
- {
- for (InetAddressAndPort peer : group.endpoints())
- {
- if (peer.equals(self) || !alive.contains(peer)) continue;
- for (TableMetadata table : tables)
- peers.computeIfAbsent(peer, i -> new HashSet<>()).add(AccordTopology.fullRange(table.id));
- }
- }
- });
+ logger.info("Accord epoch {} matches TCM. All topologies are known locally", metadata.epoch);
Review Comment:
I don't know why this is true so I am just going to have to take it on faith.
##########
src/java/org/apache/cassandra/service/accord/AccordConfigurationService.java:
##########
@@ -420,14 +424,17 @@ void maybeReportMetadata(ClusterMetadata metadata)
@Override
protected void fetchTopologyInternal(long epoch)
{
+ if (ClusterMetadata.current().epoch.getEpoch() < epoch)
Review Comment:
Had to put this on the migration stage on my branch
##########
test/distributed/org/apache/cassandra/distributed/test/accord/AccordDropKeyspaceTest.java:
##########
@@ -35,8 +34,7 @@ public void dropKeyspace() throws IOException
int steps = 5;
try (Cluster cluster = Cluster.build(3)
.withoutVNodes()
- .withConfig(c -> c.with(Feature.values())
Review Comment:
When I merged I set it explicitly to `GOSSIP, NETWORK, NATIVE_PROTOCOL` but we should probably go back to `values` if the tests can run fine now that we know that `BLANK_GOSSIP` is ignored.
##########
src/java/org/apache/cassandra/service/accord/FetchTopology.java:
##########
@@ -123,18 +116,20 @@ public Response(long epoch, Topology topology)
long epoch = message.payload.epoch;
Topology topology = AccordService.instance().topology().maybeGlobalForEpoch(epoch);
if (topology == null)
- MessagingService.instance().respond(Response.UNKNOWN, message);
+ MessagingService.instance().respond(Response.unkonwn(epoch), message);
Review Comment:
Is this for test code that can catch the actual exception?
##########
src/java/org/apache/cassandra/service/accord/AccordService.java:
##########
@@ -410,48 +413,67 @@ public synchronized void startup()
}
/**
- * Queries peers to discover min epoch
+ * Queries peers to discover min epoch, and then fetches all topologies between min and current epochs
*/
- private long fetchMinEpoch()
+ private List<Topology> fetchTopologies(long minEpoch, ClusterMetadata metadata) throws ExecutionException, InterruptedException
{
- ClusterMetadata metadata = ClusterMetadata.current();
- Map<InetAddressAndPort, Set<TokenRange>> peers = new HashMap<>();
- for (KeyspaceMetadata keyspace : metadata.schema.getKeyspaces())
+ if (configService.maxEpoch() >= metadata.epoch.getEpoch())
{
- List<TableMetadata> tables = keyspace.tables.stream().filter(TableMetadata::requiresAccordSupport).collect(Collectors.toList());
- if (tables.isEmpty())
- continue;
- DataPlacement current = metadata.placements.get(keyspace.params.replication);
- DataPlacement settled = metadata.writePlacementAllSettled(keyspace);
- Sets.SetView<InetAddressAndPort> alive = Sets.intersection(settled.writes.byEndpoint().keySet(), current.writes.byEndpoint().keySet());
- InetAddressAndPort self = FBUtilities.getBroadcastAddressAndPort();
- settled.writes.forEach((range, group) -> {
- if (group.endpoints().contains(self))
- {
- for (InetAddressAndPort peer : group.endpoints())
- {
- if (peer.equals(self) || !alive.contains(peer)) continue;
- for (TableMetadata table : tables)
- peers.computeIfAbsent(peer, i -> new HashSet<>()).add(AccordTopology.fullRange(table.id));
- }
- }
- });
+ logger.info("Accord epoch {} matches TCM. All topologies are known locally", metadata.epoch);
+ return Collections.singletonList(AccordTopology.createAccordTopology(metadata));
}
+
+ Set<InetAddressAndPort> peers = new HashSet<>();
+ peers.addAll(metadata.directory.allAddresses());
+ peers.remove(FBUtilities.getBroadcastAddressAndPort());
+
+ // No peers: single node cluster or first node to boot
if (peers.isEmpty())
- return -1;
+ return Collections.singletonList(AccordTopology.createAccordTopology(metadata));;
+
+ // Bootstrap, fetch min epoch
+ if (minEpoch == 0)
Review Comment:
OK, that makes 0 sneaky. I guess 0 is the empty epoch so we never need to fetch it? It might be better/clearer to use a boxed Long and null or a constant.
##########
test/unit/org/apache/cassandra/service/accord/FetchMinEpochTest.java:
##########
@@ -271,21 +223,21 @@ private static MessageDelivery.FailedResponseException getFailedResponseExceptio
}
catch (ExecutionException e)
{
- if (e.getCause() instanceof MessageDelivery.FailedResponseException)
+ if (e.getCause() instanceof MessageDelivery.MaxRetriesException)
{
- exception = (MessageDelivery.FailedResponseException) e.getCause();
+ maxRetries = (MessageDelivery.MaxRetriesException) e.getCause();
}
else
{
throw e;
}
}
- return exception;
+ return maxRetries;
}
- private static MessageDelivery.MaxRetriesException getMaxRetriesException(Future<Long> f) throws InterruptedException, ExecutionException
+ private static MessageDelivery.FailedResponseException getFailedResponseException(Future<Long> f) throws InterruptedException, ExecutionException
Review Comment:
This method is unused
##########
src/java/org/apache/cassandra/service/accord/AccordService.java:
##########
@@ -410,48 +413,67 @@ public synchronized void startup()
}
/**
- * Queries peers to discover min epoch
+ * Queries peers to discover min epoch, and then fetches all topologies between min and current epochs
*/
- private long fetchMinEpoch()
+ private List<Topology> fetchTopologies(long minEpoch, ClusterMetadata metadata) throws ExecutionException, InterruptedException
{
- ClusterMetadata metadata = ClusterMetadata.current();
- Map<InetAddressAndPort, Set<TokenRange>> peers = new HashMap<>();
- for (KeyspaceMetadata keyspace : metadata.schema.getKeyspaces())
+ if (configService.maxEpoch() >= metadata.epoch.getEpoch())
{
- List<TableMetadata> tables = keyspace.tables.stream().filter(TableMetadata::requiresAccordSupport).collect(Collectors.toList());
- if (tables.isEmpty())
- continue;
- DataPlacement current = metadata.placements.get(keyspace.params.replication);
- DataPlacement settled = metadata.writePlacementAllSettled(keyspace);
- Sets.SetView<InetAddressAndPort> alive = Sets.intersection(settled.writes.byEndpoint().keySet(), current.writes.byEndpoint().keySet());
- InetAddressAndPort self = FBUtilities.getBroadcastAddressAndPort();
- settled.writes.forEach((range, group) -> {
- if (group.endpoints().contains(self))
- {
- for (InetAddressAndPort peer : group.endpoints())
- {
- if (peer.equals(self) || !alive.contains(peer)) continue;
- for (TableMetadata table : tables)
- peers.computeIfAbsent(peer, i -> new HashSet<>()).add(AccordTopology.fullRange(table.id));
- }
- }
- });
+ logger.info("Accord epoch {} matches TCM. All topologies are known locally", metadata.epoch);
+ return Collections.singletonList(AccordTopology.createAccordTopology(metadata));
}
+
+ Set<InetAddressAndPort> peers = new HashSet<>();
+ peers.addAll(metadata.directory.allAddresses());
+ peers.remove(FBUtilities.getBroadcastAddressAndPort());
+
+ // No peers: single node cluster or first node to boot
if (peers.isEmpty())
- return -1;
+ return Collections.singletonList(AccordTopology.createAccordTopology(metadata));;
+
+ // Bootstrap, fetch min epoch
+ if (minEpoch == 0)
+ {
+ long fetched = findMinEpoch(SharedContext.Global.instance, peers);
+ // No other node has advanced epoch just yet
+ if (fetched == 0)
+ return Collections.singletonList(AccordTopology.createAccordTopology(metadata));
+
+ minEpoch = fetched;
+ }
+
+ long maxEpoch = metadata.epoch.getEpoch();
+
+ // If we are behind minEpoch, catch up to at least minEpoch
+ if (metadata.epoch.getEpoch() < minEpoch)
+ {
+ minEpoch = metadata.epoch.getEpoch();
+ maxEpoch = minEpoch;
+ }
+
+ List<Future<Topology>> futures = new ArrayList<>();
+ logger.info("Discovered min epoch of {}. Proceeding to fetch epochs up to {}.", minEpoch, maxEpoch);
+
+ for (long epoch = minEpoch; epoch <= maxEpoch; epoch++)
+ futures.add(FetchTopology.fetch(SharedContext.Global.instance, peers, epoch));
Review Comment:
This is going to be kind of a flood if you have to collect a lot of these. Maybe not an issue if it's just one node.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-20245: Fix problems and race conditions with topology fetching [cassandra]
Posted by "dcapwell (via GitHub)" <gi...@apache.org>.
dcapwell commented on code in PR #3842:
URL: https://github.com/apache/cassandra/pull/3842#discussion_r1934249831
##########
src/java/org/apache/cassandra/service/accord/AccordService.java:
##########
@@ -377,24 +371,33 @@ public synchronized void startup()
node.commandStores().restoreShardStateUnsafe(topology -> configService.reportTopology(topology, true, true));
configService.start();
- long minEpoch = fetchMinEpoch();
- if (minEpoch >= 0)
+ try
{
- for (long epoch = minEpoch; epoch <= metadata.epoch.getEpoch(); epoch++)
- node.configService().fetchTopologyForEpoch(epoch);
+ // Fetch topologies up to current
+ List<Topology> topologies = fetchTopologies(0, metadata);
+ for (Topology topology : topologies)
+ configService.reportTopology(topology);
- try
- {
- epochReady(metadata.epoch).get(DatabaseDescriptor.getTransactionTimeout(MILLISECONDS), MILLISECONDS);
- }
- catch (InterruptedException e)
- {
- throw new UncheckedInterruptedException(e);
- }
- catch (ExecutionException | TimeoutException e)
+ ClusterMetadataService.instance().log().addListener(configService.listener);
+ ClusterMetadata next = ClusterMetadata.current();
+
+ // if metadata was updated before we were able to add a listener, fetch remaining topologies
+ if (metadata != next)
Review Comment:
when i think about this this feels currently safe, but feels off... if for some reason in the future, if TCM ever recreates the `ClusterMetadata` object, then this logic breaks; should we just check the epochs don't match instead?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-20245: Fix problems and race conditions with topology fetching [cassandra]
Posted by "ifesdjeen (via GitHub)" <gi...@apache.org>.
ifesdjeen commented on code in PR #3842:
URL: https://github.com/apache/cassandra/pull/3842#discussion_r1934374115
##########
src/java/org/apache/cassandra/service/accord/AccordConfigurationService.java:
##########
@@ -420,14 +424,17 @@ void maybeReportMetadata(ClusterMetadata metadata)
@Override
protected void fetchTopologyInternal(long epoch)
{
+ if (ClusterMetadata.current().epoch.getEpoch() < epoch)
+ ClusterMetadataService.instance().fetchLogFromCMS(Epoch.create(epoch));
+
try
{
Set<InetAddressAndPort> peers = new HashSet<>(ClusterMetadata.current().directory.allJoinedEndpoints());
peers.remove(FBUtilities.getBroadcastAddressAndPort());
if (peers.isEmpty())
return;
Topology topology;
- while ((topology =FetchTopology.fetch(SharedContext.Global.instance, peers, epoch).get()) == null) {}
+ while ((topology = FetchTopology.fetch(SharedContext.Global.instance, peers, epoch).get()) == null) {}
Review Comment:
Discussed in chat: intention for this was gap-filling. I agree that most commonly this will actually never be used. Added logic right before that to exit early if TCM has reported the metadata already.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-20245: Fix problems and race conditions with topology fetching [cassandra]
Posted by "ifesdjeen (via GitHub)" <gi...@apache.org>.
ifesdjeen commented on code in PR #3842:
URL: https://github.com/apache/cassandra/pull/3842#discussion_r1934307605
##########
test/distributed/org/apache/cassandra/distributed/test/accord/AccordDropTableTest.java:
##########
@@ -35,8 +34,7 @@ public void dropTable() throws IOException
int steps = 5;
try (Cluster cluster = Cluster.build(3)
.withoutVNodes()
- .withConfig(c -> c.with(Feature.values())
Review Comment:
Looks like I forgot to push the rebased version, where this was gone.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-20245: Fix problems and race conditions with topology fetching [cassandra]
Posted by "dcapwell (via GitHub)" <gi...@apache.org>.
dcapwell commented on code in PR #3842:
URL: https://github.com/apache/cassandra/pull/3842#discussion_r1934218316
##########
src/java/org/apache/cassandra/service/accord/FetchTopology.java:
##########
@@ -123,18 +116,20 @@ public Response(long epoch, Topology topology)
long epoch = message.payload.epoch;
Topology topology = AccordService.instance().topology().maybeGlobalForEpoch(epoch);
if (topology == null)
- MessagingService.instance().respond(Response.UNKNOWN, message);
+ MessagingService.instance().respond(Response.unkonwn(epoch), message);
else
MessagingService.instance().respond(new Response(epoch, topology), message);
};
+ private static final Logger logger = LoggerFactory.getLogger(FetchTopology.class);
+
public static Future<Topology> fetch(SharedContext context, Collection<InetAddressAndPort> peers, long epoch)
{
FetchTopology req = new FetchTopology(epoch);
- return context.messaging().<FetchTopology, Response>sendWithRetries(Verb.ACCORD_FETCH_TOPOLOGY_REQ, req, MessagingUtils.tryAliveFirst(SharedContext.Global.instance, peers),
- // If the epoch is already discovered, no need to retry
- (attempt, from, failure) -> AccordService.instance().currentEpoch() < epoch,
- MessageDelivery.RetryErrorMessage.EMPTY)
+ return context.messaging().<FetchTopology, Response>sendWithRetries(Verb.ACCORD_FETCH_TOPOLOGY_REQ, req,
+ MessagingUtils.tryAliveFirst(SharedContext.Global.instance, peers),
Review Comment:
is it intentional that this will loop forever? the first pass only does the alive nodes, then when all peers are tested we reset the iterator and try again.
I feel that its unsafe to use a `Backoff.NO_OP.INSTANCE` in this instance, we are blocking `org.apache.cassandra.service.accord.AccordConfigurationService#fetchTopologyInternal` for an unbounded amount of time and don't have any way to notify that things are going wrong.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-20245: Fix problems and race conditions with topology fetching [cassandra]
Posted by "ifesdjeen (via GitHub)" <gi...@apache.org>.
ifesdjeen commented on code in PR #3842:
URL: https://github.com/apache/cassandra/pull/3842#discussion_r1934319833
##########
src/java/org/apache/cassandra/service/accord/FetchTopology.java:
##########
@@ -123,18 +116,20 @@ public Response(long epoch, Topology topology)
long epoch = message.payload.epoch;
Topology topology = AccordService.instance().topology().maybeGlobalForEpoch(epoch);
if (topology == null)
- MessagingService.instance().respond(Response.UNKNOWN, message);
+ MessagingService.instance().respond(Response.unkonwn(epoch), message);
Review Comment:
switched to an explicit throw so that we would return UNKNOWN via messaging.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-20245: Fix problems and race conditions with topology fetching [cassandra]
Posted by "ifesdjeen (via GitHub)" <gi...@apache.org>.
ifesdjeen commented on code in PR #3842:
URL: https://github.com/apache/cassandra/pull/3842#discussion_r1934324629
##########
src/java/org/apache/cassandra/service/accord/AccordConfigurationService.java:
##########
@@ -420,14 +424,17 @@ void maybeReportMetadata(ClusterMetadata metadata)
@Override
protected void fetchTopologyInternal(long epoch)
{
+ if (ClusterMetadata.current().epoch.getEpoch() < epoch)
Review Comment:
Yep, I did that, rebased and pushed to `-rebased` branch while waiting for CI. Pushed now here too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-20245: Fix problems and race conditions with topology fetching [cassandra]
Posted by "dcapwell (via GitHub)" <gi...@apache.org>.
dcapwell commented on code in PR #3842:
URL: https://github.com/apache/cassandra/pull/3842#discussion_r1936397378
##########
src/java/org/apache/cassandra/service/accord/FetchTopology.java:
##########
@@ -123,18 +116,20 @@ public Response(long epoch, Topology topology)
long epoch = message.payload.epoch;
Topology topology = AccordService.instance().topology().maybeGlobalForEpoch(epoch);
if (topology == null)
- MessagingService.instance().respond(Response.UNKNOWN, message);
+ MessagingService.instance().respond(Response.unkonwn(epoch), message);
Review Comment:
created https://github.com/apache/cassandra/pull/3842#discussion_r1936394993 so this doesn't get lost...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-20245: Fix problems and race conditions with topology fetching [cassandra]
Posted by "dcapwell (via GitHub)" <gi...@apache.org>.
dcapwell commented on PR #3842:
URL: https://github.com/apache/cassandra/pull/3842#issuecomment-2628040122
My only open comment is https://github.com/apache/cassandra/pull/3842/files#r1936388652
If this was an assert i would be +1, but its a if condition that adds a new control flow, which worries me.... once this issue is resolved I am +1 to this patch
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-20245: Fix problems and race conditions with topology fetching [cassandra]
Posted by "dcapwell (via GitHub)" <gi...@apache.org>.
dcapwell commented on code in PR #3842:
URL: https://github.com/apache/cassandra/pull/3842#discussion_r1934177395
##########
test/distributed/org/apache/cassandra/distributed/test/accord/AccordDropKeyspaceTest.java:
##########
@@ -35,8 +34,7 @@ public void dropKeyspace() throws IOException
int steps = 5;
try (Cluster cluster = Cluster.build(3)
.withoutVNodes()
- .withConfig(c -> c.with(Feature.values())
Review Comment:
please revert, `Feature.values()` is 100% intentional
##########
test/distributed/org/apache/cassandra/distributed/test/accord/AccordDropTableTest.java:
##########
@@ -35,8 +34,7 @@ public void dropTable() throws IOException
int steps = 5;
try (Cluster cluster = Cluster.build(3)
.withoutVNodes()
- .withConfig(c -> c.with(Feature.values())
Review Comment:
please revert, `Feature.values()` is 100% intentional
##########
src/java/org/apache/cassandra/service/accord/FetchTopology.java:
##########
@@ -37,6 +41,7 @@
public class FetchTopology
{
+ private static final Logger log = LoggerFactory.getLogger(FetchTopology.class);
Review Comment:
nit, in this project we always use `logger`
```
$ grep -r 'Logger log' src/ | awk -F: '{print $2}' | awk -F= '{print $1}' | sort | uniq -c | sort -k1,1 -h
1 Logger logBackLogger
1 Logger logbackLogger
1 private static Logger logger
1 public static void logInitializationOutcome(Logger logger)
1 private boolean hasAppenders(Logger logBackLogger)
1 private static Logger logger()
1 private static final Logger logger;
1 private static final org.slf4j.Logger logger
1 protected static final NoSpamLogger logger
1 public static <K, V> void dumpDiff(Logger logger, Map<K, V> l, Map<K, V> r)
1 public static Logger logger
1 public static NoSpamLogStatement getStatement(Logger logger, String message, long minInterval, TimeUnit unit)
1 public static NoSpamLogger getLogger(Logger logger, long minInterval, TimeUnit unit)
1 public static boolean log(Logger logger, Level level, String key, long minInterval, TimeUnit unit, String message, Object... objects)
1 public static boolean log(Logger logger, Level level, String key, long minInterval, TimeUnit unit, String message, Supplier<Object[]> objects)
1 public static boolean log(Logger logger, Level level, String key, long minInterval, TimeUnit unit, long nowNanos, String message, Object... objects)
1 public static boolean log(Logger logger, Level level, String key, long minInterval, TimeUnit unit, long nowNanos, String message, Supplier<Object[]> objects)
1 public static boolean log(Logger logger, Level level, long minInterval, TimeUnit unit, String message, Object... objects)
1 public static boolean log(Logger logger, Level level, long minInterval, TimeUnit unit, String message, Supplier<Object[]> objects)
1 public static void dumpDiff(Logger logger, Map<ReplicationParams, DataPlacement> l, Map<ReplicationParams, DataPlacement> r)
1 public static void logSystemInfo(Logger logger)
1 public void log(Logger log)
1 public void logRowCountPerLeaf(Logger logger)
1 public void logRowSizePerLeaf(Logger logger)
2 protected final static Logger logger
3 for (Logger logBackLogger
3 private final Logger logger
3 private static Logger logger
6 static final Logger logger
7 Logger logger
11 private final static Logger logger
19 private static final Logger logger
26 protected static final Logger logger
464 private static final Logger logger
```
##########
src/java/org/apache/cassandra/service/accord/FetchMinEpoch.java:
##########
@@ -53,40 +50,32 @@
// TODO (required, efficiency): this can be simplified: we seem to always use "entire range"
public class FetchMinEpoch
{
+ private static final FetchMinEpoch instance = new FetchMinEpoch();
Review Comment:
nit, we could use `NoPayload` instead
##########
src/java/org/apache/cassandra/service/accord/FetchTopology.java:
##########
@@ -67,24 +72,17 @@ public FetchTopology(long epoch)
public static class Response
{
- private static Response UNKNOWN = new Response(-1, null) {
- public String toString()
- {
- return "UNKNOWN_TOPOLOGY{}";
- }
- };
+ private static Response unkonwn(long epoch)
Review Comment:
`unknown`?
##########
src/java/org/apache/cassandra/service/accord/FetchTopology.java:
##########
@@ -123,18 +116,20 @@ public Response(long epoch, Topology topology)
long epoch = message.payload.epoch;
Topology topology = AccordService.instance().topology().maybeGlobalForEpoch(epoch);
if (topology == null)
- MessagingService.instance().respond(Response.UNKNOWN, message);
+ MessagingService.instance().respond(Response.unkonwn(epoch), message);
Review Comment:
im confused by this line
```
private static Response unkonwn(long epoch)
{
throw new IllegalStateException("Unknown topology: " + epoch);
}
```
`Response.unkonwn` only throws, so this `response` won't do anything as that method won't be called
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-20245: Fix problems and race conditions with topology fetching [cassandra]
Posted by "ifesdjeen (via GitHub)" <gi...@apache.org>.
ifesdjeen commented on code in PR #3842:
URL: https://github.com/apache/cassandra/pull/3842#discussion_r1934438540
##########
src/java/org/apache/cassandra/service/accord/FetchTopology.java:
##########
@@ -123,18 +116,20 @@ public Response(long epoch, Topology topology)
long epoch = message.payload.epoch;
Topology topology = AccordService.instance().topology().maybeGlobalForEpoch(epoch);
if (topology == null)
- MessagingService.instance().respond(Response.UNKNOWN, message);
+ MessagingService.instance().respond(Response.unkonwn(epoch), message);
else
MessagingService.instance().respond(new Response(epoch, topology), message);
};
+ private static final Logger logger = LoggerFactory.getLogger(FetchTopology.class);
+
public static Future<Topology> fetch(SharedContext context, Collection<InetAddressAndPort> peers, long epoch)
{
FetchTopology req = new FetchTopology(epoch);
- return context.messaging().<FetchTopology, Response>sendWithRetries(Verb.ACCORD_FETCH_TOPOLOGY_REQ, req, MessagingUtils.tryAliveFirst(SharedContext.Global.instance, peers),
- // If the epoch is already discovered, no need to retry
- (attempt, from, failure) -> AccordService.instance().currentEpoch() < epoch,
- MessageDelivery.RetryErrorMessage.EMPTY)
+ return context.messaging().<FetchTopology, Response>sendWithRetries(Verb.ACCORD_FETCH_TOPOLOGY_REQ, req,
+ MessagingUtils.tryAliveFirst(SharedContext.Global.instance, peers),
Review Comment:
Added an attempt counter, and a log message on each iteration.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-20245: Fix problems and race conditions with topology fetching [cassandra]
Posted by "dcapwell (via GitHub)" <gi...@apache.org>.
dcapwell commented on code in PR #3842:
URL: https://github.com/apache/cassandra/pull/3842#discussion_r1936391388
##########
src/java/org/apache/cassandra/service/accord/AccordService.java:
##########
@@ -379,25 +373,48 @@ public synchronized void startup()
node.commandStores().restoreShardStateUnsafe(topology -> configService.reportTopology(topology, true, true));
configService.start();
- long minEpoch = fetchMinEpoch();
- if (minEpoch >= 0)
+ try
{
- for (long epoch = minEpoch; epoch <= metadata.epoch.getEpoch(); epoch++)
- node.configService().fetchTopologyForEpoch(epoch);
+ // Fetch topologies up to current
+ List<Topology> topologies = fetchTopologies(null, metadata);
+ for (Topology topology : topologies)
+ configService.reportTopology(topology);
- try
- {
- epochReady(metadata.epoch).get(DatabaseDescriptor.getTransactionTimeout(MILLISECONDS), MILLISECONDS);
- }
- catch (InterruptedException e)
+ ClusterMetadataService.instance().log().addListener(configService.listener);
+ ClusterMetadata next = ClusterMetadata.current();
+
+ // if metadata was updated before we were able to add a listener, fetch remaining topologies
+ if (next.epoch.isAfter(metadata.epoch))
{
- throw new UncheckedInterruptedException(e);
+ topologies = fetchTopologies(metadata.epoch.getEpoch() + 1, next);
+ for (Topology topology : topologies)
+ configService.reportTopology(topology);
}
- catch (ExecutionException | TimeoutException e)
+
+ int attempt = 0;
+ int waitSeconds = 5;
+ while (true)
{
- throw new RuntimeException(e);
+ try
+ {
+ epochReady(metadata.epoch).get(5, SECONDS);
Review Comment:
```suggestion
epochReady(metadata.epoch).get(waitSeconds, SECONDS);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-20245: Fix problems and race conditions with topology fetching [cassandra]
Posted by "dcapwell (via GitHub)" <gi...@apache.org>.
dcapwell commented on code in PR #3842:
URL: https://github.com/apache/cassandra/pull/3842#discussion_r1936388652
##########
src/java/org/apache/cassandra/service/accord/AccordConfigurationService.java:
##########
@@ -416,14 +415,18 @@ void maybeReportMetadata(ClusterMetadata metadata)
long epoch = metadata.epoch.getEpoch();
synchronized (epochs)
{
- if (epochs.maxEpoch() == 0)
+ long maxEpoch = epochs.maxEpoch();
+ if (maxEpoch == 0)
{
getOrCreateEpochState(epoch); // touch epoch state so subsequent calls see it
reportMetadata(metadata);
return;
}
}
- getOrCreateEpochState(epoch - 1).acknowledged().addCallback(() -> reportMetadata(metadata));
+
+ // Create a -1 epoch iif we know this epoch may actually exist
+ if (metadata.epoch.getEpoch() > minEpoch())
Review Comment:
why this change? this worries me. Making sure this protocol was safe took a lot of effort, so adding this extra state into it makes me worry that we will regress...
If TCM is reporting, we won't hit this... so only non-TCM reporting should hit this... we should guard there IMO
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-20245: Fix problems and race conditions with topology fetching [cassandra]
Posted by "aweisberg (via GitHub)" <gi...@apache.org>.
aweisberg commented on code in PR #3842:
URL: https://github.com/apache/cassandra/pull/3842#discussion_r1937992423
##########
src/java/org/apache/cassandra/service/accord/AccordService.java:
##########
@@ -377,24 +371,33 @@ public synchronized void startup()
node.commandStores().restoreShardStateUnsafe(topology -> configService.reportTopology(topology, true, true));
configService.start();
- long minEpoch = fetchMinEpoch();
- if (minEpoch >= 0)
+ try
{
- for (long epoch = minEpoch; epoch <= metadata.epoch.getEpoch(); epoch++)
- node.configService().fetchTopologyForEpoch(epoch);
+ // Fetch topologies up to current
+ List<Topology> topologies = fetchTopologies(0, metadata);
Review Comment:
A named constant that is `null` is still clearer just because you can describe the semantic of it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-20245: Fix problems and race conditions with topology fetching [cassandra]
Posted by "dcapwell (via GitHub)" <gi...@apache.org>.
dcapwell commented on code in PR #3842:
URL: https://github.com/apache/cassandra/pull/3842#discussion_r1936395268
##########
src/java/org/apache/cassandra/service/accord/FetchTopology.java:
##########
@@ -122,19 +114,26 @@ public Response(long epoch, Topology topology)
public static final IVerbHandler<FetchTopology> handler = message -> {
long epoch = message.payload.epoch;
Topology topology = AccordService.instance().topology().maybeGlobalForEpoch(epoch);
- if (topology == null)
- MessagingService.instance().respond(Response.UNKNOWN, message);
- else
+ if (topology != null)
MessagingService.instance().respond(new Response(epoch, topology), message);
+ else
+ throw new IllegalStateException("Unknown topology: " + epoch);
};
public static Future<Topology> fetch(SharedContext context, Collection<InetAddressAndPort> peers, long epoch)
{
- FetchTopology req = new FetchTopology(epoch);
- return context.messaging().<FetchTopology, Response>sendWithRetries(Verb.ACCORD_FETCH_TOPOLOGY_REQ, req, MessagingUtils.tryAliveFirst(SharedContext.Global.instance, peers),
- // If the epoch is already discovered, no need to retry
- (attempt, from, failure) -> AccordService.instance().currentEpoch() < epoch,
- MessageDelivery.RetryErrorMessage.EMPTY)
+ FetchTopology request = new FetchTopology(epoch);
+ Backoff backoff = Backoff.fromConfig(context, DatabaseDescriptor.getAccord().fetchRetry);
+ return context.messaging().<FetchTopology, Response>sendWithRetries(backoff,
+ context.optionalTasks()::schedule,
+ Verb.ACCORD_FETCH_TOPOLOGY_REQ,
+ request,
+ MessagingUtils.tryAliveFirst(SharedContext.Global.instance, peers, Verb.ACCORD_FETCH_TOPOLOGY_REQ.name()),
+ (attempt, from, failure) -> {
+ System.out.println("Got " + failure + " from " + from + " while fetching " + request);
Review Comment:
please use a `logger`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-20245: Fix problems and race conditions with topology fetching [cassandra]
Posted by "dcapwell (via GitHub)" <gi...@apache.org>.
dcapwell commented on code in PR #3842:
URL: https://github.com/apache/cassandra/pull/3842#discussion_r1936382918
##########
src/java/org/apache/cassandra/config/AccordSpec.java:
##########
@@ -194,15 +194,17 @@ public enum TransactionalRangeMigration
public boolean ephemeralReadEnabled = true;
public boolean state_cache_listener_jfr_enabled = true;
public final JournalSpec journal = new JournalSpec();
- public final MinEpochRetrySpec minEpochSyncRetry = new MinEpochRetrySpec();
-
- public static class MinEpochRetrySpec extends RetrySpec
- {
- public MinEpochRetrySpec()
+ public final RetrySpec minEpochSyncRetry = new RetrySpec() {
Review Comment:
is `org.apache.cassandra.config.DatabaseDescriptorRefTest` clean?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-20245: Fix problems and race conditions with topology fetching [cassandra]
Posted by "dcapwell (via GitHub)" <gi...@apache.org>.
dcapwell commented on code in PR #3842:
URL: https://github.com/apache/cassandra/pull/3842#discussion_r1936385161
##########
src/java/org/apache/cassandra/service/accord/AccordConfigurationService.java:
##########
@@ -213,6 +213,7 @@ public EpochDiskState truncateTopologyUntil(long epoch, EpochDiskState diskState
}
}
+ // TODO: should not be public
Review Comment:
```suggestion
//TODO (???): should not be public
```
2 things
`//TODO` vs `// TODO`, not sure why intellij cares...
`(...)` should have a scope?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-20245: Fix problems and race conditions with topology fetching [cassandra]
Posted by "ifesdjeen (via GitHub)" <gi...@apache.org>.
ifesdjeen commented on code in PR #3842:
URL: https://github.com/apache/cassandra/pull/3842#discussion_r1936196300
##########
src/java/org/apache/cassandra/service/accord/FetchMinEpoch.java:
##########
@@ -145,21 +108,22 @@ public static Future<Long> fetch(SharedContext context, Map<InetAddressAndPort,
}
@VisibleForTesting
- static Future<Long> fetch(SharedContext context, InetAddressAndPort to, Set<TokenRange> value)
+ static Future<Long> fetch(SharedContext context, InetAddressAndPort to)
{
- FetchMinEpoch req = new FetchMinEpoch(value);
- return context.messaging().<FetchMinEpoch, FetchMinEpoch.Response>sendWithRetries(Backoff.NO_OP.INSTANCE,
- MessageDelivery.ImmediateRetryScheduler.instance,
- Verb.ACCORD_FETCH_MIN_EPOCH_REQ, req,
+ Backoff backoff = Backoff.fromConfig(context, DatabaseDescriptor.getAccord().minEpochSyncRetry);
+ return context.messaging().<FetchMinEpoch, FetchMinEpoch.Response>sendWithRetries(backoff,
+ context.optionalTasks()::schedule,
+ Verb.ACCORD_FETCH_MIN_EPOCH_REQ,
+ FetchMinEpoch.instance,
Iterators.cycle(to),
- RetryPredicate.times(DatabaseDescriptor.getAccord().minEpochSyncRetry.maxAttempts.value),
+ RetryPredicate.ALWAYS_RETRY,
Review Comment:
iirc this is faithful to the original implementation, which I believe might have been not entirely correct. We're collecting responses from all nodes, but we consider only successes here, which might mean we will not discover an early enough epoch.
We planned to address this when implementing epoch GC, where we'll indicate which epoch is retired, and a better success criteria for this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-20245: Fix problems and race conditions with topology fetching [cassandra]
Posted by "dcapwell (via GitHub)" <gi...@apache.org>.
dcapwell commented on code in PR #3842:
URL: https://github.com/apache/cassandra/pull/3842#discussion_r1934243353
##########
src/java/org/apache/cassandra/service/accord/AccordService.java:
##########
@@ -410,48 +413,67 @@ public synchronized void startup()
}
/**
- * Queries peers to discover min epoch
+ * Queries peers to discover min epoch, and then fetches all topologies between min and current epochs
*/
- private long fetchMinEpoch()
+ private List<Topology> fetchTopologies(long minEpoch, ClusterMetadata metadata) throws ExecutionException, InterruptedException
{
- ClusterMetadata metadata = ClusterMetadata.current();
- Map<InetAddressAndPort, Set<TokenRange>> peers = new HashMap<>();
- for (KeyspaceMetadata keyspace : metadata.schema.getKeyspaces())
+ if (configService.maxEpoch() >= metadata.epoch.getEpoch())
{
- List<TableMetadata> tables = keyspace.tables.stream().filter(TableMetadata::requiresAccordSupport).collect(Collectors.toList());
- if (tables.isEmpty())
- continue;
- DataPlacement current = metadata.placements.get(keyspace.params.replication);
- DataPlacement settled = metadata.writePlacementAllSettled(keyspace);
- Sets.SetView<InetAddressAndPort> alive = Sets.intersection(settled.writes.byEndpoint().keySet(), current.writes.byEndpoint().keySet());
- InetAddressAndPort self = FBUtilities.getBroadcastAddressAndPort();
- settled.writes.forEach((range, group) -> {
- if (group.endpoints().contains(self))
- {
- for (InetAddressAndPort peer : group.endpoints())
- {
- if (peer.equals(self) || !alive.contains(peer)) continue;
- for (TableMetadata table : tables)
- peers.computeIfAbsent(peer, i -> new HashSet<>()).add(AccordTopology.fullRange(table.id));
- }
- }
- });
+ logger.info("Accord epoch {} matches TCM. All topologies are known locally", metadata.epoch);
+ return Collections.singletonList(AccordTopology.createAccordTopology(metadata));
}
+
+ Set<InetAddressAndPort> peers = new HashSet<>();
+ peers.addAll(metadata.directory.allAddresses());
+ peers.remove(FBUtilities.getBroadcastAddressAndPort());
+
+ // No peers: single node cluster or first node to boot
if (peers.isEmpty())
- return -1;
+ return Collections.singletonList(AccordTopology.createAccordTopology(metadata));;
+
+ // Bootstrap, fetch min epoch
+ if (minEpoch == 0)
+ {
+ long fetched = findMinEpoch(SharedContext.Global.instance, peers);
+ // No other node has advanced epoch just yet
+ if (fetched == 0)
+ return Collections.singletonList(AccordTopology.createAccordTopology(metadata));
+
+ minEpoch = fetched;
+ }
+
+ long maxEpoch = metadata.epoch.getEpoch();
+
+ // If we are behind minEpoch, catch up to at least minEpoch
+ if (metadata.epoch.getEpoch() < minEpoch)
+ {
+ minEpoch = metadata.epoch.getEpoch();
+ maxEpoch = minEpoch;
+ }
+
+ List<Future<Topology>> futures = new ArrayList<>();
+ logger.info("Discovered min epoch of {}. Proceeding to fetch epochs up to {}.", minEpoch, maxEpoch);
+
+ for (long epoch = minEpoch; epoch <= maxEpoch; epoch++)
+ futures.add(FetchTopology.fetch(SharedContext.Global.instance, peers, epoch));
Review Comment:
is there a reason to not make this a batch? if you know `N` you know `N - 1`, so rather than do many concurrent fetches we could just ask for the full batch in 1 request?
thinking about the current benchmark case, you can startup with 50 epochs just in benchmarks, so we have 50 concurrent fetches, and they will go to the same node at the same time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-20245: Fix problems and race conditions with topology fetching [cassandra]
Posted by "dcapwell (via GitHub)" <gi...@apache.org>.
dcapwell commented on code in PR #3842:
URL: https://github.com/apache/cassandra/pull/3842#discussion_r1934243353
##########
src/java/org/apache/cassandra/service/accord/AccordService.java:
##########
@@ -410,48 +413,67 @@ public synchronized void startup()
}
/**
- * Queries peers to discover min epoch
+ * Queries peers to discover min epoch, and then fetches all topologies between min and current epochs
*/
- private long fetchMinEpoch()
+ private List<Topology> fetchTopologies(long minEpoch, ClusterMetadata metadata) throws ExecutionException, InterruptedException
{
- ClusterMetadata metadata = ClusterMetadata.current();
- Map<InetAddressAndPort, Set<TokenRange>> peers = new HashMap<>();
- for (KeyspaceMetadata keyspace : metadata.schema.getKeyspaces())
+ if (configService.maxEpoch() >= metadata.epoch.getEpoch())
{
- List<TableMetadata> tables = keyspace.tables.stream().filter(TableMetadata::requiresAccordSupport).collect(Collectors.toList());
- if (tables.isEmpty())
- continue;
- DataPlacement current = metadata.placements.get(keyspace.params.replication);
- DataPlacement settled = metadata.writePlacementAllSettled(keyspace);
- Sets.SetView<InetAddressAndPort> alive = Sets.intersection(settled.writes.byEndpoint().keySet(), current.writes.byEndpoint().keySet());
- InetAddressAndPort self = FBUtilities.getBroadcastAddressAndPort();
- settled.writes.forEach((range, group) -> {
- if (group.endpoints().contains(self))
- {
- for (InetAddressAndPort peer : group.endpoints())
- {
- if (peer.equals(self) || !alive.contains(peer)) continue;
- for (TableMetadata table : tables)
- peers.computeIfAbsent(peer, i -> new HashSet<>()).add(AccordTopology.fullRange(table.id));
- }
- }
- });
+ logger.info("Accord epoch {} matches TCM. All topologies are known locally", metadata.epoch);
+ return Collections.singletonList(AccordTopology.createAccordTopology(metadata));
}
+
+ Set<InetAddressAndPort> peers = new HashSet<>();
+ peers.addAll(metadata.directory.allAddresses());
+ peers.remove(FBUtilities.getBroadcastAddressAndPort());
+
+ // No peers: single node cluster or first node to boot
if (peers.isEmpty())
- return -1;
+ return Collections.singletonList(AccordTopology.createAccordTopology(metadata));;
+
+ // Bootstrap, fetch min epoch
+ if (minEpoch == 0)
+ {
+ long fetched = findMinEpoch(SharedContext.Global.instance, peers);
+ // No other node has advanced epoch just yet
+ if (fetched == 0)
+ return Collections.singletonList(AccordTopology.createAccordTopology(metadata));
+
+ minEpoch = fetched;
+ }
+
+ long maxEpoch = metadata.epoch.getEpoch();
+
+ // If we are behind minEpoch, catch up to at least minEpoch
+ if (metadata.epoch.getEpoch() < minEpoch)
+ {
+ minEpoch = metadata.epoch.getEpoch();
+ maxEpoch = minEpoch;
+ }
+
+ List<Future<Topology>> futures = new ArrayList<>();
+ logger.info("Discovered min epoch of {}. Proceeding to fetch epochs up to {}.", minEpoch, maxEpoch);
+
+ for (long epoch = minEpoch; epoch <= maxEpoch; epoch++)
+ futures.add(FetchTopology.fetch(SharedContext.Global.instance, peers, epoch));
Review Comment:
is there a reason to not make this a batch? if you know `N` you know `N - 1`, so rather than do many concurrent fetches we could just ask for the full batch in 1 request?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-20245: Fix problems and race conditions with topology fetching [cassandra]
Posted by "ifesdjeen (via GitHub)" <gi...@apache.org>.
ifesdjeen commented on code in PR #3842:
URL: https://github.com/apache/cassandra/pull/3842#discussion_r1934401867
##########
src/java/org/apache/cassandra/service/accord/AccordService.java:
##########
@@ -410,48 +413,67 @@ public synchronized void startup()
}
/**
- * Queries peers to discover min epoch
+ * Queries peers to discover min epoch, and then fetches all topologies between min and current epochs
*/
- private long fetchMinEpoch()
+ private List<Topology> fetchTopologies(long minEpoch, ClusterMetadata metadata) throws ExecutionException, InterruptedException
{
- ClusterMetadata metadata = ClusterMetadata.current();
- Map<InetAddressAndPort, Set<TokenRange>> peers = new HashMap<>();
- for (KeyspaceMetadata keyspace : metadata.schema.getKeyspaces())
+ if (configService.maxEpoch() >= metadata.epoch.getEpoch())
{
- List<TableMetadata> tables = keyspace.tables.stream().filter(TableMetadata::requiresAccordSupport).collect(Collectors.toList());
- if (tables.isEmpty())
- continue;
- DataPlacement current = metadata.placements.get(keyspace.params.replication);
- DataPlacement settled = metadata.writePlacementAllSettled(keyspace);
- Sets.SetView<InetAddressAndPort> alive = Sets.intersection(settled.writes.byEndpoint().keySet(), current.writes.byEndpoint().keySet());
- InetAddressAndPort self = FBUtilities.getBroadcastAddressAndPort();
- settled.writes.forEach((range, group) -> {
- if (group.endpoints().contains(self))
- {
- for (InetAddressAndPort peer : group.endpoints())
- {
- if (peer.equals(self) || !alive.contains(peer)) continue;
- for (TableMetadata table : tables)
- peers.computeIfAbsent(peer, i -> new HashSet<>()).add(AccordTopology.fullRange(table.id));
- }
- }
- });
+ logger.info("Accord epoch {} matches TCM. All topologies are known locally", metadata.epoch);
+ return Collections.singletonList(AccordTopology.createAccordTopology(metadata));
}
+
+ Set<InetAddressAndPort> peers = new HashSet<>();
+ peers.addAll(metadata.directory.allAddresses());
+ peers.remove(FBUtilities.getBroadcastAddressAndPort());
+
+ // No peers: single node cluster or first node to boot
if (peers.isEmpty())
- return -1;
+ return Collections.singletonList(AccordTopology.createAccordTopology(metadata));;
+
+ // Bootstrap, fetch min epoch
+ if (minEpoch == 0)
+ {
+ long fetched = findMinEpoch(SharedContext.Global.instance, peers);
+ // No other node has advanced epoch just yet
+ if (fetched == 0)
+ return Collections.singletonList(AccordTopology.createAccordTopology(metadata));
+
+ minEpoch = fetched;
+ }
+
+ long maxEpoch = metadata.epoch.getEpoch();
+
+ // If we are behind minEpoch, catch up to at least minEpoch
+ if (metadata.epoch.getEpoch() < minEpoch)
+ {
+ minEpoch = metadata.epoch.getEpoch();
+ maxEpoch = minEpoch;
+ }
+
+ List<Future<Topology>> futures = new ArrayList<>();
+ logger.info("Discovered min epoch of {}. Proceeding to fetch epochs up to {}.", minEpoch, maxEpoch);
+
+ for (long epoch = minEpoch; epoch <= maxEpoch; epoch++)
+ futures.add(FetchTopology.fetch(SharedContext.Global.instance, peers, epoch));
Review Comment:
I agree here. Talked to @dcapwell and we agreed to switch to batching (i.e. querying as many epochs as we can) in one run, in a follow-up patch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-20245: Fix problems and race conditions with topology fetching [cassandra]
Posted by "dcapwell (via GitHub)" <gi...@apache.org>.
dcapwell commented on code in PR #3842:
URL: https://github.com/apache/cassandra/pull/3842#discussion_r1936389078
##########
src/java/org/apache/cassandra/service/accord/AccordConfigurationService.java:
##########
@@ -433,16 +436,25 @@ protected void fetchTopologyInternal(long epoch)
Stage.ACCORD_MIGRATION.execute(() -> {
if (ClusterMetadata.current().epoch.getEpoch() < epoch)
ClusterMetadataService.instance().fetchLogFromCMS(Epoch.create(epoch));
+
+ // In most cases, after fetching log from CMS, we will be caught up to the required epoch.
+ // This TCM will also notify Accord via reportMetadata, so we do not need to fetch topologies.
+ // If metadata has reported has skipped one or more eopchs, and is _ahead_ of the requested epoch,
Review Comment:
```suggestion
// If metadata has reported has skipped one or more epochs, and is _ahead_ of the requested epoch,
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-20245: Fix problems and race conditions with topology fetching [cassandra]
Posted by "dcapwell (via GitHub)" <gi...@apache.org>.
dcapwell commented on code in PR #3842:
URL: https://github.com/apache/cassandra/pull/3842#discussion_r1937741779
##########
src/java/org/apache/cassandra/service/accord/AccordConfigurationService.java:
##########
@@ -433,16 +436,25 @@ protected void fetchTopologyInternal(long epoch)
Stage.ACCORD_MIGRATION.execute(() -> {
if (ClusterMetadata.current().epoch.getEpoch() < epoch)
ClusterMetadataService.instance().fetchLogFromCMS(Epoch.create(epoch));
+
+ // In most cases, after fetching log from CMS, we will be caught up to the required epoch.
+ // This TCM will also notify Accord via reportMetadata, so we do not need to fetch topologies.
+ // If metadata has reported has skipped one or more epochs, and is _ahead_ of the requested epoch,
+ // we need to fetch topologies from peers to fill in the gap.
+ ClusterMetadata metadata = ClusterMetadata.current();
+ if (metadata.epoch.getEpoch() == epoch)
+ return;
+
try
{
- Set<InetAddressAndPort> peers = new HashSet<>(ClusterMetadata.current().directory.allJoinedEndpoints());
+ Set<InetAddressAndPort> peers = new HashSet<>(metadata.directory.allJoinedEndpoints());
peers.remove(FBUtilities.getBroadcastAddressAndPort());
if (peers.isEmpty())
return;
- Topology topology;
- while ((topology = FetchTopology.fetch(SharedContext.Global.instance, peers, epoch).get()) == null)
- {
- }
+
+ // TODO (required): fetch only _missing_ topologies.
Review Comment:
one additional optimization... now that we waiting for TCM, TCM might have already informed us... we could check if accord has this epoch and avoid this. Maybe add a TODO for that?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-20245: Fix problems and race conditions with topology fetching [cassandra]
Posted by "aweisberg (via GitHub)" <gi...@apache.org>.
aweisberg commented on code in PR #3842:
URL: https://github.com/apache/cassandra/pull/3842#discussion_r1937993680
##########
src/java/org/apache/cassandra/service/accord/FetchMinEpoch.java:
##########
@@ -145,21 +108,22 @@ public static Future<Long> fetch(SharedContext context, Map<InetAddressAndPort,
}
@VisibleForTesting
- static Future<Long> fetch(SharedContext context, InetAddressAndPort to, Set<TokenRange> value)
+ static Future<Long> fetch(SharedContext context, InetAddressAndPort to)
{
- FetchMinEpoch req = new FetchMinEpoch(value);
- return context.messaging().<FetchMinEpoch, FetchMinEpoch.Response>sendWithRetries(Backoff.NO_OP.INSTANCE,
- MessageDelivery.ImmediateRetryScheduler.instance,
- Verb.ACCORD_FETCH_MIN_EPOCH_REQ, req,
+ Backoff backoff = Backoff.fromConfig(context, DatabaseDescriptor.getAccord().minEpochSyncRetry);
+ return context.messaging().<FetchMinEpoch, FetchMinEpoch.Response>sendWithRetries(backoff,
+ context.optionalTasks()::schedule,
+ Verb.ACCORD_FETCH_MIN_EPOCH_REQ,
+ FetchMinEpoch.instance,
Iterators.cycle(to),
- RetryPredicate.times(DatabaseDescriptor.getAccord().minEpochSyncRetry.maxAttempts.value),
+ RetryPredicate.ALWAYS_RETRY,
Review Comment:
I think that's not faithful to what was here, because it had a maximum number of attempts before so eventually it would complete? Now it retries forever so it will never stop?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-20245: Fix problems and race conditions with topology fetching [cassandra]
Posted by "dcapwell (via GitHub)" <gi...@apache.org>.
dcapwell commented on code in PR #3842:
URL: https://github.com/apache/cassandra/pull/3842#discussion_r1934253137
##########
src/java/org/apache/cassandra/service/accord/AccordService.java:
##########
@@ -377,24 +371,33 @@ public synchronized void startup()
node.commandStores().restoreShardStateUnsafe(topology -> configService.reportTopology(topology, true, true));
configService.start();
- long minEpoch = fetchMinEpoch();
- if (minEpoch >= 0)
+ try
{
- for (long epoch = minEpoch; epoch <= metadata.epoch.getEpoch(); epoch++)
- node.configService().fetchTopologyForEpoch(epoch);
+ // Fetch topologies up to current
+ List<Topology> topologies = fetchTopologies(0, metadata);
+ for (Topology topology : topologies)
+ configService.reportTopology(topology);
- try
- {
- epochReady(metadata.epoch).get(DatabaseDescriptor.getTransactionTimeout(MILLISECONDS), MILLISECONDS);
- }
- catch (InterruptedException e)
- {
- throw new UncheckedInterruptedException(e);
- }
- catch (ExecutionException | TimeoutException e)
+ ClusterMetadataService.instance().log().addListener(configService.listener);
+ ClusterMetadata next = ClusterMetadata.current();
+
+ // if metadata was updated before we were able to add a listener, fetch remaining topologies
+ if (metadata != next)
{
- throw new RuntimeException(e);
+ topologies = fetchTopologies(metadata.epoch.getEpoch(), next);
+ for (Topology topology : topologies)
+ configService.reportTopology(topology);
}
+
+ epochReady(metadata.epoch).get();
+ }
+ catch (InterruptedException e)
+ {
+ throw new UncheckedInterruptedException(e);
Review Comment:
```suggestion
Thread.currentThread().interrupt();
throw new UncheckedInterruptedException(e);
```
since we don't handle, we just rethrow we should reset the interrupt flag so it doesn't get swallowed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-20245: Fix problems and race conditions with topology fetching [cassandra]
Posted by "ifesdjeen (via GitHub)" <gi...@apache.org>.
ifesdjeen commented on code in PR #3842:
URL: https://github.com/apache/cassandra/pull/3842#discussion_r1934397876
##########
src/java/org/apache/cassandra/service/accord/FetchTopology.java:
##########
@@ -123,18 +116,20 @@ public Response(long epoch, Topology topology)
long epoch = message.payload.epoch;
Topology topology = AccordService.instance().topology().maybeGlobalForEpoch(epoch);
if (topology == null)
- MessagingService.instance().respond(Response.UNKNOWN, message);
+ MessagingService.instance().respond(Response.unkonwn(epoch), message);
else
MessagingService.instance().respond(new Response(epoch, topology), message);
};
+ private static final Logger logger = LoggerFactory.getLogger(FetchTopology.class);
Review Comment:
Removed as unused.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-20245: Fix problems and race conditions with topology fetching [cassandra]
Posted by "ifesdjeen (via GitHub)" <gi...@apache.org>.
ifesdjeen commented on code in PR #3842:
URL: https://github.com/apache/cassandra/pull/3842#discussion_r1934400646
##########
src/java/org/apache/cassandra/service/accord/AccordService.java:
##########
@@ -410,48 +413,67 @@ public synchronized void startup()
}
/**
- * Queries peers to discover min epoch
+ * Queries peers to discover min epoch, and then fetches all topologies between min and current epochs
*/
- private long fetchMinEpoch()
+ private List<Topology> fetchTopologies(long minEpoch, ClusterMetadata metadata) throws ExecutionException, InterruptedException
{
- ClusterMetadata metadata = ClusterMetadata.current();
- Map<InetAddressAndPort, Set<TokenRange>> peers = new HashMap<>();
- for (KeyspaceMetadata keyspace : metadata.schema.getKeyspaces())
+ if (configService.maxEpoch() >= metadata.epoch.getEpoch())
{
- List<TableMetadata> tables = keyspace.tables.stream().filter(TableMetadata::requiresAccordSupport).collect(Collectors.toList());
- if (tables.isEmpty())
- continue;
- DataPlacement current = metadata.placements.get(keyspace.params.replication);
- DataPlacement settled = metadata.writePlacementAllSettled(keyspace);
- Sets.SetView<InetAddressAndPort> alive = Sets.intersection(settled.writes.byEndpoint().keySet(), current.writes.byEndpoint().keySet());
- InetAddressAndPort self = FBUtilities.getBroadcastAddressAndPort();
- settled.writes.forEach((range, group) -> {
- if (group.endpoints().contains(self))
- {
- for (InetAddressAndPort peer : group.endpoints())
- {
- if (peer.equals(self) || !alive.contains(peer)) continue;
- for (TableMetadata table : tables)
- peers.computeIfAbsent(peer, i -> new HashSet<>()).add(AccordTopology.fullRange(table.id));
- }
- }
- });
+ logger.info("Accord epoch {} matches TCM. All topologies are known locally", metadata.epoch);
Review Comment:
I think phrasing here was poor, and was stemming from an older version. I think the most correct version would be
```
if (minEpoch == metadata.epoch.getEpoch())
return Collections.singletonList(AccordTopology.createAccordTopology(metadata));
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-20245: Fix problems and race conditions with topology fetching [cassandra]
Posted by "ifesdjeen (via GitHub)" <gi...@apache.org>.
ifesdjeen commented on code in PR #3842:
URL: https://github.com/apache/cassandra/pull/3842#discussion_r1934397453
##########
src/java/org/apache/cassandra/service/accord/AccordService.java:
##########
@@ -377,24 +371,33 @@ public synchronized void startup()
node.commandStores().restoreShardStateUnsafe(topology -> configService.reportTopology(topology, true, true));
configService.start();
- long minEpoch = fetchMinEpoch();
- if (minEpoch >= 0)
+ try
{
- for (long epoch = minEpoch; epoch <= metadata.epoch.getEpoch(); epoch++)
- node.configService().fetchTopologyForEpoch(epoch);
+ // Fetch topologies up to current
+ List<Topology> topologies = fetchTopologies(0, metadata);
Review Comment:
Switched to capital L `Long.
##########
src/java/org/apache/cassandra/service/accord/AccordService.java:
##########
@@ -377,24 +371,33 @@ public synchronized void startup()
node.commandStores().restoreShardStateUnsafe(topology -> configService.reportTopology(topology, true, true));
configService.start();
- long minEpoch = fetchMinEpoch();
- if (minEpoch >= 0)
+ try
{
- for (long epoch = minEpoch; epoch <= metadata.epoch.getEpoch(); epoch++)
- node.configService().fetchTopologyForEpoch(epoch);
+ // Fetch topologies up to current
+ List<Topology> topologies = fetchTopologies(0, metadata);
Review Comment:
Switched to capital L `Long`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-20245: Fix problems and race conditions with topology fetching [cassandra]
Posted by "ifesdjeen (via GitHub)" <gi...@apache.org>.
ifesdjeen commented on code in PR #3842:
URL: https://github.com/apache/cassandra/pull/3842#discussion_r1934395260
##########
src/java/org/apache/cassandra/service/accord/AccordConfigurationService.java:
##########
@@ -407,7 +407,11 @@ void maybeReportMetadata(ClusterMetadata metadata)
long epoch = metadata.epoch.getEpoch();
synchronized (epochs)
{
- if (epochs.maxEpoch() == 0)
+ long maxEpoch = epochs.maxEpoch();
+ if (maxEpoch >= epoch)
Review Comment:
Good point.
I think this was also an edge-case. If I remember it correctly: if you're a second node to be brought up, querying other nodes for min epochs will yield epoch 4, which you will instantiate with a fetched topology.
But then if you race with a TCM callback, you will create epoch - 1 epoch, which no-one has ever heard about.
I should have fixed it differently, maybe this way:
```
// Create a -1 epoch iif we know this epoch may actually exist
if (metadata.epoch.getEpoch() > minEpoch())
getOrCreateEpochState(epoch - 1).acknowledged().addCallback(() -> reportMetadata(metadata));
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-20245: Fix problems and race conditions with topology fetching [cassandra]
Posted by "dcapwell (via GitHub)" <gi...@apache.org>.
dcapwell commented on code in PR #3842:
URL: https://github.com/apache/cassandra/pull/3842#discussion_r1936381336
##########
src/java/org/apache/cassandra/service/accord/AccordService.java:
##########
@@ -377,24 +371,33 @@ public synchronized void startup()
node.commandStores().restoreShardStateUnsafe(topology -> configService.reportTopology(topology, true, true));
configService.start();
- long minEpoch = fetchMinEpoch();
- if (minEpoch >= 0)
+ try
{
- for (long epoch = minEpoch; epoch <= metadata.epoch.getEpoch(); epoch++)
- node.configService().fetchTopologyForEpoch(epoch);
+ // Fetch topologies up to current
+ List<Topology> topologies = fetchTopologies(0, metadata);
+ for (Topology topology : topologies)
+ configService.reportTopology(topology);
- try
- {
- epochReady(metadata.epoch).get(DatabaseDescriptor.getTransactionTimeout(MILLISECONDS), MILLISECONDS);
- }
- catch (InterruptedException e)
- {
- throw new UncheckedInterruptedException(e);
- }
- catch (ExecutionException | TimeoutException e)
+ ClusterMetadataService.instance().log().addListener(configService.listener);
+ ClusterMetadata next = ClusterMetadata.current();
+
+ // if metadata was updated before we were able to add a listener, fetch remaining topologies
+ if (metadata != next)
Review Comment:
I can't speak for TCM, but in Accord this pattern has caused bugs... refactors break assumptions, and you have no clean way to find all the places where your refactor would break things... we had a bug a long time ago when we would do `if (cmd.txnId() == txnId)` and had to switch to `if (cmd.txnId().equals(txnId))`... a refactor broke the assumption that the objects won't be recreated, which then caused that condition to be `false` and do the wrong thing...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-20245: Fix problems and race conditions with topology fetching [cassandra]
Posted by "dcapwell (via GitHub)" <gi...@apache.org>.
dcapwell commented on code in PR #3842:
URL: https://github.com/apache/cassandra/pull/3842#discussion_r1936372190
##########
src/java/org/apache/cassandra/service/accord/FetchTopology.java:
##########
@@ -123,18 +116,20 @@ public Response(long epoch, Topology topology)
long epoch = message.payload.epoch;
Topology topology = AccordService.instance().topology().maybeGlobalForEpoch(epoch);
if (topology == null)
- MessagingService.instance().respond(Response.UNKNOWN, message);
+ MessagingService.instance().respond(Response.unkonwn(epoch), message);
Review Comment:
I don't feel this is a good idea.... `UNKNOWN` can also mean NPE... its truly unknown whats going on and isn't a good answer to "do you know about this epoch". For example, if the epoch isn't known there isn't a reason to retry on that node... but with `UNKNOWN` error we then will retry
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-20245: Fix problems and race conditions with topology fetching [cassandra]
Posted by "ifesdjeen (via GitHub)" <gi...@apache.org>.
ifesdjeen commented on PR #3842:
URL: https://github.com/apache/cassandra/pull/3842#issuecomment-2628458248
Pushed a new version of the change @dcapwell is referring to, and added a bit more motivation for the change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-20245: Fix problems and race conditions with topology fetching [cassandra]
Posted by "dcapwell (via GitHub)" <gi...@apache.org>.
dcapwell commented on code in PR #3842:
URL: https://github.com/apache/cassandra/pull/3842#discussion_r1934190635
##########
src/java/org/apache/cassandra/service/accord/FetchTopology.java:
##########
@@ -37,6 +41,7 @@
public class FetchTopology
{
+ private static final Logger log = LoggerFactory.getLogger(FetchTopology.class);
Review Comment:
nit, any reason not to use `logger`?
```
$ grep -r 'Logger log' src/ | awk -F: '{print $2}' | awk -F= '{print $1}' | sort | uniq -c | sort -k1,1 -h
1 Logger logBackLogger
1 Logger logbackLogger
1 private static Logger logger
1 public static void logInitializationOutcome(Logger logger)
1 private boolean hasAppenders(Logger logBackLogger)
1 private static Logger logger()
1 private static final Logger logger;
1 private static final org.slf4j.Logger logger
1 protected static final NoSpamLogger logger
1 public static <K, V> void dumpDiff(Logger logger, Map<K, V> l, Map<K, V> r)
1 public static Logger logger
1 public static NoSpamLogStatement getStatement(Logger logger, String message, long minInterval, TimeUnit unit)
1 public static NoSpamLogger getLogger(Logger logger, long minInterval, TimeUnit unit)
1 public static boolean log(Logger logger, Level level, String key, long minInterval, TimeUnit unit, String message, Object... objects)
1 public static boolean log(Logger logger, Level level, String key, long minInterval, TimeUnit unit, String message, Supplier<Object[]> objects)
1 public static boolean log(Logger logger, Level level, String key, long minInterval, TimeUnit unit, long nowNanos, String message, Object... objects)
1 public static boolean log(Logger logger, Level level, String key, long minInterval, TimeUnit unit, long nowNanos, String message, Supplier<Object[]> objects)
1 public static boolean log(Logger logger, Level level, long minInterval, TimeUnit unit, String message, Object... objects)
1 public static boolean log(Logger logger, Level level, long minInterval, TimeUnit unit, String message, Supplier<Object[]> objects)
1 public static void dumpDiff(Logger logger, Map<ReplicationParams, DataPlacement> l, Map<ReplicationParams, DataPlacement> r)
1 public static void logSystemInfo(Logger logger)
1 public void log(Logger log)
1 public void logRowCountPerLeaf(Logger logger)
1 public void logRowSizePerLeaf(Logger logger)
2 protected final static Logger logger
3 for (Logger logBackLogger
3 private final Logger logger
3 private static Logger logger
6 static final Logger logger
7 Logger logger
11 private final static Logger logger
19 private static final Logger logger
26 protected static final Logger logger
464 private static final Logger logger
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-20245: Fix problems and race conditions with topology fetching [cassandra]
Posted by "ifesdjeen (via GitHub)" <gi...@apache.org>.
ifesdjeen commented on code in PR #3842:
URL: https://github.com/apache/cassandra/pull/3842#discussion_r1934328966
##########
src/java/org/apache/cassandra/service/accord/AccordConfigurationService.java:
##########
@@ -420,14 +424,17 @@ void maybeReportMetadata(ClusterMetadata metadata)
@Override
protected void fetchTopologyInternal(long epoch)
{
+ if (ClusterMetadata.current().epoch.getEpoch() < epoch)
Review Comment:
Yes, sorry: was waiting for CI on a rebase before pushing it here. Pushed now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-20245: Fix problems and race conditions with topology fetching [cassandra]
Posted by "ifesdjeen (via GitHub)" <gi...@apache.org>.
ifesdjeen commented on code in PR #3842:
URL: https://github.com/apache/cassandra/pull/3842#discussion_r1934397123
##########
src/java/org/apache/cassandra/service/accord/AccordService.java:
##########
@@ -410,48 +413,67 @@ public synchronized void startup()
}
/**
- * Queries peers to discover min epoch
+ * Queries peers to discover min epoch, and then fetches all topologies between min and current epochs
*/
- private long fetchMinEpoch()
+ private List<Topology> fetchTopologies(long minEpoch, ClusterMetadata metadata) throws ExecutionException, InterruptedException
{
- ClusterMetadata metadata = ClusterMetadata.current();
- Map<InetAddressAndPort, Set<TokenRange>> peers = new HashMap<>();
- for (KeyspaceMetadata keyspace : metadata.schema.getKeyspaces())
+ if (configService.maxEpoch() >= metadata.epoch.getEpoch())
{
- List<TableMetadata> tables = keyspace.tables.stream().filter(TableMetadata::requiresAccordSupport).collect(Collectors.toList());
- if (tables.isEmpty())
- continue;
- DataPlacement current = metadata.placements.get(keyspace.params.replication);
- DataPlacement settled = metadata.writePlacementAllSettled(keyspace);
- Sets.SetView<InetAddressAndPort> alive = Sets.intersection(settled.writes.byEndpoint().keySet(), current.writes.byEndpoint().keySet());
- InetAddressAndPort self = FBUtilities.getBroadcastAddressAndPort();
- settled.writes.forEach((range, group) -> {
- if (group.endpoints().contains(self))
- {
- for (InetAddressAndPort peer : group.endpoints())
- {
- if (peer.equals(self) || !alive.contains(peer)) continue;
- for (TableMetadata table : tables)
- peers.computeIfAbsent(peer, i -> new HashSet<>()).add(AccordTopology.fullRange(table.id));
- }
- }
- });
+ logger.info("Accord epoch {} matches TCM. All topologies are known locally", metadata.epoch);
+ return Collections.singletonList(AccordTopology.createAccordTopology(metadata));
}
+
+ Set<InetAddressAndPort> peers = new HashSet<>();
+ peers.addAll(metadata.directory.allAddresses());
+ peers.remove(FBUtilities.getBroadcastAddressAndPort());
+
+ // No peers: single node cluster or first node to boot
if (peers.isEmpty())
- return -1;
+ return Collections.singletonList(AccordTopology.createAccordTopology(metadata));;
+
+ // Bootstrap, fetch min epoch
+ if (minEpoch == 0)
Review Comment:
Switched to capital L `Long`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-20245: Fix problems and race conditions with topology fetching [cassandra]
Posted by "ifesdjeen (via GitHub)" <gi...@apache.org>.
ifesdjeen closed pull request #3842: CASSANDRA-20245: Fix problems and race conditions with topology fetching
URL: https://github.com/apache/cassandra/pull/3842
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org