You are viewing a plain text version of this content. The canonical link for it is here.
Posted to pr@cassandra.apache.org by "rhuffy (via GitHub)" <gi...@apache.org> on 2024/04/23 18:56:44 UTC
[PR] [CASSANDRA-15439] Add option to override the FatClient timeout for Bootstrapping nodes [cassandra]
rhuffy opened a new pull request, #3270:
URL: https://github.com/apache/cassandra/pull/3270
This implements the suggestion by @driftx in [CASSANDRA-15439](https://issues.apache.org/jira/browse/CASSANDRA-15439). See https://the-asf.slack.com/archives/CK23JSY2K/p1713820180414349 for further discussion.
If a bootstrapping node experiences a GC pause such that it fails to gossip for 30s, it may be removed from another node's Gossip state, as it has exceeded the FatClient timeout, which by default is equal to RING_DELAY=30s.
This violates Cassandra's consistency guarantees. When there are pending ranges, the number of nodes that must ACK a write at quorum is increased. For example, with RF=3, quorum is 2. If there are pending ranges, quorum is 3.
For operators who are particularly concerned with the durability of writes during expansion, this option can be set to a higher value - ideally longer than the longest expected GC pause. If this value is increased significantly, like to multiple hours, or set to -1, operators will need to take manual action when a bootstrap has failed, like assassinating the failed node.
Example usage:
```
-Dcassandra.bootstrapping_fat_client_timeout_ms=300000
```
The [Cassandra Jira](https://issues.apache.org/jira/projects/CASSANDRA/issues/)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] [CASSANDRA-15439] Add option to override the FatClient timeout for Bootstrapping nodes [cassandra]
Posted by "dcapwell (via GitHub)" <gi...@apache.org>.
dcapwell commented on code in PR #3270:
URL: https://github.com/apache/cassandra/pull/3270#discussion_r1592834109
##########
src/java/org/apache/cassandra/gms/Gossiper.java:
##########
@@ -272,6 +278,25 @@ private static long getVeryLongTime()
return 259200 * 1000; // 3 days
}
+ private static long getFailedBootstrapTimeout()
+ {
+ String newtimeout = CassandraRelevantProperties.FAILED_BOOTSTRAP_TIMEOUT.getString();
+ if (newtimeout != null)
+ {
+ long longValue = Long.parseLong(newtimeout);
Review Comment:
why not just call `getLong(-1)`? These properties expose the parsing in a generic way
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] [CASSANDRA-15439] Add option to override the FatClient timeout for Bootstrapping nodes [cassandra]
Posted by "rhuffy (via GitHub)" <gi...@apache.org>.
rhuffy commented on code in PR #3270:
URL: https://github.com/apache/cassandra/pull/3270#discussion_r1592978577
##########
src/java/org/apache/cassandra/gms/Gossiper.java:
##########
@@ -123,6 +123,9 @@ public static class Props
SILENT_SHUTDOWN_STATES.add(VersionedValue.STATUS_BOOTSTRAPPING);
SILENT_SHUTDOWN_STATES.add(VersionedValue.STATUS_BOOTSTRAPPING_REPLACE);
}
+
+ static final List<String> BOOTSTRAPPING_STATES = Arrays.asList(VersionedValue.STATUS_BOOTSTRAPPING,
Review Comment:
👍 That was added in 5.0. I can add it to this patch to minimize the diff between versions
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] [CASSANDRA-15439] Add option to override the FatClient timeout for Bootstrapping nodes [cassandra]
Posted by "driftx (via GitHub)" <gi...@apache.org>.
driftx commented on code in PR #3270:
URL: https://github.com/apache/cassandra/pull/3270#discussion_r1576853341
##########
src/java/org/apache/cassandra/config/CassandraRelevantProperties.java:
##########
@@ -155,6 +155,8 @@ public enum CassandraRelevantProperties
RING_DELAY("cassandra.ring_delay_ms"),
+ BOOTSTRAPPING_FAT_CLIENT_TIMEOUT("cassandra.bootstrapping_fat_client_timeout_ms"),
+
Review Comment:
I think we should name this something like FAILED_BOOTSTRAP_TIMEOUT, I don't want to expose 'fat client' to users since that is not immediately understandable and we have moved to calling them 'gossip only clients'
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] [CASSANDRA-15439] Add option to override the FatClient timeout for Bootstrapping nodes [cassandra]
Posted by "smiklosovic (via GitHub)" <gi...@apache.org>.
smiklosovic closed pull request #3270: [CASSANDRA-15439] Add option to override the FatClient timeout for Bootstrapping nodes
URL: https://github.com/apache/cassandra/pull/3270
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] [CASSANDRA-15439] Add option to override the FatClient timeout for Bootstrapping nodes [cassandra]
Posted by "dcapwell (via GitHub)" <gi...@apache.org>.
dcapwell commented on code in PR #3270:
URL: https://github.com/apache/cassandra/pull/3270#discussion_r1592828504
##########
src/java/org/apache/cassandra/gms/Gossiper.java:
##########
@@ -123,6 +123,9 @@ public static class Props
SILENT_SHUTDOWN_STATES.add(VersionedValue.STATUS_BOOTSTRAPPING);
SILENT_SHUTDOWN_STATES.add(VersionedValue.STATUS_BOOTSTRAPPING_REPLACE);
}
+
+ static final List<String> BOOTSTRAPPING_STATES = Arrays.asList(VersionedValue.STATUS_BOOTSTRAPPING,
Review Comment:
This duplicates `org.apache.cassandra.gms.VersionedValue#BOOTSTRAPPING_STATUS`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] [CASSANDRA-15439] Add option to override the FatClient timeout for Bootstrapping nodes [cassandra]
Posted by "dcapwell (via GitHub)" <gi...@apache.org>.
dcapwell commented on code in PR #3270:
URL: https://github.com/apache/cassandra/pull/3270#discussion_r1592834109
##########
src/java/org/apache/cassandra/gms/Gossiper.java:
##########
@@ -272,6 +278,25 @@ private static long getVeryLongTime()
return 259200 * 1000; // 3 days
}
+ private static long getFailedBootstrapTimeout()
+ {
+ String newtimeout = CassandraRelevantProperties.FAILED_BOOTSTRAP_TIMEOUT.getString();
+ if (newtimeout != null)
+ {
+ long longValue = Long.parseLong(newtimeout);
Review Comment:
why not just call `getLong(-1)`? These properties expose the parsing in a generic way
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org