You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@cassandra.apache.org by Radim Vansa <rv...@azul.com> on 2025/03/06 20:34:53 UTC
Cassandra Java Driver and OpenJDK CRaC
Hi all,
I would like to make applications using Cassandra Java Driver,
particularly those built with Spring Boot, Quarkus or similar
frameworks, work with OpenJDK CRaC project [1]. I've already created a
patch for Spring Boot [2] but Spring folks think that these changes are
too dependent on driver internals, suggesting to contribute a support to
Cassandra directly.
The patch involves closing all connections before checkpoint, and
re-establishing these after restore. I have implemented that though
sending a `NodeStateEvent -> FORCED_DOWN` on the bus for all connected
nodes. As a follow-up I could develop some way to inform the session
about a new topology e.g. if the cluster addresses change.
Before jumping onto implementing a PR I would like to ask what you think
is the best approach to do this. I can think of two ways:
1) Native CRaC support
The driver would have a dependency on `org.crac:crac` [3]; this is a
small (13kB) library that provides the interfaces and a dummy noop
implementation if the target JVM does not support CRaC. Then
`DefaultSession` would register a `org.crac.Resource` implementation
that would handle the checkpoint. This has the advantage of providing
best fan-out into any project consuming the driver without any further work.
2) Exposing neutral methods
To save frameworks of relying on internals, `DefaultSession` would
expose `.suspend()` and `.resume()` methods that would implement the
connection cut-off without importing any dependency. After upgrade to
latest release, frameworks could use these methods in a way that suits
them. I wouldn't add those methods to the `CqlSession` interface (as
that would be breaking change) but only to `DefaultSession`.
Would Cassandra accept either of these, to let people checkpoint
(snapshot) their applications and restore them within tens of
milliseconds? Naturally it is possible to close the session object
completely and create a new one, but the ideal solution would require no
application changes beyond dependency upgrade.
Btw. I am aware that there is an inherent race between possible topology
change and shutdown of current nodes (and I am listening for hints that
would let us prevent that), but it is reasonable to expect that users
will checkpoint the application in a quiescent state. And if the
topology update breaks the checkpoint, it is always possible to try it
again.
Thank you for your opinions and ideas!
Radim Vansa
[1] https://wiki.openjdk.org/display/crac
[2] https://github.com/spring-projects/spring-boot/pull/44505
[3] https://mvnrepository.com/artifact/org.crac/crac/1.5.0
Re: Cassandra Java Driver and OpenJDK CRaC
Posted by Radim Vansa <rv...@azul.com>.
Hi Patrick,
> attacking some of the same requirements that Graal and Quarkus are
trying to solve
thanks for the support! Yes, Graal (and Leyden) are kind of competing
solutions for the startup problem. We're trying to hit the sweet spot
between not requiring significant redesign (as is sometimes the case
with Graal AOT) and having more bang than what a fully transparent
solution can give. Quarkus is also known for fast startup but it is more
orthogonal to CRaC - in fact Quarkus already has some support for CRaC
and the two can be aligned to get even better performance together.
> Topology information shouldn't be assumed
Is there already an automatic process that will update the topology
information on reconnect? I guess that what we should prevent is the
'manual' update (forcing node back up) to override fresh topology
update. Also, if there's a process invoking the driver concurrently to
checkpoint, we might get the control connection established too early;
that's not a big problem since the checkpoint will fail and we can retry.
Thank you!
Radim
On 10. 03. 25 18:17, Patrick McFadin wrote:
>
>
> Caution: This email originated from outside of the organization. Do
> not click links or open attachments unless you recognize the sender
> and know the content is safe.
>
>
> Just speaking up as a supporter for considering this change. From a
> userland perspective, I've been reading up on CRaC, and I see this
> attacking some of the same requirements that Graal and Quarkus are
> trying to solve. This is a worth direction to pursue.
>
> The CqlSession will need to re-connect, and I think that's worth
> testing. Topology information shouldn't be assumed, especially with
> something like Token-Aware Routing. Some shortcuts could speed it up,
> but I can't think of any right now. I like the idea of making it
> optional and putting it through some scenarios.
>
> Patrick
>
> On Mon, Mar 10, 2025 at 8:03 AM Radim Vansa <rv...@azul.com> wrote:
>
> Hello Josh,
>
> thanks for reaching back; answers inline:
> On 10. 03. 25 13:03, Josh McKenzie wrote:
>>
>> From skimming the PR on the Spring side and the conversation
>> there, it looks like the argument is to have this live inside the
>> java driver for Cassandra instead of in the spring-boot lib which
>> I can see the argument for.
>
>
> Yes; for us it does not really matter where the fix lives as long
> as it's available for the end users. Pushing it towards Cassandra
> has the advantage to provide the greatest fan-out to users, even
> those not consuming through frameworks.
>
>>
>> If we distill this to speak to precisely the problem we're trying
>> to address or improvement we're going for here, how would you
>> phrase that? i.e. "Take application startup from Nms down to Mms"?
>>
>
> Yes, optimizing startup time is the most common use-case for CRaC.
> It's rather hard to provide such general numbers: it should be
> order(s) of magnitude. If we speak about hello-world style Spring
> Boot application booting, CRaC improves the startup from seconds
> to tens of milliseconds. That shouldn't differ too much from the
> expected times for a small micro-service, improving latency in
> scale-from-zero situations. This is not limited to microservices,
> though; we've been experimenting with real applications consuming
> hundreds of GB of memory. In that case the application boot can be
> rather complex, loading and pre-processing data from DB etc. where
> the boot takes minutes or more. CRaC can restore such instance in
> a few seconds.
>
>>
>> I ask because that's the "pro" we'll need to weigh against
>> updating the driver's topology map of the cluster, resource
>> handling and potential leaks on shutdown/startup, and the
>> complexity of taking an implementation like this into the driver
>> code. Nothing insurmountable of course, just worth weighing the two.
>
> Can you elaborate about other use cases where the nodes are forced
> down, and what risk does that bring to the overall stability? Is
> there a difference between marking only a subset of nodes down and
> taking all of the nodes down? When we force-close the control
> connection (as the first step), is it possible to get a topology
> update at all and race on the cluster members?
>
> Thank you!
>
> Radim
>
>
>>
>> On Thu, Mar 6, 2025, at 3:34 PM, Radim Vansa wrote:
>>> Hi all,
>>>
>>> I would like to make applications using Cassandra Java Driver,
>>> particularly those built with Spring Boot, Quarkus or similar
>>> frameworks, work with OpenJDK CRaC project [1]. I've already
>>> created a
>>> patch for Spring Boot [2] but Spring folks think that these
>>> changes are
>>> too dependent on driver internals, suggesting to contribute a
>>> support to
>>> Cassandra directly.
>>>
>>> The patch involves closing all connections before checkpoint, and
>>> re-establishing these after restore. I have implemented that though
>>> sending a `NodeStateEvent -> FORCED_DOWN` on the bus for all
>>> connected
>>> nodes. As a follow-up I could develop some way to inform the
>>> session
>>> about a new topology e.g. if the cluster addresses change.
>>>
>>> Before jumping onto implementing a PR I would like to ask what
>>> you think
>>> is the best approach to do this. I can think of two ways:
>>>
>>> 1) Native CRaC support
>>>
>>> The driver would have a dependency on `org.crac:crac` [3]; this
>>> is a
>>> small (13kB) library that provides the interfaces and a dummy noop
>>> implementation if the target JVM does not support CRaC. Then
>>> `DefaultSession` would register a `org.crac.Resource`
>>> implementation
>>> that would handle the checkpoint. This has the advantage of
>>> providing
>>> best fan-out into any project consuming the driver without any
>>> further work.
>>>
>>> 2) Exposing neutral methods
>>>
>>> To save frameworks of relying on internals, `DefaultSession` would
>>> expose `.suspend()` and `.resume()` methods that would implement
>>> the
>>> connection cut-off without importing any dependency. After
>>> upgrade to
>>> latest release, frameworks could use these methods in a way that
>>> suits
>>> them. I wouldn't add those methods to the `CqlSession` interface
>>> (as
>>> that would be breaking change) but only to `DefaultSession`.
>>>
>>> Would Cassandra accept either of these, to let people checkpoint
>>> (snapshot) their applications and restore them within tens of
>>> milliseconds? Naturally it is possible to close the session object
>>> completely and create a new one, but the ideal solution would
>>> require no
>>> application changes beyond dependency upgrade.
>>>
>>> Btw. I am aware that there is an inherent race between possible
>>> topology
>>> change and shutdown of current nodes (and I am listening for
>>> hints that
>>> would let us prevent that), but it is reasonable to expect that
>>> users
>>> will checkpoint the application in a quiescent state. And if the
>>> topology update breaks the checkpoint, it is always possible to
>>> try it
>>> again.
>>>
>>> Thank you for your opinions and ideas!
>>>
>>> Radim Vansa
>>>
>>>
>>> [1] https://wiki.openjdk.org/display/crac
>>>
>>> [2] https://github.com/spring-projects/spring-boot/pull/44505
>>>
>>> [3] https://mvnrepository.com/artifact/org.crac/crac/1.5.0
>>>
>>>
>>
Re: Cassandra Java Driver and OpenJDK CRaC
Posted by Patrick McFadin <pm...@gmail.com>.
Just speaking up as a supporter for considering this change. From a
userland perspective, I've been reading up on CRaC, and I see this
attacking some of the same requirements that Graal and Quarkus are trying
to solve. This is a worth direction to pursue.
The CqlSession will need to re-connect, and I think that's worth testing.
Topology information shouldn't be assumed, especially with something like
Token-Aware Routing. Some shortcuts could speed it up, but I can't think of
any right now. I like the idea of making it optional and putting it through
some scenarios.
Patrick
On Mon, Mar 10, 2025 at 8:03 AM Radim Vansa <rv...@azul.com> wrote:
> Hello Josh,
> thanks for reaching back; answers inline:
>
> On 10. 03. 25 13:03, Josh McKenzie wrote:
>
>
> From skimming the PR on the Spring side and the conversation there, it
> looks like the argument is to have this live inside the java driver for
> Cassandra instead of in the spring-boot lib which I can see the argument
> for.
>
>
> Yes; for us it does not really matter where the fix lives as long as it's
> available for the end users. Pushing it towards Cassandra has the advantage
> to provide the greatest fan-out to users, even those not consuming through
> frameworks.
>
>
> If we distill this to speak to precisely the problem we're trying to
> address or improvement we're going for here, how would you phrase that?
> i.e. "Take application startup from Nms down to Mms"?
>
>
> Yes, optimizing startup time is the most common use-case for CRaC. It's
> rather hard to provide such general numbers: it should be order(s) of
> magnitude. If we speak about hello-world style Spring Boot application
> booting, CRaC improves the startup from seconds to tens of milliseconds.
> That shouldn't differ too much from the expected times for a small
> micro-service, improving latency in scale-from-zero situations. This is not
> limited to microservices, though; we've been experimenting with real
> applications consuming hundreds of GB of memory. In that case the
> application boot can be rather complex, loading and pre-processing data
> from DB etc. where the boot takes minutes or more. CRaC can restore such
> instance in a few seconds.
>
>
> I ask because that's the "pro" we'll need to weigh against updating the
> driver's topology map of the cluster, resource handling and potential leaks
> on shutdown/startup, and the complexity of taking an implementation like
> this into the driver code. Nothing insurmountable of course, just worth
> weighing the two.
>
>
> Can you elaborate about other use cases where the nodes are forced down,
> and what risk does that bring to the overall stability? Is there a
> difference between marking only a subset of nodes down and taking all of
> the nodes down? When we force-close the control connection (as the first
> step), is it possible to get a topology update at all and race on the
> cluster members?
>
> Thank you!
>
> Radim
>
>
>
> On Thu, Mar 6, 2025, at 3:34 PM, Radim Vansa wrote:
>
> Hi all,
>
> I would like to make applications using Cassandra Java Driver,
> particularly those built with Spring Boot, Quarkus or similar
> frameworks, work with OpenJDK CRaC project [1]. I've already created a
> patch for Spring Boot [2] but Spring folks think that these changes are
> too dependent on driver internals, suggesting to contribute a support to
> Cassandra directly.
>
> The patch involves closing all connections before checkpoint, and
> re-establishing these after restore. I have implemented that though
> sending a `NodeStateEvent -> FORCED_DOWN` on the bus for all connected
> nodes. As a follow-up I could develop some way to inform the session
> about a new topology e.g. if the cluster addresses change.
>
> Before jumping onto implementing a PR I would like to ask what you think
> is the best approach to do this. I can think of two ways:
>
> 1) Native CRaC support
>
> The driver would have a dependency on `org.crac:crac` [3]; this is a
> small (13kB) library that provides the interfaces and a dummy noop
> implementation if the target JVM does not support CRaC. Then
> `DefaultSession` would register a `org.crac.Resource` implementation
> that would handle the checkpoint. This has the advantage of providing
> best fan-out into any project consuming the driver without any further
> work.
>
> 2) Exposing neutral methods
>
> To save frameworks of relying on internals, `DefaultSession` would
> expose `.suspend()` and `.resume()` methods that would implement the
> connection cut-off without importing any dependency. After upgrade to
> latest release, frameworks could use these methods in a way that suits
> them. I wouldn't add those methods to the `CqlSession` interface (as
> that would be breaking change) but only to `DefaultSession`.
>
> Would Cassandra accept either of these, to let people checkpoint
> (snapshot) their applications and restore them within tens of
> milliseconds? Naturally it is possible to close the session object
> completely and create a new one, but the ideal solution would require no
> application changes beyond dependency upgrade.
>
> Btw. I am aware that there is an inherent race between possible topology
> change and shutdown of current nodes (and I am listening for hints that
> would let us prevent that), but it is reasonable to expect that users
> will checkpoint the application in a quiescent state. And if the
> topology update breaks the checkpoint, it is always possible to try it
> again.
>
> Thank you for your opinions and ideas!
>
> Radim Vansa
>
>
> [1] https://wiki.openjdk.org/display/crac
>
> [2] https://github.com/spring-projects/spring-boot/pull/44505
>
> [3] https://mvnrepository.com/artifact/org.crac/crac/1.5.0
>
>
>
>
Re: Cassandra Java Driver and OpenJDK CRaC
Posted by Radim Vansa <rv...@azul.com>.
Hi Abe,
I would expect that if the control connection is terminated, on the next
request it would be re-established including the handshake you mention.
This shouldn't be different from a connection being broken due to
network error. In the POC PR I am calling just `DriverChannel.close()`,
though as I see this sends a graceful termination to the wire as well.
> Would it be possible to have a separate CqlSession implementation
that includes CRaC's checkpoint and restore hooks[2] to close and open
the session at the appropriate times
That is an option, though how convenient would that be for use? In
Spring Boot case, we would could add Spring Boot-specific module that
would 'override' the type that should be used, with a dependency on a
'generic' module implementing the session. I think that this could be
transparent enough, but if these are both managed as third-party module,
it would be almost easier to just keep the `CassandraSessionLifecycle`
managing the connections 'from the outside'. The fragility problem
remains (modifications that break the third-party module are not
observed immediately, e.g. in a testsuite). The advantage is in being
able to eventually take the unmodified class into the driver.
For non-framework use-cases, I think that this is a bigger issue: since
CqlSession.builder().withWhatever(...).build() will always return the
`DefaultSession` the code would have to be modified to use a different
builder. There's no way to 'configure' this (reflection, service
loader...); the TPM would have a `CracableSessionBuilder` with a
non-trivial override of `SessionBuilder.buildDefaultSessionAsync` and
the application would have to change its code to use it.
Thanks for your thoughts!
Radim
[1]
https://github.com/spring-projects/spring-boot/pull/44505/files#diff-780d2cdd9f860039f0a5a198303af2fe04ea05991ec15e63b09d3f094e3ea8a9R92
On 10. 03. 25 17:28, Abe Ratnofsky wrote:
>
>
> Caution: This email originated from outside of the organization. Do
> not click links or open attachments unless you recognize the sender
> and know the content is safe.
>
>
> Hey Radim, thanks for bringing this to the list.
>
> In general, I'm supportive of the second option you shared ("Exposing
> neutral methods") but want to make sure I understand how CRaC would
> work in practice.
>
> Could you clarify this part:
>
> > Naturally it is possible to close the session object completely and
> create a new one, but the ideal solution would require no application
> changes beyond dependency upgrade.
>
> CRaC doesn't support checkpointing with open sockets, and the
> Cassandra client protocol requires a few roundtrips after connection
> establishment before a session can be used[1]. Would it be possible to
> have a separate CqlSession implementation that includes CRaC's
> checkpoint and restore hooks[2] to close and open the session at the
> appropriate times? This CRaC implementation could live as a
> third-party module initially as it is proven out.
>
> [1]:
> https://github.com/apache/cassandra/blob/trunk/doc/native_protocol_v5.spec#L176
> [2]:
> https://docs.azul.com/core/crac/crac-guidelines#implementing-crac-resource
>
Re: Cassandra Java Driver and OpenJDK CRaC
Posted by Abe Ratnofsky <ab...@aber.io>.
Hey Radim, thanks for bringing this to the list.
In general, I'm supportive of the second option you shared ("Exposing neutral methods") but want to make sure I understand how CRaC would work in practice.
Could you clarify this part:
> Naturally it is possible to close the session object completely and create a new one, but the ideal solution would require no application changes beyond dependency upgrade.
CRaC doesn't support checkpointing with open sockets, and the Cassandra client protocol requires a few roundtrips after connection establishment before a session can be used[1]. Would it be possible to have a separate CqlSession implementation that includes CRaC's checkpoint and restore hooks[2] to close and open the session at the appropriate times? This CRaC implementation could live as a third-party module initially as it is proven out.
[1]: https://github.com/apache/cassandra/blob/trunk/doc/native_protocol_v5.spec#L176
[2]: https://docs.azul.com/core/crac/crac-guidelines#implementing-crac-resource
Re: Cassandra Java Driver and OpenJDK CRaC
Posted by Radim Vansa <rv...@azul.com>.
Hello Josh,
thanks for reaching back; answers inline:
On 10. 03. 25 13:03, Josh McKenzie wrote:
>
> From skimming the PR on the Spring side and the conversation there, it
> looks like the argument is to have this live inside the java driver
> for Cassandra instead of in the spring-boot lib which I can see the
> argument for.
Yes; for us it does not really matter where the fix lives as long as
it's available for the end users. Pushing it towards Cassandra has the
advantage to provide the greatest fan-out to users, even those not
consuming through frameworks.
>
> If we distill this to speak to precisely the problem we're trying to
> address or improvement we're going for here, how would you phrase
> that? i.e. "Take application startup from Nms down to Mms"?
>
Yes, optimizing startup time is the most common use-case for CRaC. It's
rather hard to provide such general numbers: it should be order(s) of
magnitude. If we speak about hello-world style Spring Boot application
booting, CRaC improves the startup from seconds to tens of milliseconds.
That shouldn't differ too much from the expected times for a small
micro-service, improving latency in scale-from-zero situations. This is
not limited to microservices, though; we've been experimenting with real
applications consuming hundreds of GB of memory. In that case the
application boot can be rather complex, loading and pre-processing data
from DB etc. where the boot takes minutes or more. CRaC can restore such
instance in a few seconds.
>
> I ask because that's the "pro" we'll need to weigh against updating
> the driver's topology map of the cluster, resource handling and
> potential leaks on shutdown/startup, and the complexity of taking an
> implementation like this into the driver code. Nothing insurmountable
> of course, just worth weighing the two.
Can you elaborate about other use cases where the nodes are forced down,
and what risk does that bring to the overall stability? Is there a
difference between marking only a subset of nodes down and taking all of
the nodes down? When we force-close the control connection (as the first
step), is it possible to get a topology update at all and race on the
cluster members?
Thank you!
Radim
>
> On Thu, Mar 6, 2025, at 3:34 PM, Radim Vansa wrote:
>> Hi all,
>>
>> I would like to make applications using Cassandra Java Driver,
>> particularly those built with Spring Boot, Quarkus or similar
>> frameworks, work with OpenJDK CRaC project [1]. I've already created a
>> patch for Spring Boot [2] but Spring folks think that these changes are
>> too dependent on driver internals, suggesting to contribute a support to
>> Cassandra directly.
>>
>> The patch involves closing all connections before checkpoint, and
>> re-establishing these after restore. I have implemented that though
>> sending a `NodeStateEvent -> FORCED_DOWN` on the bus for all connected
>> nodes. As a follow-up I could develop some way to inform the session
>> about a new topology e.g. if the cluster addresses change.
>>
>> Before jumping onto implementing a PR I would like to ask what you think
>> is the best approach to do this. I can think of two ways:
>>
>> 1) Native CRaC support
>>
>> The driver would have a dependency on `org.crac:crac` [3]; this is a
>> small (13kB) library that provides the interfaces and a dummy noop
>> implementation if the target JVM does not support CRaC. Then
>> `DefaultSession` would register a `org.crac.Resource` implementation
>> that would handle the checkpoint. This has the advantage of providing
>> best fan-out into any project consuming the driver without any
>> further work.
>>
>> 2) Exposing neutral methods
>>
>> To save frameworks of relying on internals, `DefaultSession` would
>> expose `.suspend()` and `.resume()` methods that would implement the
>> connection cut-off without importing any dependency. After upgrade to
>> latest release, frameworks could use these methods in a way that suits
>> them. I wouldn't add those methods to the `CqlSession` interface (as
>> that would be breaking change) but only to `DefaultSession`.
>>
>> Would Cassandra accept either of these, to let people checkpoint
>> (snapshot) their applications and restore them within tens of
>> milliseconds? Naturally it is possible to close the session object
>> completely and create a new one, but the ideal solution would require no
>> application changes beyond dependency upgrade.
>>
>> Btw. I am aware that there is an inherent race between possible topology
>> change and shutdown of current nodes (and I am listening for hints that
>> would let us prevent that), but it is reasonable to expect that users
>> will checkpoint the application in a quiescent state. And if the
>> topology update breaks the checkpoint, it is always possible to try it
>> again.
>>
>> Thank you for your opinions and ideas!
>>
>> Radim Vansa
>>
>>
>> [1] https://wiki.openjdk.org/display/crac
>>
>> [2] https://github.com/spring-projects/spring-boot/pull/44505
>>
>> [3] https://mvnrepository.com/artifact/org.crac/crac/1.5.0
>>
>>
>
Re: Cassandra Java Driver and OpenJDK CRaC
Posted by Josh McKenzie <jm...@apache.org>.
Thanks for reaching out to the list Radim; this is interesting stuff.
From skimming the PR on the Spring side and the conversation there, it looks like the argument is to have this live inside the java driver for Cassandra instead of in the spring-boot lib which I can see the argument for.
If we distill this to speak to precisely the problem we're trying to address or improvement we're going for here, how would you phrase that? i.e. "Take application startup from Nms down to Mms"?
I ask because that's the "pro" we'll need to weigh against updating the driver's topology map of the cluster, resource handling and potential leaks on shutdown/startup, and the complexity of taking an implementation like this into the driver code. Nothing insurmountable of course, just worth weighing the two.
On Thu, Mar 6, 2025, at 3:34 PM, Radim Vansa wrote:
> Hi all,
>
> I would like to make applications using Cassandra Java Driver,
> particularly those built with Spring Boot, Quarkus or similar
> frameworks, work with OpenJDK CRaC project [1]. I've already created a
> patch for Spring Boot [2] but Spring folks think that these changes are
> too dependent on driver internals, suggesting to contribute a support to
> Cassandra directly.
>
> The patch involves closing all connections before checkpoint, and
> re-establishing these after restore. I have implemented that though
> sending a `NodeStateEvent -> FORCED_DOWN` on the bus for all connected
> nodes. As a follow-up I could develop some way to inform the session
> about a new topology e.g. if the cluster addresses change.
>
> Before jumping onto implementing a PR I would like to ask what you think
> is the best approach to do this. I can think of two ways:
>
> 1) Native CRaC support
>
> The driver would have a dependency on `org.crac:crac` [3]; this is a
> small (13kB) library that provides the interfaces and a dummy noop
> implementation if the target JVM does not support CRaC. Then
> `DefaultSession` would register a `org.crac.Resource` implementation
> that would handle the checkpoint. This has the advantage of providing
> best fan-out into any project consuming the driver without any further work.
>
> 2) Exposing neutral methods
>
> To save frameworks of relying on internals, `DefaultSession` would
> expose `.suspend()` and `.resume()` methods that would implement the
> connection cut-off without importing any dependency. After upgrade to
> latest release, frameworks could use these methods in a way that suits
> them. I wouldn't add those methods to the `CqlSession` interface (as
> that would be breaking change) but only to `DefaultSession`.
>
> Would Cassandra accept either of these, to let people checkpoint
> (snapshot) their applications and restore them within tens of
> milliseconds? Naturally it is possible to close the session object
> completely and create a new one, but the ideal solution would require no
> application changes beyond dependency upgrade.
>
> Btw. I am aware that there is an inherent race between possible topology
> change and shutdown of current nodes (and I am listening for hints that
> would let us prevent that), but it is reasonable to expect that users
> will checkpoint the application in a quiescent state. And if the
> topology update breaks the checkpoint, it is always possible to try it
> again.
>
> Thank you for your opinions and ideas!
>
> Radim Vansa
>
>
> [1] https://wiki.openjdk.org/display/crac
>
> [2] https://github.com/spring-projects/spring-boot/pull/44505
>
> [3] https://mvnrepository.com/artifact/org.crac/crac/1.5.0
>
>