[DISCUSS] CEP 53: Cassandra Rolling Restarts via Sidecar

[Andrés Beck-Ruiz <an...@gmail.com>]

Hello everyone,

We would like to propose CEP 53: Cassandra Rolling Restarts via Sidecar (
https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-53%3A+Cassandra+Rolling+Restarts+via+Sidecar
)

This CEP builds off of CEP-1
<https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-1%3A+Apache+Cassandra+Management+Process%28es%29+-+Deprecated>
and proposes a design for safe, efficient, and operator friendly rolling
restarts on Cassandra clusters, as well as an extensible approach for
persisting future cluster-wide operations in Cassandra Sidecar. We hope to
leverage this infrastructure in the future to implement upgrade automation.

We welcome all feedback and discussion. Thank you in advance for your time
and consideration of this proposal!

Best,
Andrés and Paulo

Re: [DISCUSS] CEP 53: Cassandra Rolling Restarts via Sidecar

[Štefan Miklošovič <sm...@apache.org>]

Yes, I agree with you, Isaac. We might look at the restart operation (or
any other operation which can be done cluster-wide, really), as it is an
operation which is performed on a subset of nodes. A subset containing one
node is still a subset of one node. The number of nodes involved should not
matter. It is just by accident that restarting a whole cluster would
require the specification of a subset which is equal to a set of all nodes.

On Wed, Sep 3, 2025 at 6:08 PM Isaac Reath <is...@gmail.com> wrote:

> Thank you for the CEP and getting this discussion started Andrés!
>
> I also think it makes sense to use the term "cluster-wide operations" to
> refer to these types of orchestrated operations. Another option could be
> “multi-node operations” to contrast with “single-node operations” since an
> operation might operate on a subset of nodes in the cluster rather than the
> entire cluster. In addition to CEP-44 and rolling restarts, there are other
> cases such as config changes (
> https://issues.apache.org/jira/browse/CASSSIDECAR-275) and version
> upgrades (https://issues.apache.org/jira/browse/CASSSIDECAR-276) which
> could also build off of these APIs.
>
> On the question of reusing the OperationalJob interface: from an
> implementation perspective, it seems generic enough to reuse for
> cluster-wide operations. However, it needs to be extended to add support
> for the fields we need to support cluster-wide jobs (for example, the nodes
> involved in the operation, parallelism across racks, etc.)
>
> As for the API, I think the question that needs to be answered is if it is
> worthwhile to have a distinction between single-node operations and
> cluster-wide operations. For example, if I wanted to restart a single node
> using the API proposed in CEP-53, I could submit a restart job with a
> single node in the “nodes” list. This provides API simplicity at the cost
> of ergonomics. It also means that all inter-sidecar communication would go
> through the proposed cluster_ops_node_state table. Personally, I think
> these are acceptable tradeoffs to provide a unified API for operations that
> is simpler for a user or operator to use and learn.
> Isaac
>
> On Tue, Sep 2, 2025 at 2:58 PM Andrés Beck-Ruiz <an...@gmail.com>
> wrote:
>
>> Thanks everyone for the feedback. +1 to using the term 'cluster-wide
>> operations'.
>>
>> > The only suggestion I have is to keep in mind the pluggability aspect of
>> > Sidecar. For example, for the Distributed Restart portion of the work,
>> we
>> > should consider making interfaces that would allow us to potentially
>> move
>> > the responsibility of keeping the state outside of Cassandra.
>>
>> Are you referring to tracking the state of a restart job (and
>> cluster-wide operations in general) outside of sidecar_internal Cassandra
>> tables?
>>
>> > What do you think about broadening the scope of the CEP to propose a
>> way (API) to perform bulk operations, and propose the current Rolling
>> restarts as the first implementation for that bulk operations API? I’m
>> proposing this as I see value to reuse this proposal for other bulk
>> operations such as enabling CDC (it requires enabling cdc on cassandra.yml
>> and some other
>> > operations) for better supporting CEP-44.
>>
>> We propose a way to persist and monitor cluster-wide operations in the
>> new sidecar_internal system tables. (
>> https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-53%3A+Cassandra+Rolling+Restarts+via+Sidecar#CEP53:CassandraRollingRestartsviaSidecar-CassandraSidecarSystemTables).
>> I think it would make sense to also generalize the API to apply to
>> cluster-wide operations. I'm curious about any feedback on whether this
>> should be a separate API from the current operational job framework and
>> live under the /cluster resource. We've discussed why we didn't propose to
>> use the existing API and how the current framework would need to be
>> extended here (
>> https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-53%3A+Cassandra+Rolling+Restarts+via+Sidecar#CEP53:CassandraRollingRestartsviaSidecar-OperationalJobFramework
>> ).
>>
>> > I’m not quite sold on using a PATCH to move from pending state to
>> running state. Quick question, what is the goal of the pending state? I see
>> a PATCH operation as modifying part of an object data. In this case,
>> modifying the state looks like a change on the operation state, not on its
>> metadata. I’d love to hear your thoughts on this one.
>>
>> The "PENDING" state allows for an operator to double check a submitted
>> cluster-wide operation, which could have unintended consequences, before
>> starting it. For example, performing a rolling restart could prevent other
>> operations on the cluster that might be scheduled or needed, such as
>> replacing a Cassandra instance. While an operator should be able to abort a
>> restart job, I see value in having this guard against operator error.
>>
>> Given that we are applying a partial update to the resource, which in
>> this context would be the restart job, we chose PATCH for this API.
>>
>> Best,
>> Andrés
>>
>>
>> On Tue, Sep 2, 2025 at 12:33 PM Dinesh Joshi <dj...@apache.org> wrote:
>>
>>> I would like to chime in and say that we need to refine our vocabulary.
>>> The term 'bulk commands' was used originally in CEP-1. This is my fault
>>> totally as I originally wrote that down. But over time it has caused
>>> confusion. I believe 'cluster-wide operations' is a better term to describe
>>> those operations. We have also used 'Bulk' in the context of CEP-28 which
>>> means something rather different which leads to confusion. So I propose
>>> using the term 'cluster-wide operations' for operations that have to be run
>>> across all nodes in the cluster.
>>>
>>> Thanks,
>>>
>>> Dinesh
>>>
>>>
>>> On Tue, Sep 2, 2025 at 9:21 AM Bernardo Botella <
>>> contacto@bernardobotella.com> wrote:
>>>
>>>> This is an incredible contribution. Thanks a lot!
>>>>
>>>> Now, let me throw some thoughts :-)
>>>>
>>>> Rolling restarts is a great example of a broader feature that could be
>>>> seen as bulk operations on a cluster via Sidecar.
>>>>
>>>> What do you think about broadening the scope of the CEP to propose a
>>>> way (API) to perform bulk operations, and propose the current Rolling
>>>> restarts as the first implementation for that bulk operations API? I’m
>>>> proposing this as I see value to reuse this proposal for other bulk
>>>> operations such as enabling CDC (it requires enabling cdc on cassandra.yml
>>>> and some other operations) for better supporting CEP-44.
>>>>
>>>> I’m not quite sold on using a PATCH to move from pending state to
>>>> running state. Quick question, what is the goal of the pending state? I see
>>>> a PATCH operation as modifying part of an object data. In this case,
>>>> modifying the state looks like a change on the operation state, not on its
>>>> metadata. I’d love to hear your thoughts on this one.
>>>>
>>>> Again, thanks a lot for the contribution!
>>>> Bernardo
>>>>
>>>>
>>>> > On Aug 30, 2025, at 7:02 AM, Francisco Guerrero <fr...@apache.org>
>>>> wrote:
>>>> >
>>>> > Thanks Andrés for the CEP. This is a great contribution to the
>>>> project and
>>>> > aligns with the original intent of the Sidecar stated in CEP-1. I've
>>>> gone
>>>> > over the CEP details and it is consistent with the internals of
>>>> Sidecar.
>>>> >
>>>> > The only suggestion I have is to keep in mind the pluggability aspect
>>>> of
>>>> > Sidecar. For example, for the Distributed Restart portion of the
>>>> work, we
>>>> > should consider making interfaces that would allow us to potentially
>>>> move
>>>> > the responsibility of keeping the state outside of Cassandra.
>>>> >
>>>> > Best,
>>>> > - Francisco
>>>> >
>>>> > On 2025/08/29 19:56:08 Andrés Beck-Ruiz wrote:
>>>> >> Hello everyone,
>>>> >>
>>>> >> We would like to propose CEP 53: Cassandra Rolling Restarts via
>>>> Sidecar (
>>>> >>
>>>> https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-53%3A+Cassandra+Rolling+Restarts+via+Sidecar
>>>> >> )
>>>> >>
>>>> >> This CEP builds off of CEP-1
>>>> >> <
>>>> https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-1%3A+Apache+Cassandra+Management+Process%28es%29+-+Deprecated
>>>> >
>>>> >> and proposes a design for safe, efficient, and operator friendly
>>>> rolling
>>>> >> restarts on Cassandra clusters, as well as an extensible approach for
>>>> >> persisting future cluster-wide operations in Cassandra Sidecar. We
>>>> hope to
>>>> >> leverage this infrastructure in the future to implement upgrade
>>>> automation.
>>>> >>
>>>> >> We welcome all feedback and discussion. Thank you in advance for
>>>> your time
>>>> >> and consideration of this proposal!
>>>> >>
>>>> >> Best,
>>>> >> Andrés and Paulo
>>>> >>
>>>>
>>>>

Re: [DISCUSS] CEP 53: Cassandra Rolling Restarts via Sidecar

[Isaac Reath <is...@gmail.com>]

Thank you for the CEP and getting this discussion started Andrés!

I also think it makes sense to use the term "cluster-wide operations" to
refer to these types of orchestrated operations. Another option could be
“multi-node operations” to contrast with “single-node operations” since an
operation might operate on a subset of nodes in the cluster rather than the
entire cluster. In addition to CEP-44 and rolling restarts, there are other
cases such as config changes (
https://issues.apache.org/jira/browse/CASSSIDECAR-275) and version upgrades
(https://issues.apache.org/jira/browse/CASSSIDECAR-276) which could also
build off of these APIs.

On the question of reusing the OperationalJob interface: from an
implementation perspective, it seems generic enough to reuse for
cluster-wide operations. However, it needs to be extended to add support
for the fields we need to support cluster-wide jobs (for example, the nodes
involved in the operation, parallelism across racks, etc.)

As for the API, I think the question that needs to be answered is if it is
worthwhile to have a distinction between single-node operations and
cluster-wide operations. For example, if I wanted to restart a single node
using the API proposed in CEP-53, I could submit a restart job with a
single node in the “nodes” list. This provides API simplicity at the cost
of ergonomics. It also means that all inter-sidecar communication would go
through the proposed cluster_ops_node_state table. Personally, I think
these are acceptable tradeoffs to provide a unified API for operations that
is simpler for a user or operator to use and learn.
Isaac

On Tue, Sep 2, 2025 at 2:58 PM Andrés Beck-Ruiz <an...@gmail.com>
wrote:

> Thanks everyone for the feedback. +1 to using the term 'cluster-wide
> operations'.
>
> > The only suggestion I have is to keep in mind the pluggability aspect of
> > Sidecar. For example, for the Distributed Restart portion of the work, we
> > should consider making interfaces that would allow us to potentially move
> > the responsibility of keeping the state outside of Cassandra.
>
> Are you referring to tracking the state of a restart job (and cluster-wide
> operations in general) outside of sidecar_internal Cassandra tables?
>
> > What do you think about broadening the scope of the CEP to propose a way
> (API) to perform bulk operations, and propose the current Rolling restarts
> as the first implementation for that bulk operations API? I’m proposing
> this as I see value to reuse this proposal for other bulk operations such
> as enabling CDC (it requires enabling cdc on cassandra.yml and some other
> > operations) for better supporting CEP-44.
>
> We propose a way to persist and monitor cluster-wide operations in the new
> sidecar_internal system tables. (
> https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-53%3A+Cassandra+Rolling+Restarts+via+Sidecar#CEP53:CassandraRollingRestartsviaSidecar-CassandraSidecarSystemTables).
> I think it would make sense to also generalize the API to apply to
> cluster-wide operations. I'm curious about any feedback on whether this
> should be a separate API from the current operational job framework and
> live under the /cluster resource. We've discussed why we didn't propose to
> use the existing API and how the current framework would need to be
> extended here (
> https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-53%3A+Cassandra+Rolling+Restarts+via+Sidecar#CEP53:CassandraRollingRestartsviaSidecar-OperationalJobFramework
> ).
>
> > I’m not quite sold on using a PATCH to move from pending state to
> running state. Quick question, what is the goal of the pending state? I see
> a PATCH operation as modifying part of an object data. In this case,
> modifying the state looks like a change on the operation state, not on its
> metadata. I’d love to hear your thoughts on this one.
>
> The "PENDING" state allows for an operator to double check a submitted
> cluster-wide operation, which could have unintended consequences, before
> starting it. For example, performing a rolling restart could prevent other
> operations on the cluster that might be scheduled or needed, such as
> replacing a Cassandra instance. While an operator should be able to abort a
> restart job, I see value in having this guard against operator error.
>
> Given that we are applying a partial update to the resource, which in this
> context would be the restart job, we chose PATCH for this API.
>
> Best,
> Andrés
>
>
> On Tue, Sep 2, 2025 at 12:33 PM Dinesh Joshi <dj...@apache.org> wrote:
>
>> I would like to chime in and say that we need to refine our vocabulary.
>> The term 'bulk commands' was used originally in CEP-1. This is my fault
>> totally as I originally wrote that down. But over time it has caused
>> confusion. I believe 'cluster-wide operations' is a better term to describe
>> those operations. We have also used 'Bulk' in the context of CEP-28 which
>> means something rather different which leads to confusion. So I propose
>> using the term 'cluster-wide operations' for operations that have to be run
>> across all nodes in the cluster.
>>
>> Thanks,
>>
>> Dinesh
>>
>>
>> On Tue, Sep 2, 2025 at 9:21 AM Bernardo Botella <
>> contacto@bernardobotella.com> wrote:
>>
>>> This is an incredible contribution. Thanks a lot!
>>>
>>> Now, let me throw some thoughts :-)
>>>
>>> Rolling restarts is a great example of a broader feature that could be
>>> seen as bulk operations on a cluster via Sidecar.
>>>
>>> What do you think about broadening the scope of the CEP to propose a way
>>> (API) to perform bulk operations, and propose the current Rolling restarts
>>> as the first implementation for that bulk operations API? I’m proposing
>>> this as I see value to reuse this proposal for other bulk operations such
>>> as enabling CDC (it requires enabling cdc on cassandra.yml and some other
>>> operations) for better supporting CEP-44.
>>>
>>> I’m not quite sold on using a PATCH to move from pending state to
>>> running state. Quick question, what is the goal of the pending state? I see
>>> a PATCH operation as modifying part of an object data. In this case,
>>> modifying the state looks like a change on the operation state, not on its
>>> metadata. I’d love to hear your thoughts on this one.
>>>
>>> Again, thanks a lot for the contribution!
>>> Bernardo
>>>
>>>
>>> > On Aug 30, 2025, at 7:02 AM, Francisco Guerrero <fr...@apache.org>
>>> wrote:
>>> >
>>> > Thanks Andrés for the CEP. This is a great contribution to the project
>>> and
>>> > aligns with the original intent of the Sidecar stated in CEP-1. I've
>>> gone
>>> > over the CEP details and it is consistent with the internals of
>>> Sidecar.
>>> >
>>> > The only suggestion I have is to keep in mind the pluggability aspect
>>> of
>>> > Sidecar. For example, for the Distributed Restart portion of the work,
>>> we
>>> > should consider making interfaces that would allow us to potentially
>>> move
>>> > the responsibility of keeping the state outside of Cassandra.
>>> >
>>> > Best,
>>> > - Francisco
>>> >
>>> > On 2025/08/29 19:56:08 Andrés Beck-Ruiz wrote:
>>> >> Hello everyone,
>>> >>
>>> >> We would like to propose CEP 53: Cassandra Rolling Restarts via
>>> Sidecar (
>>> >>
>>> https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-53%3A+Cassandra+Rolling+Restarts+via+Sidecar
>>> >> )
>>> >>
>>> >> This CEP builds off of CEP-1
>>> >> <
>>> https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-1%3A+Apache+Cassandra+Management+Process%28es%29+-+Deprecated
>>> >
>>> >> and proposes a design for safe, efficient, and operator friendly
>>> rolling
>>> >> restarts on Cassandra clusters, as well as an extensible approach for
>>> >> persisting future cluster-wide operations in Cassandra Sidecar. We
>>> hope to
>>> >> leverage this infrastructure in the future to implement upgrade
>>> automation.
>>> >>
>>> >> We welcome all feedback and discussion. Thank you in advance for your
>>> time
>>> >> and consideration of this proposal!
>>> >>
>>> >> Best,
>>> >> Andrés and Paulo
>>> >>
>>>
>>>

Re: [DISCUSS] CEP 53: Cassandra Rolling Restarts via Sidecar

[Venkata Hari Krishna Nukala <n....@gmail.com>]

Thanks for the CEP! It fills the gaps of CEP-40 as well.

A small question from my side: I see that the underlying assumption is that
Sidecar is able to query Cassandra instances before bouncing/recognizing
the bounce. What if it could not communicate with the Cassandra instance
(e.g., binary protocol disabled, C* process experiencing issues, or C*
process starting as part of a new DC)?

+1 to Francisco's points on pluggability. Different implementations should
be able to maintain state in their own way.

On Wed, Sep 3, 2025 at 11:20 PM Dinesh Joshi <dj...@apache.org> wrote:

> The persistent state store for long running operations is an open
> question. Although we have a jira (CASSSIDECAR-341), it does not articulate
> a specific plan.
>
> It is generally accepted practice that we want to store the status of the
> operations outside of the database / cluster that is being operated. This
> circular dependency is not good.
>
> Consider a scenario where rolling restart is progressing and part of the
> cluster goes down due to unexpected hardware or network failure. In this
> case, if the state store is the cluster itself, you will require manual
> intervention to recover from it.
>
> Therefore the state storage should be made pluggable and its default
> implementation could leverage Cassandra. It doesn't need to be the same
> Cassandra cluster that is being managed by the Cassandra Sidecar.
>
> Thanks,
>
> Dinesh
>
> On Tue, Sep 2, 2025 at 11:58 AM Andrés Beck-Ruiz <an...@gmail.com>
> wrote:
>
>> Thanks everyone for the feedback. +1 to using the term 'cluster-wide
>> operations'.
>>
>> > The only suggestion I have is to keep in mind the pluggability aspect of
>> > Sidecar. For example, for the Distributed Restart portion of the work,
>> we
>> > should consider making interfaces that would allow us to potentially
>> move
>> > the responsibility of keeping the state outside of Cassandra.
>>
>> Are you referring to tracking the state of a restart job (and
>> cluster-wide operations in general) outside of sidecar_internal Cassandra
>> tables?
>>
>> > What do you think about broadening the scope of the CEP to propose a
>> way (API) to perform bulk operations, and propose the current Rolling
>> restarts as the first implementation for that bulk operations API? I’m
>> proposing this as I see value to reuse this proposal for other bulk
>> operations such as enabling CDC (it requires enabling cdc on cassandra.yml
>> and some other
>> > operations) for better supporting CEP-44.
>>
>> We propose a way to persist and monitor cluster-wide operations in the
>> new sidecar_internal system tables. (
>> https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-53%3A+Cassandra+Rolling+Restarts+via+Sidecar#CEP53:CassandraRollingRestartsviaSidecar-CassandraSidecarSystemTables).
>> I think it would make sense to also generalize the API to apply to
>> cluster-wide operations. I'm curious about any feedback on whether this
>> should be a separate API from the current operational job framework and
>> live under the /cluster resource. We've discussed why we didn't propose to
>> use the existing API and how the current framework would need to be
>> extended here (
>> https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-53%3A+Cassandra+Rolling+Restarts+via+Sidecar#CEP53:CassandraRollingRestartsviaSidecar-OperationalJobFramework
>> ).
>>
>> > I’m not quite sold on using a PATCH to move from pending state to
>> running state. Quick question, what is the goal of the pending state? I see
>> a PATCH operation as modifying part of an object data. In this case,
>> modifying the state looks like a change on the operation state, not on its
>> metadata. I’d love to hear your thoughts on this one.
>>
>> The "PENDING" state allows for an operator to double check a submitted
>> cluster-wide operation, which could have unintended consequences, before
>> starting it. For example, performing a rolling restart could prevent other
>> operations on the cluster that might be scheduled or needed, such as
>> replacing a Cassandra instance. While an operator should be able to abort a
>> restart job, I see value in having this guard against operator error.
>>
>> Given that we are applying a partial update to the resource, which in
>> this context would be the restart job, we chose PATCH for this API.
>>
>> Best,
>> Andrés
>>
>>
>> On Tue, Sep 2, 2025 at 12:33 PM Dinesh Joshi <dj...@apache.org> wrote:
>>
>>> I would like to chime in and say that we need to refine our vocabulary.
>>> The term 'bulk commands' was used originally in CEP-1. This is my fault
>>> totally as I originally wrote that down. But over time it has caused
>>> confusion. I believe 'cluster-wide operations' is a better term to describe
>>> those operations. We have also used 'Bulk' in the context of CEP-28 which
>>> means something rather different which leads to confusion. So I propose
>>> using the term 'cluster-wide operations' for operations that have to be run
>>> across all nodes in the cluster.
>>>
>>> Thanks,
>>>
>>> Dinesh
>>>
>>>
>>> On Tue, Sep 2, 2025 at 9:21 AM Bernardo Botella <
>>> contacto@bernardobotella.com> wrote:
>>>
>>>> This is an incredible contribution. Thanks a lot!
>>>>
>>>> Now, let me throw some thoughts :-)
>>>>
>>>> Rolling restarts is a great example of a broader feature that could be
>>>> seen as bulk operations on a cluster via Sidecar.
>>>>
>>>> What do you think about broadening the scope of the CEP to propose a
>>>> way (API) to perform bulk operations, and propose the current Rolling
>>>> restarts as the first implementation for that bulk operations API? I’m
>>>> proposing this as I see value to reuse this proposal for other bulk
>>>> operations such as enabling CDC (it requires enabling cdc on cassandra.yml
>>>> and some other operations) for better supporting CEP-44.
>>>>
>>>> I’m not quite sold on using a PATCH to move from pending state to
>>>> running state. Quick question, what is the goal of the pending state? I see
>>>> a PATCH operation as modifying part of an object data. In this case,
>>>> modifying the state looks like a change on the operation state, not on its
>>>> metadata. I’d love to hear your thoughts on this one.
>>>>
>>>> Again, thanks a lot for the contribution!
>>>> Bernardo
>>>>
>>>>
>>>> > On Aug 30, 2025, at 7:02 AM, Francisco Guerrero <fr...@apache.org>
>>>> wrote:
>>>> >
>>>> > Thanks Andrés for the CEP. This is a great contribution to the
>>>> project and
>>>> > aligns with the original intent of the Sidecar stated in CEP-1. I've
>>>> gone
>>>> > over the CEP details and it is consistent with the internals of
>>>> Sidecar.
>>>> >
>>>> > The only suggestion I have is to keep in mind the pluggability aspect
>>>> of
>>>> > Sidecar. For example, for the Distributed Restart portion of the
>>>> work, we
>>>> > should consider making interfaces that would allow us to potentially
>>>> move
>>>> > the responsibility of keeping the state outside of Cassandra.
>>>> >
>>>> > Best,
>>>> > - Francisco
>>>> >
>>>> > On 2025/08/29 19:56:08 Andrés Beck-Ruiz wrote:
>>>> >> Hello everyone,
>>>> >>
>>>> >> We would like to propose CEP 53: Cassandra Rolling Restarts via
>>>> Sidecar (
>>>> >>
>>>> https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-53%3A+Cassandra+Rolling+Restarts+via+Sidecar
>>>> >> )
>>>> >>
>>>> >> This CEP builds off of CEP-1
>>>> >> <
>>>> https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-1%3A+Apache+Cassandra+Management+Process%28es%29+-+Deprecated
>>>> >
>>>> >> and proposes a design for safe, efficient, and operator friendly
>>>> rolling
>>>> >> restarts on Cassandra clusters, as well as an extensible approach for
>>>> >> persisting future cluster-wide operations in Cassandra Sidecar. We
>>>> hope to
>>>> >> leverage this infrastructure in the future to implement upgrade
>>>> automation.
>>>> >>
>>>> >> We welcome all feedback and discussion. Thank you in advance for
>>>> your time
>>>> >> and consideration of this proposal!
>>>> >>
>>>> >> Best,
>>>> >> Andrés and Paulo
>>>> >>
>>>>
>>>>

Re: [DISCUSS] CEP 53: Cassandra Rolling Restarts via Sidecar

[Dinesh Joshi <dj...@apache.org>]

The persistent state store for long running operations is an open question.
Although we have a jira (CASSSIDECAR-341), it does not articulate a
specific plan.

It is generally accepted practice that we want to store the status of the
operations outside of the database / cluster that is being operated. This
circular dependency is not good.

Consider a scenario where rolling restart is progressing and part of the
cluster goes down due to unexpected hardware or network failure. In this
case, if the state store is the cluster itself, you will require manual
intervention to recover from it.

Therefore the state storage should be made pluggable and its default
implementation could leverage Cassandra. It doesn't need to be the same
Cassandra cluster that is being managed by the Cassandra Sidecar.

Thanks,

Dinesh

On Tue, Sep 2, 2025 at 11:58 AM Andrés Beck-Ruiz <an...@gmail.com>
wrote:

> Thanks everyone for the feedback. +1 to using the term 'cluster-wide
> operations'.
>
> > The only suggestion I have is to keep in mind the pluggability aspect of
> > Sidecar. For example, for the Distributed Restart portion of the work, we
> > should consider making interfaces that would allow us to potentially move
> > the responsibility of keeping the state outside of Cassandra.
>
> Are you referring to tracking the state of a restart job (and cluster-wide
> operations in general) outside of sidecar_internal Cassandra tables?
>
> > What do you think about broadening the scope of the CEP to propose a way
> (API) to perform bulk operations, and propose the current Rolling restarts
> as the first implementation for that bulk operations API? I’m proposing
> this as I see value to reuse this proposal for other bulk operations such
> as enabling CDC (it requires enabling cdc on cassandra.yml and some other
> > operations) for better supporting CEP-44.
>
> We propose a way to persist and monitor cluster-wide operations in the new
> sidecar_internal system tables. (
> https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-53%3A+Cassandra+Rolling+Restarts+via+Sidecar#CEP53:CassandraRollingRestartsviaSidecar-CassandraSidecarSystemTables).
> I think it would make sense to also generalize the API to apply to
> cluster-wide operations. I'm curious about any feedback on whether this
> should be a separate API from the current operational job framework and
> live under the /cluster resource. We've discussed why we didn't propose to
> use the existing API and how the current framework would need to be
> extended here (
> https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-53%3A+Cassandra+Rolling+Restarts+via+Sidecar#CEP53:CassandraRollingRestartsviaSidecar-OperationalJobFramework
> ).
>
> > I’m not quite sold on using a PATCH to move from pending state to
> running state. Quick question, what is the goal of the pending state? I see
> a PATCH operation as modifying part of an object data. In this case,
> modifying the state looks like a change on the operation state, not on its
> metadata. I’d love to hear your thoughts on this one.
>
> The "PENDING" state allows for an operator to double check a submitted
> cluster-wide operation, which could have unintended consequences, before
> starting it. For example, performing a rolling restart could prevent other
> operations on the cluster that might be scheduled or needed, such as
> replacing a Cassandra instance. While an operator should be able to abort a
> restart job, I see value in having this guard against operator error.
>
> Given that we are applying a partial update to the resource, which in this
> context would be the restart job, we chose PATCH for this API.
>
> Best,
> Andrés
>
>
> On Tue, Sep 2, 2025 at 12:33 PM Dinesh Joshi <dj...@apache.org> wrote:
>
>> I would like to chime in and say that we need to refine our vocabulary.
>> The term 'bulk commands' was used originally in CEP-1. This is my fault
>> totally as I originally wrote that down. But over time it has caused
>> confusion. I believe 'cluster-wide operations' is a better term to describe
>> those operations. We have also used 'Bulk' in the context of CEP-28 which
>> means something rather different which leads to confusion. So I propose
>> using the term 'cluster-wide operations' for operations that have to be run
>> across all nodes in the cluster.
>>
>> Thanks,
>>
>> Dinesh
>>
>>
>> On Tue, Sep 2, 2025 at 9:21 AM Bernardo Botella <
>> contacto@bernardobotella.com> wrote:
>>
>>> This is an incredible contribution. Thanks a lot!
>>>
>>> Now, let me throw some thoughts :-)
>>>
>>> Rolling restarts is a great example of a broader feature that could be
>>> seen as bulk operations on a cluster via Sidecar.
>>>
>>> What do you think about broadening the scope of the CEP to propose a way
>>> (API) to perform bulk operations, and propose the current Rolling restarts
>>> as the first implementation for that bulk operations API? I’m proposing
>>> this as I see value to reuse this proposal for other bulk operations such
>>> as enabling CDC (it requires enabling cdc on cassandra.yml and some other
>>> operations) for better supporting CEP-44.
>>>
>>> I’m not quite sold on using a PATCH to move from pending state to
>>> running state. Quick question, what is the goal of the pending state? I see
>>> a PATCH operation as modifying part of an object data. In this case,
>>> modifying the state looks like a change on the operation state, not on its
>>> metadata. I’d love to hear your thoughts on this one.
>>>
>>> Again, thanks a lot for the contribution!
>>> Bernardo
>>>
>>>
>>> > On Aug 30, 2025, at 7:02 AM, Francisco Guerrero <fr...@apache.org>
>>> wrote:
>>> >
>>> > Thanks Andrés for the CEP. This is a great contribution to the project
>>> and
>>> > aligns with the original intent of the Sidecar stated in CEP-1. I've
>>> gone
>>> > over the CEP details and it is consistent with the internals of
>>> Sidecar.
>>> >
>>> > The only suggestion I have is to keep in mind the pluggability aspect
>>> of
>>> > Sidecar. For example, for the Distributed Restart portion of the work,
>>> we
>>> > should consider making interfaces that would allow us to potentially
>>> move
>>> > the responsibility of keeping the state outside of Cassandra.
>>> >
>>> > Best,
>>> > - Francisco
>>> >
>>> > On 2025/08/29 19:56:08 Andrés Beck-Ruiz wrote:
>>> >> Hello everyone,
>>> >>
>>> >> We would like to propose CEP 53: Cassandra Rolling Restarts via
>>> Sidecar (
>>> >>
>>> https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-53%3A+Cassandra+Rolling+Restarts+via+Sidecar
>>> >> )
>>> >>
>>> >> This CEP builds off of CEP-1
>>> >> <
>>> https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-1%3A+Apache+Cassandra+Management+Process%28es%29+-+Deprecated
>>> >
>>> >> and proposes a design for safe, efficient, and operator friendly
>>> rolling
>>> >> restarts on Cassandra clusters, as well as an extensible approach for
>>> >> persisting future cluster-wide operations in Cassandra Sidecar. We
>>> hope to
>>> >> leverage this infrastructure in the future to implement upgrade
>>> automation.
>>> >>
>>> >> We welcome all feedback and discussion. Thank you in advance for your
>>> time
>>> >> and consideration of this proposal!
>>> >>
>>> >> Best,
>>> >> Andrés and Paulo
>>> >>
>>>
>>>

Re: [DISCUSS] CEP 53: Cassandra Rolling Restarts via Sidecar

[Jaydeep Chovatia <ch...@gmail.com>]

Sorry if this was already discussed and I missed it — I had a quick
question on the health check scope.

The design seems focused on cluster/availability signals (ring stable,
peers up), which is a great start, but doesn’t mention pluggable workload
signals like:  1) compaction load (nodetool compactionstats)  2) netstats
activity (nodetool netstats) 3) hints backlog / streaming  pending flushes
or memtable pressure.
Since restarting during heavy compaction/hints can add risk, are these
kinds of workload-aware checks in scope for the MVP, or considered future
work?

In some cases, even if the CQL port is up, operators may want to add an
additional delay (e.g. 5 minutes) before proceeding to the next batch of
nodes. Would it make sense to support this as a configurable option, or via
some hook mechanism, so that operators can insert a pause between batches
if desired?

Jaydeep

On Thu, Sep 4, 2025 at 9:44 AM Jindal, Himanshu <hi...@amazon.com> wrote:

> Hi Andres,
>
>
>
> This looks like a great CEP. Having official, source-controlled code
> within Cassandra (or a sidecar in this case) to handle common operator
> actions would centralize best practices and make the operator experience
> smoother—especially for users who may not have deep Cassandra expertise.
>
>
>
> A couple of questions:
>
>    1. Have we considered introducing the concept of a *datacenter*
>    alongside *cluster*? I imagine there will be cases where a user wants
>    to perform a rolling restart on a single datacenter rather than across the
>    entire cluster.
>    2. Do we see this framework extending to other cluster- or
>    datacenter-wide operations, such as scale-up/scale-down operations, or
>    backups/restores, or nodetool rebuilds run as part of adding a new
>    datacenter?
>
>
>
> Best,
> Himanshu
>
>
>
>
>
> *From: *Andrés Beck-Ruiz <an...@gmail.com>
> *Reply-To: *"dev@cassandra.apache.org" <de...@cassandra.apache.org>
> *Date: *Tuesday, September 2, 2025 at 11:58 AM
> *To: *"dev@cassandra.apache.org" <de...@cassandra.apache.org>
> *Subject: *RE: [EXTERNAL] [DISCUSS] CEP 53: Cassandra Rolling Restarts
> via Sidecar
>
>
>
> *CAUTION*: This email originated from outside of the organization. Do not
> click links or open attachments unless you can confirm the sender and know
> the content is safe.
>
>
>
> Thanks everyone for the feedback. +1 to using the term 'cluster-wide
> operations'.
>
> > The only suggestion I have is to keep in mind the pluggability aspect of
> > Sidecar. For example, for the Distributed Restart portion of the work, we
> > should consider making interfaces that would allow us to potentially move
> > the responsibility of keeping the state outside of Cassandra.
>
> Are you referring to tracking the state of a restart job (and cluster-wide
> operations in general) outside of sidecar_internal Cassandra tables?
>
> > What do you think about broadening the scope of the CEP to propose a way
> (API) to perform bulk operations, and propose the current Rolling restarts
> as the first implementation for that bulk operations API? I’m proposing
> this as I see value to reuse this proposal for other bulk operations such
> as enabling CDC (it requires enabling cdc on cassandra.yml and some other
> > operations) for better supporting CEP-44.
>
> We propose a way to persist and monitor cluster-wide operations in the new
> sidecar_internal system tables. (
> https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-53%3A+Cassandra+Rolling+Restarts+via+Sidecar#CEP53:CassandraRollingRestartsviaSidecar-CassandraSidecarSystemTables).
> I think it would make sense to also generalize the API to apply to
> cluster-wide operations. I'm curious about any feedback on whether this
> should be a separate API from the current operational job framework and
> live under the /cluster resource. We've discussed why we didn't propose to
> use the existing API and how the current framework would need to be
> extended here (
> https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-53%3A+Cassandra+Rolling+Restarts+via+Sidecar#CEP53:CassandraRollingRestartsviaSidecar-OperationalJobFramework
> ).
>
> > I’m not quite sold on using a PATCH to move from pending state to
> running state. Quick question, what is the goal of the pending state? I see
> a PATCH operation as modifying part of an object data. In this case,
> modifying the state looks like a change on the operation state, not on its
> metadata. I’d love to hear your thoughts on this one.
>
> The "PENDING" state allows for an operator to double check a submitted
> cluster-wide operation, which could have unintended consequences, before
> starting it. For example, performing a rolling restart could prevent other
> operations on the cluster that might be scheduled or needed, such as
> replacing a Cassandra instance. While an operator should be able to abort a
> restart job, I see value in having this guard against operator error.
>
> Given that we are applying a partial update to the resource, which in this
> context would be the restart job, we chose PATCH for this API.
>
> Best,
> Andrés
>
>
>
> On Tue, Sep 2, 2025 at 12:33 PM Dinesh Joshi <dj...@apache.org> wrote:
>
> I would like to chime in and say that we need to refine our vocabulary.
> The term 'bulk commands' was used originally in CEP-1. This is my fault
> totally as I originally wrote that down. But over time it has caused
> confusion. I believe 'cluster-wide operations' is a better term to describe
> those operations. We have also used 'Bulk' in the context of CEP-28 which
> means something rather different which leads to confusion. So I propose
> using the term 'cluster-wide operations' for operations that have to be run
> across all nodes in the cluster.
>
>
>
> Thanks,
>
>
>
> Dinesh
>
>
>
>
>
> On Tue, Sep 2, 2025 at 9:21 AM Bernardo Botella <
> contacto@bernardobotella.com> wrote:
>
> This is an incredible contribution. Thanks a lot!
>
> Now, let me throw some thoughts :-)
>
> Rolling restarts is a great example of a broader feature that could be
> seen as bulk operations on a cluster via Sidecar.
>
> What do you think about broadening the scope of the CEP to propose a way
> (API) to perform bulk operations, and propose the current Rolling restarts
> as the first implementation for that bulk operations API? I’m proposing
> this as I see value to reuse this proposal for other bulk operations such
> as enabling CDC (it requires enabling cdc on cassandra.yml and some other
> operations) for better supporting CEP-44.
>
> I’m not quite sold on using a PATCH to move from pending state to running
> state. Quick question, what is the goal of the pending state? I see a PATCH
> operation as modifying part of an object data. In this case, modifying the
> state looks like a change on the operation state, not on its metadata. I’d
> love to hear your thoughts on this one.
>
> Again, thanks a lot for the contribution!
> Bernardo
>
>
> > On Aug 30, 2025, at 7:02 AM, Francisco Guerrero <fr...@apache.org>
> wrote:
> >
> > Thanks Andrés for the CEP. This is a great contribution to the project
> and
> > aligns with the original intent of the Sidecar stated in CEP-1. I've gone
> > over the CEP details and it is consistent with the internals of Sidecar.
> >
> > The only suggestion I have is to keep in mind the pluggability aspect of
> > Sidecar. For example, for the Distributed Restart portion of the work, we
> > should consider making interfaces that would allow us to potentially move
> > the responsibility of keeping the state outside of Cassandra.
> >
> > Best,
> > - Francisco
> >
> > On 2025/08/29 19:56:08 Andrés Beck-Ruiz wrote:
> >> Hello everyone,
> >>
> >> We would like to propose CEP 53: Cassandra Rolling Restarts via Sidecar
> (
> >>
> https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-53%3A+Cassandra+Rolling+Restarts+via+Sidecar
> >> )
> >>
> >> This CEP builds off of CEP-1
> >> <
> https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-1%3A+Apache+Cassandra+Management+Process%28es%29+-+Deprecated
> >
> >> and proposes a design for safe, efficient, and operator friendly rolling
> >> restarts on Cassandra clusters, as well as an extensible approach for
> >> persisting future cluster-wide operations in Cassandra Sidecar. We hope
> to
> >> leverage this infrastructure in the future to implement upgrade
> automation.
> >>
> >> We welcome all feedback and discussion. Thank you in advance for your
> time
> >> and consideration of this proposal!
> >>
> >> Best,
> >> Andrés and Paulo
> >>
>
>

Re: [DISCUSS] CEP 53: Cassandra Rolling Restarts via Sidecar

["Jindal, Himanshu" <hi...@amazon.com>]

Hi Andres,

This looks like a great CEP. Having official, source-controlled code within Cassandra (or a sidecar in this case) to handle common operator actions would centralize best practices and make the operator experience smoother—especially for users who may not have deep Cassandra expertise.

A couple of questions:

  1.  Have we considered introducing the concept of a datacenter alongside cluster? I imagine there will be cases where a user wants to perform a rolling restart on a single datacenter rather than across the entire cluster.
  2.  Do we see this framework extending to other cluster- or datacenter-wide operations, such as scale-up/scale-down operations, or backups/restores, or nodetool rebuilds run as part of adding a new datacenter?

Best,
Himanshu


From: Andrés Beck-Ruiz <an...@gmail.com>
Reply-To: "dev@cassandra.apache.org" <de...@cassandra.apache.org>
Date: Tuesday, September 2, 2025 at 11:58 AM
To: "dev@cassandra.apache.org" <de...@cassandra.apache.org>
Subject: RE: [EXTERNAL] [DISCUSS] CEP 53: Cassandra Rolling Restarts via Sidecar


CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.

Thanks everyone for the feedback. +1 to using the term 'cluster-wide operations'.

> The only suggestion I have is to keep in mind the pluggability aspect of
> Sidecar. For example, for the Distributed Restart portion of the work, we
> should consider making interfaces that would allow us to potentially move
> the responsibility of keeping the state outside of Cassandra.

Are you referring to tracking the state of a restart job (and cluster-wide operations in general) outside of sidecar_internal Cassandra tables?

> What do you think about broadening the scope of the CEP to propose a way (API) to perform bulk operations, and propose the current Rolling restarts as the first implementation for that bulk operations API? I’m proposing this as I see value to reuse this proposal for other bulk operations such as enabling CDC (it requires enabling cdc on cassandra.yml and some other
> operations) for better supporting CEP-44.

We propose a way to persist and monitor cluster-wide operations in the new sidecar_internal system tables. (https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-53%3A+Cassandra+Rolling+Restarts+via+Sidecar#CEP53:CassandraRollingRestartsviaSidecar-CassandraSidecarSystemTables). I think it would make sense to also generalize the API to apply to cluster-wide operations. I'm curious about any feedback on whether this should be a separate API from the current operational job framework and live under the /cluster resource. We've discussed why we didn't propose to use the existing API and how the current framework would need to be extended here (https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-53%3A+Cassandra+Rolling+Restarts+via+Sidecar#CEP53:CassandraRollingRestartsviaSidecar-OperationalJobFramework).

> I’m not quite sold on using a PATCH to move from pending state to running state. Quick question, what is the goal of the pending state? I see a PATCH operation as modifying part of an object data. In this case, modifying the state looks like a change on the operation state, not on its metadata. I’d love to hear your thoughts on this one.

The "PENDING" state allows for an operator to double check a submitted cluster-wide operation, which could have unintended consequences, before starting it. For example, performing a rolling restart could prevent other operations on the cluster that might be scheduled or needed, such as replacing a Cassandra instance. While an operator should be able to abort a restart job, I see value in having this guard against operator error.

Given that we are applying a partial update to the resource, which in this context would be the restart job, we chose PATCH for this API.

Best,
Andrés

On Tue, Sep 2, 2025 at 12:33 PM Dinesh Joshi <dj...@apache.org>> wrote:
I would like to chime in and say that we need to refine our vocabulary. The term 'bulk commands' was used originally in CEP-1. This is my fault totally as I originally wrote that down. But over time it has caused confusion. I believe 'cluster-wide operations' is a better term to describe those operations. We have also used 'Bulk' in the context of CEP-28 which means something rather different which leads to confusion. So I propose using the term 'cluster-wide operations' for operations that have to be run across all nodes in the cluster.

Thanks,

Dinesh


On Tue, Sep 2, 2025 at 9:21 AM Bernardo Botella <co...@bernardobotella.com>> wrote:
This is an incredible contribution. Thanks a lot!

Now, let me throw some thoughts :-)

Rolling restarts is a great example of a broader feature that could be seen as bulk operations on a cluster via Sidecar.

What do you think about broadening the scope of the CEP to propose a way (API) to perform bulk operations, and propose the current Rolling restarts as the first implementation for that bulk operations API? I’m proposing this as I see value to reuse this proposal for other bulk operations such as enabling CDC (it requires enabling cdc on cassandra.yml and some other operations) for better supporting CEP-44.

I’m not quite sold on using a PATCH to move from pending state to running state. Quick question, what is the goal of the pending state? I see a PATCH operation as modifying part of an object data. In this case, modifying the state looks like a change on the operation state, not on its metadata. I’d love to hear your thoughts on this one.

Again, thanks a lot for the contribution!
Bernardo


> On Aug 30, 2025, at 7:02 AM, Francisco Guerrero <fr...@apache.org>> wrote:
>
> Thanks Andrés for the CEP. This is a great contribution to the project and
> aligns with the original intent of the Sidecar stated in CEP-1. I've gone
> over the CEP details and it is consistent with the internals of Sidecar.
>
> The only suggestion I have is to keep in mind the pluggability aspect of
> Sidecar. For example, for the Distributed Restart portion of the work, we
> should consider making interfaces that would allow us to potentially move
> the responsibility of keeping the state outside of Cassandra.
>
> Best,
> - Francisco
>
> On 2025/08/29 19:56:08 Andrés Beck-Ruiz wrote:
>> Hello everyone,
>>
>> We would like to propose CEP 53: Cassandra Rolling Restarts via Sidecar (
>> https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-53%3A+Cassandra+Rolling+Restarts+via+Sidecar
>> )
>>
>> This CEP builds off of CEP-1
>> <https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-1%3A+Apache+Cassandra+Management+Process%28es%29+-+Deprecated>
>> and proposes a design for safe, efficient, and operator friendly rolling
>> restarts on Cassandra clusters, as well as an extensible approach for
>> persisting future cluster-wide operations in Cassandra Sidecar. We hope to
>> leverage this infrastructure in the future to implement upgrade automation.
>>
>> We welcome all feedback and discussion. Thank you in advance for your time
>> and consideration of this proposal!
>>
>> Best,
>> Andrés and Paulo
>>

Re: [DISCUSS] CEP 53: Cassandra Rolling Restarts via Sidecar

[Andrés Beck-Ruiz <an...@gmail.com>]

Thanks everyone for the feedback. +1 to using the term 'cluster-wide
operations'.

> The only suggestion I have is to keep in mind the pluggability aspect of
> Sidecar. For example, for the Distributed Restart portion of the work, we
> should consider making interfaces that would allow us to potentially move
> the responsibility of keeping the state outside of Cassandra.

Are you referring to tracking the state of a restart job (and cluster-wide
operations in general) outside of sidecar_internal Cassandra tables?

> What do you think about broadening the scope of the CEP to propose a way
(API) to perform bulk operations, and propose the current Rolling restarts
as the first implementation for that bulk operations API? I’m proposing
this as I see value to reuse this proposal for other bulk operations such
as enabling CDC (it requires enabling cdc on cassandra.yml and some other
> operations) for better supporting CEP-44.

We propose a way to persist and monitor cluster-wide operations in the new
sidecar_internal system tables. (
https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-53%3A+Cassandra+Rolling+Restarts+via+Sidecar#CEP53:CassandraRollingRestartsviaSidecar-CassandraSidecarSystemTables).
I think it would make sense to also generalize the API to apply to
cluster-wide operations. I'm curious about any feedback on whether this
should be a separate API from the current operational job framework and
live under the /cluster resource. We've discussed why we didn't propose to
use the existing API and how the current framework would need to be
extended here (
https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-53%3A+Cassandra+Rolling+Restarts+via+Sidecar#CEP53:CassandraRollingRestartsviaSidecar-OperationalJobFramework
).

> I’m not quite sold on using a PATCH to move from pending state to running
state. Quick question, what is the goal of the pending state? I see a PATCH
operation as modifying part of an object data. In this case, modifying the
state looks like a change on the operation state, not on its metadata. I’d
love to hear your thoughts on this one.

The "PENDING" state allows for an operator to double check a submitted
cluster-wide operation, which could have unintended consequences, before
starting it. For example, performing a rolling restart could prevent other
operations on the cluster that might be scheduled or needed, such as
replacing a Cassandra instance. While an operator should be able to abort a
restart job, I see value in having this guard against operator error.

Given that we are applying a partial update to the resource, which in this
context would be the restart job, we chose PATCH for this API.

Best,
Andrés


On Tue, Sep 2, 2025 at 12:33 PM Dinesh Joshi <dj...@apache.org> wrote:

> I would like to chime in and say that we need to refine our vocabulary.
> The term 'bulk commands' was used originally in CEP-1. This is my fault
> totally as I originally wrote that down. But over time it has caused
> confusion. I believe 'cluster-wide operations' is a better term to describe
> those operations. We have also used 'Bulk' in the context of CEP-28 which
> means something rather different which leads to confusion. So I propose
> using the term 'cluster-wide operations' for operations that have to be run
> across all nodes in the cluster.
>
> Thanks,
>
> Dinesh
>
>
> On Tue, Sep 2, 2025 at 9:21 AM Bernardo Botella <
> contacto@bernardobotella.com> wrote:
>
>> This is an incredible contribution. Thanks a lot!
>>
>> Now, let me throw some thoughts :-)
>>
>> Rolling restarts is a great example of a broader feature that could be
>> seen as bulk operations on a cluster via Sidecar.
>>
>> What do you think about broadening the scope of the CEP to propose a way
>> (API) to perform bulk operations, and propose the current Rolling restarts
>> as the first implementation for that bulk operations API? I’m proposing
>> this as I see value to reuse this proposal for other bulk operations such
>> as enabling CDC (it requires enabling cdc on cassandra.yml and some other
>> operations) for better supporting CEP-44.
>>
>> I’m not quite sold on using a PATCH to move from pending state to running
>> state. Quick question, what is the goal of the pending state? I see a PATCH
>> operation as modifying part of an object data. In this case, modifying the
>> state looks like a change on the operation state, not on its metadata. I’d
>> love to hear your thoughts on this one.
>>
>> Again, thanks a lot for the contribution!
>> Bernardo
>>
>>
>> > On Aug 30, 2025, at 7:02 AM, Francisco Guerrero <fr...@apache.org>
>> wrote:
>> >
>> > Thanks Andrés for the CEP. This is a great contribution to the project
>> and
>> > aligns with the original intent of the Sidecar stated in CEP-1. I've
>> gone
>> > over the CEP details and it is consistent with the internals of Sidecar.
>> >
>> > The only suggestion I have is to keep in mind the pluggability aspect of
>> > Sidecar. For example, for the Distributed Restart portion of the work,
>> we
>> > should consider making interfaces that would allow us to potentially
>> move
>> > the responsibility of keeping the state outside of Cassandra.
>> >
>> > Best,
>> > - Francisco
>> >
>> > On 2025/08/29 19:56:08 Andrés Beck-Ruiz wrote:
>> >> Hello everyone,
>> >>
>> >> We would like to propose CEP 53: Cassandra Rolling Restarts via
>> Sidecar (
>> >>
>> https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-53%3A+Cassandra+Rolling+Restarts+via+Sidecar
>> >> )
>> >>
>> >> This CEP builds off of CEP-1
>> >> <
>> https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-1%3A+Apache+Cassandra+Management+Process%28es%29+-+Deprecated
>> >
>> >> and proposes a design for safe, efficient, and operator friendly
>> rolling
>> >> restarts on Cassandra clusters, as well as an extensible approach for
>> >> persisting future cluster-wide operations in Cassandra Sidecar. We
>> hope to
>> >> leverage this infrastructure in the future to implement upgrade
>> automation.
>> >>
>> >> We welcome all feedback and discussion. Thank you in advance for your
>> time
>> >> and consideration of this proposal!
>> >>
>> >> Best,
>> >> Andrés and Paulo
>> >>
>>
>>

Re: [DISCUSS] CEP 53: Cassandra Rolling Restarts via Sidecar

[Dinesh Joshi <dj...@apache.org>]

I would like to chime in and say that we need to refine our vocabulary. The
term 'bulk commands' was used originally in CEP-1. This is my fault totally
as I originally wrote that down. But over time it has caused confusion. I
believe 'cluster-wide operations' is a better term to describe those
operations. We have also used 'Bulk' in the context of CEP-28 which means
something rather different which leads to confusion. So I propose using the
term 'cluster-wide operations' for operations that have to be run across
all nodes in the cluster.

Thanks,

Dinesh


On Tue, Sep 2, 2025 at 9:21 AM Bernardo Botella <
contacto@bernardobotella.com> wrote:

> This is an incredible contribution. Thanks a lot!
>
> Now, let me throw some thoughts :-)
>
> Rolling restarts is a great example of a broader feature that could be
> seen as bulk operations on a cluster via Sidecar.
>
> What do you think about broadening the scope of the CEP to propose a way
> (API) to perform bulk operations, and propose the current Rolling restarts
> as the first implementation for that bulk operations API? I’m proposing
> this as I see value to reuse this proposal for other bulk operations such
> as enabling CDC (it requires enabling cdc on cassandra.yml and some other
> operations) for better supporting CEP-44.
>
> I’m not quite sold on using a PATCH to move from pending state to running
> state. Quick question, what is the goal of the pending state? I see a PATCH
> operation as modifying part of an object data. In this case, modifying the
> state looks like a change on the operation state, not on its metadata. I’d
> love to hear your thoughts on this one.
>
> Again, thanks a lot for the contribution!
> Bernardo
>
>
> > On Aug 30, 2025, at 7:02 AM, Francisco Guerrero <fr...@apache.org>
> wrote:
> >
> > Thanks Andrés for the CEP. This is a great contribution to the project
> and
> > aligns with the original intent of the Sidecar stated in CEP-1. I've gone
> > over the CEP details and it is consistent with the internals of Sidecar.
> >
> > The only suggestion I have is to keep in mind the pluggability aspect of
> > Sidecar. For example, for the Distributed Restart portion of the work, we
> > should consider making interfaces that would allow us to potentially move
> > the responsibility of keeping the state outside of Cassandra.
> >
> > Best,
> > - Francisco
> >
> > On 2025/08/29 19:56:08 Andrés Beck-Ruiz wrote:
> >> Hello everyone,
> >>
> >> We would like to propose CEP 53: Cassandra Rolling Restarts via Sidecar
> (
> >>
> https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-53%3A+Cassandra+Rolling+Restarts+via+Sidecar
> >> )
> >>
> >> This CEP builds off of CEP-1
> >> <
> https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-1%3A+Apache+Cassandra+Management+Process%28es%29+-+Deprecated
> >
> >> and proposes a design for safe, efficient, and operator friendly rolling
> >> restarts on Cassandra clusters, as well as an extensible approach for
> >> persisting future cluster-wide operations in Cassandra Sidecar. We hope
> to
> >> leverage this infrastructure in the future to implement upgrade
> automation.
> >>
> >> We welcome all feedback and discussion. Thank you in advance for your
> time
> >> and consideration of this proposal!
> >>
> >> Best,
> >> Andrés and Paulo
> >>
>
>

Re: [DISCUSS] CEP 53: Cassandra Rolling Restarts via Sidecar

[Bernardo Botella <co...@bernardobotella.com>]

This is an incredible contribution. Thanks a lot!

Now, let me throw some thoughts :-)

Rolling restarts is a great example of a broader feature that could be seen as bulk operations on a cluster via Sidecar.

What do you think about broadening the scope of the CEP to propose a way (API) to perform bulk operations, and propose the current Rolling restarts as the first implementation for that bulk operations API? I’m proposing this as I see value to reuse this proposal for other bulk operations such as enabling CDC (it requires enabling cdc on cassandra.yml and some other operations) for better supporting CEP-44.

I’m not quite sold on using a PATCH to move from pending state to running state. Quick question, what is the goal of the pending state? I see a PATCH operation as modifying part of an object data. In this case, modifying the state looks like a change on the operation state, not on its metadata. I’d love to hear your thoughts on this one.

Again, thanks a lot for the contribution!
Bernardo


> On Aug 30, 2025, at 7:02 AM, Francisco Guerrero <fr...@apache.org> wrote:
> 
> Thanks Andrés for the CEP. This is a great contribution to the project and
> aligns with the original intent of the Sidecar stated in CEP-1. I've gone
> over the CEP details and it is consistent with the internals of Sidecar.
> 
> The only suggestion I have is to keep in mind the pluggability aspect of
> Sidecar. For example, for the Distributed Restart portion of the work, we
> should consider making interfaces that would allow us to potentially move
> the responsibility of keeping the state outside of Cassandra.
> 
> Best,
> - Francisco
> 
> On 2025/08/29 19:56:08 Andrés Beck-Ruiz wrote:
>> Hello everyone,
>> 
>> We would like to propose CEP 53: Cassandra Rolling Restarts via Sidecar (
>> https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-53%3A+Cassandra+Rolling+Restarts+via+Sidecar
>> )
>> 
>> This CEP builds off of CEP-1
>> <https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-1%3A+Apache+Cassandra+Management+Process%28es%29+-+Deprecated>
>> and proposes a design for safe, efficient, and operator friendly rolling
>> restarts on Cassandra clusters, as well as an extensible approach for
>> persisting future cluster-wide operations in Cassandra Sidecar. We hope to
>> leverage this infrastructure in the future to implement upgrade automation.
>> 
>> We welcome all feedback and discussion. Thank you in advance for your time
>> and consideration of this proposal!
>> 
>> Best,
>> Andrés and Paulo
>> 

Re: [DISCUSS] CEP 53: Cassandra Rolling Restarts via Sidecar

[Francisco Guerrero <fr...@apache.org>]

Thanks Andrés for the CEP. This is a great contribution to the project and
aligns with the original intent of the Sidecar stated in CEP-1. I've gone
over the CEP details and it is consistent with the internals of Sidecar.

The only suggestion I have is to keep in mind the pluggability aspect of
Sidecar. For example, for the Distributed Restart portion of the work, we
should consider making interfaces that would allow us to potentially move
the responsibility of keeping the state outside of Cassandra.

Best,
- Francisco
 
On 2025/08/29 19:56:08 Andrés Beck-Ruiz wrote:
> Hello everyone,
> 
> We would like to propose CEP 53: Cassandra Rolling Restarts via Sidecar (
> https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-53%3A+Cassandra+Rolling+Restarts+via+Sidecar
> )
> 
> This CEP builds off of CEP-1
> <https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-1%3A+Apache+Cassandra+Management+Process%28es%29+-+Deprecated>
> and proposes a design for safe, efficient, and operator friendly rolling
> restarts on Cassandra clusters, as well as an extensible approach for
> persisting future cluster-wide operations in Cassandra Sidecar. We hope to
> leverage this infrastructure in the future to implement upgrade automation.
> 
> We welcome all feedback and discussion. Thank you in advance for your time
> and consideration of this proposal!
> 
> Best,
> Andrés and Paulo
> 

Re: [DISCUSS] CEP 53: Cassandra Rolling Restarts via Sidecar

[Andrés Beck-Ruiz <an...@gmail.com>]

Hi Chris-- yes, the focus of the initial design is ensuring that Sidecar
can conduct a restart without affecting any downstream clients and while
maintaining availability.

For the initial implementation, I think that availability should be
prioritized. However, I do think that it should be possible for an operator
to skip health checks and/or allow for up to all Cassandra instances to be
restarted in parallel, if users value restart speed over availability--
this could be implemented with an "unsafeRestart" flag, maybe.

I think this should be left out of scope for the initial implementation,
but can be added in a future iteration.

On Mon, Sep 29, 2025 at 2:15 PM Christopher Bradford <br...@gmail.com>
wrote:

> Hi Andrés,
>
> Non-blocking commentary on my part:
>
> I see the terms "distributed restart" and "rolling restart" used both in
> the CEP and in this thread. There are subtle differences between the two
> which should be explored. IMO a "*rolling*" restart focuses on
> availability, limiting any potential impact to downstream clients.
> Separately, a *distributed* restart could be perceived as a coordinated
> effort to restart a cluster with optional requirements around availability.
> I have worked with some users where the application of changes quickly is
> more important than availability to clients. In some cases this is handled
> by bringing down the cluster immediately, applying the change, and using a
> fast-path start (seeds first, then all other nodes simultaneously).
>
> Given the CEP title and implementation, focus on availability appears to
> be the clear goal here. Is there a follow-up goal for a "fast-path" restart
> which does not provide the availability guarantee? Maybe I missed if this
> is possible today in the proposed implementation.
>
> ~Chris
>
> Christopher Bradford
>
>
>
> On Mon, Sep 29, 2025 at 12:25 PM Andrés Beck-Ruiz <
> andresbeckruiz@gmail.com> wrote:
>
>> Hello all, I will be bringing CEP-53 to a vote on Wednesday unless there
>> are any more outstanding comments. Thanks!
>>
>>
>> On Wed, Sep 24, 2025 at 5:49 PM Andrés Beck-Ruiz <
>> andresbeckruiz@gmail.com> wrote:
>>
>>> Hey everyone, thanks again for all of the feedback. I've updated CEP-53
>>> <https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-53%3A+Cassandra+Rolling+Restarts+via+Sidecar>
>>> to discuss how rolling restarts and future cluster-wide operations can be
>>> integrated into the existing operational job framework. For the most part,
>>> you can find these updates in the OperationalJob Framework
>>> <https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-53%3A+Cassandra+Rolling+Restarts+via+Sidecar#CEP53:CassandraRollingRestartsviaSidecar-OperationalJobFramework>,
>>> API
>>> <https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-53%3A+Cassandra+Rolling+Restarts+via+Sidecar#CEP53:CassandraRollingRestartsviaSidecar-API>,
>>> and New or Changed Public Interfaces
>>> <https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-53%3A+Cassandra+Rolling+Restarts+via+Sidecar#CEP53:CassandraRollingRestartsviaSidecar-NeworChangedPublicInterfaces>
>>> sections. I also addressed any outstanding comments or questions that were
>>> raised in this discussion thread.
>>>
>>> I'll give folks a few days to look over the CEP again in case there are
>>> any other questions, and hope to start the voting process next week.
>>>
>>> On Fri, Aug 29, 2025 at 3:56 PM Andrés Beck-Ruiz <
>>> andresbeckruiz@gmail.com> wrote:
>>>
>>>> Hello everyone,
>>>>
>>>> We would like to propose CEP 53: Cassandra Rolling Restarts via Sidecar
>>>> (
>>>> https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-53%3A+Cassandra+Rolling+Restarts+via+Sidecar
>>>> )
>>>>
>>>> This CEP builds off of CEP-1
>>>> <https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-1%3A+Apache+Cassandra+Management+Process%28es%29+-+Deprecated>
>>>> and proposes a design for safe, efficient, and operator friendly rolling
>>>> restarts on Cassandra clusters, as well as an extensible approach for
>>>> persisting future cluster-wide operations in Cassandra Sidecar. We hope to
>>>> leverage this infrastructure in the future to implement upgrade automation.
>>>>
>>>> We welcome all feedback and discussion. Thank you in advance for your
>>>> time and consideration of this proposal!
>>>>
>>>> Best,
>>>> Andrés and Paulo
>>>>
>>>

Re: [DISCUSS] CEP 53: Cassandra Rolling Restarts via Sidecar

[Christopher Bradford <br...@gmail.com>]

Hi Andrés,

Non-blocking commentary on my part:

I see the terms "distributed restart" and "rolling restart" used both in
the CEP and in this thread. There are subtle differences between the two
which should be explored. IMO a "*rolling*" restart focuses on
availability, limiting any potential impact to downstream clients.
Separately, a *distributed* restart could be perceived as a coordinated
effort to restart a cluster with optional requirements around availability.
I have worked with some users where the application of changes quickly is
more important than availability to clients. In some cases this is handled
by bringing down the cluster immediately, applying the change, and using a
fast-path start (seeds first, then all other nodes simultaneously).

Given the CEP title and implementation, focus on availability appears to be
the clear goal here. Is there a follow-up goal for a "fast-path" restart
which does not provide the availability guarantee? Maybe I missed if this
is possible today in the proposed implementation.

~Chris

Christopher Bradford



On Mon, Sep 29, 2025 at 12:25 PM Andrés Beck-Ruiz <an...@gmail.com>
wrote:

> Hello all, I will be bringing CEP-53 to a vote on Wednesday unless there
> are any more outstanding comments. Thanks!
>
>
> On Wed, Sep 24, 2025 at 5:49 PM Andrés Beck-Ruiz <an...@gmail.com>
> wrote:
>
>> Hey everyone, thanks again for all of the feedback. I've updated CEP-53
>> <https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-53%3A+Cassandra+Rolling+Restarts+via+Sidecar>
>> to discuss how rolling restarts and future cluster-wide operations can be
>> integrated into the existing operational job framework. For the most part,
>> you can find these updates in the OperationalJob Framework
>> <https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-53%3A+Cassandra+Rolling+Restarts+via+Sidecar#CEP53:CassandraRollingRestartsviaSidecar-OperationalJobFramework>,
>> API
>> <https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-53%3A+Cassandra+Rolling+Restarts+via+Sidecar#CEP53:CassandraRollingRestartsviaSidecar-API>,
>> and New or Changed Public Interfaces
>> <https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-53%3A+Cassandra+Rolling+Restarts+via+Sidecar#CEP53:CassandraRollingRestartsviaSidecar-NeworChangedPublicInterfaces>
>> sections. I also addressed any outstanding comments or questions that were
>> raised in this discussion thread.
>>
>> I'll give folks a few days to look over the CEP again in case there are
>> any other questions, and hope to start the voting process next week.
>>
>> On Fri, Aug 29, 2025 at 3:56 PM Andrés Beck-Ruiz <
>> andresbeckruiz@gmail.com> wrote:
>>
>>> Hello everyone,
>>>
>>> We would like to propose CEP 53: Cassandra Rolling Restarts via Sidecar (
>>> https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-53%3A+Cassandra+Rolling+Restarts+via+Sidecar
>>> )
>>>
>>> This CEP builds off of CEP-1
>>> <https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-1%3A+Apache+Cassandra+Management+Process%28es%29+-+Deprecated>
>>> and proposes a design for safe, efficient, and operator friendly rolling
>>> restarts on Cassandra clusters, as well as an extensible approach for
>>> persisting future cluster-wide operations in Cassandra Sidecar. We hope to
>>> leverage this infrastructure in the future to implement upgrade automation.
>>>
>>> We welcome all feedback and discussion. Thank you in advance for your
>>> time and consideration of this proposal!
>>>
>>> Best,
>>> Andrés and Paulo
>>>
>>

Re: [DISCUSS] CEP 53: Cassandra Rolling Restarts via Sidecar

[Andrés Beck-Ruiz <an...@gmail.com>]

Hello all, I will be bringing CEP-53 to a vote on Wednesday unless there
are any more outstanding comments. Thanks!


On Wed, Sep 24, 2025 at 5:49 PM Andrés Beck-Ruiz <an...@gmail.com>
wrote:

> Hey everyone, thanks again for all of the feedback. I've updated CEP-53
> <https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-53%3A+Cassandra+Rolling+Restarts+via+Sidecar>
> to discuss how rolling restarts and future cluster-wide operations can be
> integrated into the existing operational job framework. For the most part,
> you can find these updates in the OperationalJob Framework
> <https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-53%3A+Cassandra+Rolling+Restarts+via+Sidecar#CEP53:CassandraRollingRestartsviaSidecar-OperationalJobFramework>,
> API
> <https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-53%3A+Cassandra+Rolling+Restarts+via+Sidecar#CEP53:CassandraRollingRestartsviaSidecar-API>,
> and New or Changed Public Interfaces
> <https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-53%3A+Cassandra+Rolling+Restarts+via+Sidecar#CEP53:CassandraRollingRestartsviaSidecar-NeworChangedPublicInterfaces>
> sections. I also addressed any outstanding comments or questions that were
> raised in this discussion thread.
>
> I'll give folks a few days to look over the CEP again in case there are
> any other questions, and hope to start the voting process next week.
>
> On Fri, Aug 29, 2025 at 3:56 PM Andrés Beck-Ruiz <an...@gmail.com>
> wrote:
>
>> Hello everyone,
>>
>> We would like to propose CEP 53: Cassandra Rolling Restarts via Sidecar (
>> https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-53%3A+Cassandra+Rolling+Restarts+via+Sidecar
>> )
>>
>> This CEP builds off of CEP-1
>> <https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-1%3A+Apache+Cassandra+Management+Process%28es%29+-+Deprecated>
>> and proposes a design for safe, efficient, and operator friendly rolling
>> restarts on Cassandra clusters, as well as an extensible approach for
>> persisting future cluster-wide operations in Cassandra Sidecar. We hope to
>> leverage this infrastructure in the future to implement upgrade automation.
>>
>> We welcome all feedback and discussion. Thank you in advance for your
>> time and consideration of this proposal!
>>
>> Best,
>> Andrés and Paulo
>>
>

Re: [DISCUSS] CEP 53: Cassandra Rolling Restarts via Sidecar

[Andrés Beck-Ruiz <an...@gmail.com>]

Hey everyone, thanks again for all of the feedback. I've updated CEP-53
<https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-53%3A+Cassandra+Rolling+Restarts+via+Sidecar>
to discuss how rolling restarts and future cluster-wide operations can be
integrated into the existing operational job framework. For the most part,
you can find these updates in the OperationalJob Framework
<https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-53%3A+Cassandra+Rolling+Restarts+via+Sidecar#CEP53:CassandraRollingRestartsviaSidecar-OperationalJobFramework>,
API
<https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-53%3A+Cassandra+Rolling+Restarts+via+Sidecar#CEP53:CassandraRollingRestartsviaSidecar-API>,
and New or Changed Public Interfaces
<https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-53%3A+Cassandra+Rolling+Restarts+via+Sidecar#CEP53:CassandraRollingRestartsviaSidecar-NeworChangedPublicInterfaces>
sections. I also addressed any outstanding comments or questions that were
raised in this discussion thread.

I'll give folks a few days to look over the CEP again in case there are any
other questions, and hope to start the voting process next week.

On Fri, Aug 29, 2025 at 3:56 PM Andrés Beck-Ruiz <an...@gmail.com>
wrote:

> Hello everyone,
>
> We would like to propose CEP 53: Cassandra Rolling Restarts via Sidecar (
> https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-53%3A+Cassandra+Rolling+Restarts+via+Sidecar
> )
>
> This CEP builds off of CEP-1
> <https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-1%3A+Apache+Cassandra+Management+Process%28es%29+-+Deprecated>
> and proposes a design for safe, efficient, and operator friendly rolling
> restarts on Cassandra clusters, as well as an extensible approach for
> persisting future cluster-wide operations in Cassandra Sidecar. We hope to
> leverage this infrastructure in the future to implement upgrade automation.
>
> We welcome all feedback and discussion. Thank you in advance for your time
> and consideration of this proposal!
>
> Best,
> Andrés and Paulo
>

Re: [DISCUSS] CEP 53: Cassandra Rolling Restarts via Sidecar

[Andrés Beck-Ruiz <an...@gmail.com>]

> A hard dependency on Cassandra instances would also create challenges for
startup scenarios
> where operators need to start C* instances for the first time—such as
brand new cluster
> creation, replacing an existing node, or starting nodes in a new
datacenter.

Are you referring to depending on Cassandra for storing and
communicating operation state? The goal of an operation state management
interface is that Cassandra wouldn't need to be a dependency, as we can add
implementations for different storages in the future. Any operation storage
implementation would also track operations beyond just restarts.

Furthermore, we can add cluster_name to the partition keys of the tables in
the Cassandra storage implementation. This would allow a separate cluster
from the one being managed by the Cassandra Sidecar to be used for
operation state storage, so if any operation fails we can still communicate
with the storage cluster which isn't being operated on.

You can also bounce Cassandra nodes with the lifecycle APIs which don't
depend on operation state storage or communication with the Cassandra
instance being managed (
https://issues.apache.org/jira/projects/CASSSIDECAR/issues/CASSSIDECAR-266?filter=allopenissues
).

On Wed, Sep 10, 2025 at 3:05 PM Venkata Harikrishna Nukala <
n.v.harikrishna.apache@gmail.com> wrote:

>
> > A small question from my side: I see that the underlying assumption is
>> that
>>
>> > Sidecar is able to query Cassandra instances before bouncing/recognizing
>>
>> > the bounce. What if it could not communicate with the Cassandra instance
>>
>> > (e.g., binary protocol disabled, C* process experiencing issues, or C*
>>
>> > process starting as part of a new DC)?
>>
>> This would fall under scenario #2 in the Error Handling
>> <https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-53%3A+Cassandra+Rolling+Restarts+via+Sidecar#CEP53:CassandraRollingRestartsviaSidecar-ErrorHandling> section
>> of the CEP. If a Sidecar instance can’t communicate with Cassandra, after a
>> configurable timeout and amount of retries, the Sidecar instance should
>> mark the job as failed.
>>
>
> Returning an error is indeed appropriate for communication failures, but
> this means operators would need to find alternative ways to bounce nodes
> and couldn't rely on these APIs consistently. A hard dependency on
> Cassandra instances would also create challenges for startup scenarios
> where operators need to start C* instances for the first time—such as brand
> new cluster creation, replacing an existing node, or starting nodes in a
> new datacenter.
>
> If the responsibility of updating Cassandra tables is delegated to a
> plugin or default implementation for communicating bounces, it would allow
> others to implement their own solutions more freely (potentially bypassing
> C* table updates for bounce communication). I understand that this CEP
> focuses on restart operations (stop + start), but I believe an interface
> and plugin pattern could address a broader range of operational challenges.
>
> On Mon, Sep 8, 2025 at 10:22 PM Andrés Beck-Ruiz <an...@gmail.com>
> wrote:
>
>> Our original thinking was that we could store node UUIDs but return IP
>> addresses so that operators can better identify Cassandra nodes, but I
>> recognize that it could also cause confusion as the address is not a
>> persistent node identifier. I agree that the benefit of unifying the API
>> and schema by using node UUIDs exclusively outweighs the cost of API
>> ergonomics. I can update the CEP to reflect this unless there are differing
>> opinions.
>>
>> Best,
>> Andrés
>>
>> On Mon, Sep 8, 2025 at 5:03 AM Sam Tunnicliffe <sa...@beobal.com> wrote:
>>
>>> Hi Andrés, thanks for this comprehensive CEP.
>>>
>>> I have a query about the representation of nodes in the various tables
>>> and responses.
>>>
>>> In the sidecar_internal.cluster_ops_node_state table, "We store the node
>>> UUID instead of IP address to guarantee that the correct Cassandra nodes
>>> are restarted in case any node moves to another host.". However, in the
>>> main sidecar_internal.cluster_ops table the nodes participating in the
>>> operation are represented as a list of IP addresses. Likewise, in the
>>> sample HTTP responses nodes always appear to be identified by their
>>> address, not ID.
>>>
>>> It's true that operators are more accustomed to dealing with addresses
>>> than IDs but it's equally the case that the address is not a persistent
>>> node identifier, as noted in this CEP. For that reason, in C* trunk the
>>> emphasis is shifting to lean more on node IDs, so I feel it would be a
>>> retrograde step to introduce new APIs which include only addresses. Could
>>> the schema and API responses in this CEP be unified in some way, either by
>>> using IDs exclusively or by extending the node representation to something
>>> that can incorporate both an ID and address?
>>>
>>> Among other things, Accord relies on persistent node IDs for correctness
>>> and a unique and persistent identifier is now assigned to every node as a
>>> prerequisite for it joining the cluster. This ID is a simple integer and is
>>> encoded into the node's Host ID which is the UUID available in various
>>> system tables, gossip state and nodetool commands. The initial thinking
>>> behind encoding in the Host ID was to maintain compatibility with existing
>>> tooling but at some point we will start to expose the ID directly in more
>>> places. Right now there is a vtable which shows the IDs directly,
>>> system_views.cluster_metadata_directory.
>>>
>>> Thanks,
>>> Sam
>>>
>>> > On 8 Sep 2025, at 02:36, Andrés Beck-Ruiz <an...@gmail.com>
>>> wrote:
>>> >
>>> > Hello all,
>>> >
>>> > Thanks for the feedback. I agree with the suggestions that operation
>>> state storage should be pluggable, with an initial implementation
>>> leveraging Cassandra as we have proposed. I have made edits to the
>>> Distributed Restart Design section in the CEP to reflect this.
>>> >
>>> > > As for the API, I think the question that needs to be answered is if
>>> it is
>>> > > worthwhile to have a distinction between single-node operations and
>>> > > cluster-wide operations. For example, if I wanted to restart a
>>> single node
>>> > > using the API proposed in CEP-53, I could submit a restart job with a
>>> > > single node in the “nodes” list. This provides API simplicity at the
>>> cost
>>> > > of ergonomics. It also means that all inter-sidecar communication
>>> would go
>>> > > through the proposed cluster_ops_node_state table. Personally, I
>>> think
>>> > > these are acceptable tradeoffs to provide a unified API for
>>> operations that
>>> > > is simpler for a user or operator to use and learn.
>>> >
>>> > I agree that we should provide a unified API that does not distinguish
>>> between single-node and cluster-wide operations. I think the benefit of API
>>> simplicity from a development and client perspective outweighs the cost of
>>> ergonomics.
>>> >
>>> > > A small question from my side: I see that the underlying assumption
>>> is that
>>> > > Sidecar is able to query Cassandra instances before
>>> bouncing/recognizing
>>> > > the bounce. What if it could not communicate with the Cassandra
>>> instance
>>> > > (e.g., binary protocol disabled, C* process experiencing issues, or
>>> C*
>>> > > process starting as part of a new DC)?
>>> >
>>> > This would fall under scenario #2 in the Error Handling section of the
>>> CEP. If a Sidecar instance can’t communicate with Cassandra, after a
>>> configurable timeout and amount of retries, the Sidecar instance should
>>> mark the job as failed.
>>> >
>>> > > 1. Have we considered introducing the concept of a datacenter
>>> alongside cluster?
>>> > > I imagine there will be cases where a user wants to perform a
>>> rolling restart on a
>>> > > single datacenter rather than across the entire cluster.
>>> >
>>> > I think this could be added in the future, but for this initial
>>> implementation an operator would submit the nodes part of a datacenter to
>>> restart a datacenter. I prefer providing a unified API that can handle
>>> single node and cluster (or datacenter) wide operations over separate APIs
>>> which might be easier to use in isolation but complicate development and
>>> discoverability.
>>> >
>>> > >2. Do we see this framework extending to other cluster- or
>>> datacenter-wide operations,
>>> > > such as scale-up/scale-down operations, or backups/restores, or
>>> nodetool rebuilds
>>> > > run as part of adding a new datacenter?
>>> >
>>> > Yes, our goal with this design is that it is extensible for future
>>> operations, as well as currently supported operations (such as node
>>> decommissions) that already exist in Sidecar. In the initial Cassandra
>>> storage implementation, all inter-sidecar communication and operation
>>> tracking could occur in the proposed cluster_ops_node_state table.
>>> >
>>> >
>>> > > The design seems focused on cluster/availability signals (ring
>>> stable,
>>> > > peers up), which is a great start, but doesn’t mention pluggable
>>> workload
>>> > > signals like: 1) compaction load (nodetool compactionstats) 2)
>>> netstats
>>> > > activity (nodetool netstats) 3) hints backlog / streaming pending
>>> flushes
>>> > > or memtable pressure.
>>> > > Since restarting during heavy compaction/hints can add risk, are
>>> these
>>> > > kinds of workload-aware checks in scope for the MVP, or considered
>>> future
>>> > > work?
>>> >
>>> > I agree that the health check should be pluggable as well— this was
>>> also proposed in CEP-1. For the first iteration of rolling restarts, we are
>>> thinking of providing a health check implementation that checks for all
>>> other Cassandra peers being up, and future work can add more robust health
>>> checks.
>>> >
>>> > Best,
>>> > Andrés
>>> >
>>> > On Fri, Aug 29, 2025 at 3:56 PM Andrés Beck-Ruiz <
>>> andresbeckruiz@gmail.com> wrote:
>>> > Hello everyone,
>>> >
>>> > We would like to propose CEP 53: Cassandra Rolling Restarts via
>>> Sidecar (
>>> https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-53%3A+Cassandra+Rolling+Restarts+via+Sidecar
>>> )
>>> >
>>> > This CEP builds off of CEP-1 and proposes a design for safe,
>>> efficient, and operator friendly rolling restarts on Cassandra clusters, as
>>> well as an extensible approach for persisting future cluster-wide
>>> operations in Cassandra Sidecar. We hope to leverage this infrastructure in
>>> the future to implement upgrade automation.
>>> >
>>> > We welcome all feedback and discussion. Thank you in advance for your
>>> time and consideration of this proposal!
>>> >
>>> > Best,
>>> > Andrés and Paulo
>>>
>>>

Re: [DISCUSS] CEP 53: Cassandra Rolling Restarts via Sidecar

[Venkata Harikrishna Nukala <n....@gmail.com>]

> A small question from my side: I see that the underlying assumption is
> that
>
> > Sidecar is able to query Cassandra instances before bouncing/recognizing
>
> > the bounce. What if it could not communicate with the Cassandra instance
>
> > (e.g., binary protocol disabled, C* process experiencing issues, or C*
>
> > process starting as part of a new DC)?
>
> This would fall under scenario #2 in the Error Handling
> <https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-53%3A+Cassandra+Rolling+Restarts+via+Sidecar#CEP53:CassandraRollingRestartsviaSidecar-ErrorHandling> section
> of the CEP. If a Sidecar instance can’t communicate with Cassandra, after a
> configurable timeout and amount of retries, the Sidecar instance should
> mark the job as failed.
>

Returning an error is indeed appropriate for communication failures, but
this means operators would need to find alternative ways to bounce nodes
and couldn't rely on these APIs consistently. A hard dependency on
Cassandra instances would also create challenges for startup scenarios
where operators need to start C* instances for the first time—such as brand
new cluster creation, replacing an existing node, or starting nodes in a
new datacenter.

If the responsibility of updating Cassandra tables is delegated to a plugin
or default implementation for communicating bounces, it would allow others
to implement their own solutions more freely (potentially bypassing C*
table updates for bounce communication). I understand that this CEP focuses
on restart operations (stop + start), but I believe an interface and plugin
pattern could address a broader range of operational challenges.

On Mon, Sep 8, 2025 at 10:22 PM Andrés Beck-Ruiz <an...@gmail.com>
wrote:

> Our original thinking was that we could store node UUIDs but return IP
> addresses so that operators can better identify Cassandra nodes, but I
> recognize that it could also cause confusion as the address is not a
> persistent node identifier. I agree that the benefit of unifying the API
> and schema by using node UUIDs exclusively outweighs the cost of API
> ergonomics. I can update the CEP to reflect this unless there are differing
> opinions.
>
> Best,
> Andrés
>
> On Mon, Sep 8, 2025 at 5:03 AM Sam Tunnicliffe <sa...@beobal.com> wrote:
>
>> Hi Andrés, thanks for this comprehensive CEP.
>>
>> I have a query about the representation of nodes in the various tables
>> and responses.
>>
>> In the sidecar_internal.cluster_ops_node_state table, "We store the node
>> UUID instead of IP address to guarantee that the correct Cassandra nodes
>> are restarted in case any node moves to another host.". However, in the
>> main sidecar_internal.cluster_ops table the nodes participating in the
>> operation are represented as a list of IP addresses. Likewise, in the
>> sample HTTP responses nodes always appear to be identified by their
>> address, not ID.
>>
>> It's true that operators are more accustomed to dealing with addresses
>> than IDs but it's equally the case that the address is not a persistent
>> node identifier, as noted in this CEP. For that reason, in C* trunk the
>> emphasis is shifting to lean more on node IDs, so I feel it would be a
>> retrograde step to introduce new APIs which include only addresses. Could
>> the schema and API responses in this CEP be unified in some way, either by
>> using IDs exclusively or by extending the node representation to something
>> that can incorporate both an ID and address?
>>
>> Among other things, Accord relies on persistent node IDs for correctness
>> and a unique and persistent identifier is now assigned to every node as a
>> prerequisite for it joining the cluster. This ID is a simple integer and is
>> encoded into the node's Host ID which is the UUID available in various
>> system tables, gossip state and nodetool commands. The initial thinking
>> behind encoding in the Host ID was to maintain compatibility with existing
>> tooling but at some point we will start to expose the ID directly in more
>> places. Right now there is a vtable which shows the IDs directly,
>> system_views.cluster_metadata_directory.
>>
>> Thanks,
>> Sam
>>
>> > On 8 Sep 2025, at 02:36, Andrés Beck-Ruiz <an...@gmail.com>
>> wrote:
>> >
>> > Hello all,
>> >
>> > Thanks for the feedback. I agree with the suggestions that operation
>> state storage should be pluggable, with an initial implementation
>> leveraging Cassandra as we have proposed. I have made edits to the
>> Distributed Restart Design section in the CEP to reflect this.
>> >
>> > > As for the API, I think the question that needs to be answered is if
>> it is
>> > > worthwhile to have a distinction between single-node operations and
>> > > cluster-wide operations. For example, if I wanted to restart a single
>> node
>> > > using the API proposed in CEP-53, I could submit a restart job with a
>> > > single node in the “nodes” list. This provides API simplicity at the
>> cost
>> > > of ergonomics. It also means that all inter-sidecar communication
>> would go
>> > > through the proposed cluster_ops_node_state table. Personally, I think
>> > > these are acceptable tradeoffs to provide a unified API for
>> operations that
>> > > is simpler for a user or operator to use and learn.
>> >
>> > I agree that we should provide a unified API that does not distinguish
>> between single-node and cluster-wide operations. I think the benefit of API
>> simplicity from a development and client perspective outweighs the cost of
>> ergonomics.
>> >
>> > > A small question from my side: I see that the underlying assumption
>> is that
>> > > Sidecar is able to query Cassandra instances before
>> bouncing/recognizing
>> > > the bounce. What if it could not communicate with the Cassandra
>> instance
>> > > (e.g., binary protocol disabled, C* process experiencing issues, or C*
>> > > process starting as part of a new DC)?
>> >
>> > This would fall under scenario #2 in the Error Handling section of the
>> CEP. If a Sidecar instance can’t communicate with Cassandra, after a
>> configurable timeout and amount of retries, the Sidecar instance should
>> mark the job as failed.
>> >
>> > > 1. Have we considered introducing the concept of a datacenter
>> alongside cluster?
>> > > I imagine there will be cases where a user wants to perform a rolling
>> restart on a
>> > > single datacenter rather than across the entire cluster.
>> >
>> > I think this could be added in the future, but for this initial
>> implementation an operator would submit the nodes part of a datacenter to
>> restart a datacenter. I prefer providing a unified API that can handle
>> single node and cluster (or datacenter) wide operations over separate APIs
>> which might be easier to use in isolation but complicate development and
>> discoverability.
>> >
>> > >2. Do we see this framework extending to other cluster- or
>> datacenter-wide operations,
>> > > such as scale-up/scale-down operations, or backups/restores, or
>> nodetool rebuilds
>> > > run as part of adding a new datacenter?
>> >
>> > Yes, our goal with this design is that it is extensible for future
>> operations, as well as currently supported operations (such as node
>> decommissions) that already exist in Sidecar. In the initial Cassandra
>> storage implementation, all inter-sidecar communication and operation
>> tracking could occur in the proposed cluster_ops_node_state table.
>> >
>> >
>> > > The design seems focused on cluster/availability signals (ring stable,
>> > > peers up), which is a great start, but doesn’t mention pluggable
>> workload
>> > > signals like: 1) compaction load (nodetool compactionstats) 2)
>> netstats
>> > > activity (nodetool netstats) 3) hints backlog / streaming pending
>> flushes
>> > > or memtable pressure.
>> > > Since restarting during heavy compaction/hints can add risk, are these
>> > > kinds of workload-aware checks in scope for the MVP, or considered
>> future
>> > > work?
>> >
>> > I agree that the health check should be pluggable as well— this was
>> also proposed in CEP-1. For the first iteration of rolling restarts, we are
>> thinking of providing a health check implementation that checks for all
>> other Cassandra peers being up, and future work can add more robust health
>> checks.
>> >
>> > Best,
>> > Andrés
>> >
>> > On Fri, Aug 29, 2025 at 3:56 PM Andrés Beck-Ruiz <
>> andresbeckruiz@gmail.com> wrote:
>> > Hello everyone,
>> >
>> > We would like to propose CEP 53: Cassandra Rolling Restarts via Sidecar
>> (
>> https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-53%3A+Cassandra+Rolling+Restarts+via+Sidecar
>> )
>> >
>> > This CEP builds off of CEP-1 and proposes a design for safe, efficient,
>> and operator friendly rolling restarts on Cassandra clusters, as well as an
>> extensible approach for persisting future cluster-wide operations in
>> Cassandra Sidecar. We hope to leverage this infrastructure in the future to
>> implement upgrade automation.
>> >
>> > We welcome all feedback and discussion. Thank you in advance for your
>> time and consideration of this proposal!
>> >
>> > Best,
>> > Andrés and Paulo
>>
>>

Re: [DISCUSS] CEP 53: Cassandra Rolling Restarts via Sidecar

[Andrés Beck-Ruiz <an...@gmail.com>]

Our original thinking was that we could store node UUIDs but return IP
addresses so that operators can better identify Cassandra nodes, but I
recognize that it could also cause confusion as the address is not a
persistent node identifier. I agree that the benefit of unifying the API
and schema by using node UUIDs exclusively outweighs the cost of API
ergonomics. I can update the CEP to reflect this unless there are differing
opinions.

Best,
Andrés

On Mon, Sep 8, 2025 at 5:03 AM Sam Tunnicliffe <sa...@beobal.com> wrote:

> Hi Andrés, thanks for this comprehensive CEP.
>
> I have a query about the representation of nodes in the various tables and
> responses.
>
> In the sidecar_internal.cluster_ops_node_state table, "We store the node
> UUID instead of IP address to guarantee that the correct Cassandra nodes
> are restarted in case any node moves to another host.". However, in the
> main sidecar_internal.cluster_ops table the nodes participating in the
> operation are represented as a list of IP addresses. Likewise, in the
> sample HTTP responses nodes always appear to be identified by their
> address, not ID.
>
> It's true that operators are more accustomed to dealing with addresses
> than IDs but it's equally the case that the address is not a persistent
> node identifier, as noted in this CEP. For that reason, in C* trunk the
> emphasis is shifting to lean more on node IDs, so I feel it would be a
> retrograde step to introduce new APIs which include only addresses. Could
> the schema and API responses in this CEP be unified in some way, either by
> using IDs exclusively or by extending the node representation to something
> that can incorporate both an ID and address?
>
> Among other things, Accord relies on persistent node IDs for correctness
> and a unique and persistent identifier is now assigned to every node as a
> prerequisite for it joining the cluster. This ID is a simple integer and is
> encoded into the node's Host ID which is the UUID available in various
> system tables, gossip state and nodetool commands. The initial thinking
> behind encoding in the Host ID was to maintain compatibility with existing
> tooling but at some point we will start to expose the ID directly in more
> places. Right now there is a vtable which shows the IDs directly,
> system_views.cluster_metadata_directory.
>
> Thanks,
> Sam
>
> > On 8 Sep 2025, at 02:36, Andrés Beck-Ruiz <an...@gmail.com>
> wrote:
> >
> > Hello all,
> >
> > Thanks for the feedback. I agree with the suggestions that operation
> state storage should be pluggable, with an initial implementation
> leveraging Cassandra as we have proposed. I have made edits to the
> Distributed Restart Design section in the CEP to reflect this.
> >
> > > As for the API, I think the question that needs to be answered is if
> it is
> > > worthwhile to have a distinction between single-node operations and
> > > cluster-wide operations. For example, if I wanted to restart a single
> node
> > > using the API proposed in CEP-53, I could submit a restart job with a
> > > single node in the “nodes” list. This provides API simplicity at the
> cost
> > > of ergonomics. It also means that all inter-sidecar communication
> would go
> > > through the proposed cluster_ops_node_state table. Personally, I think
> > > these are acceptable tradeoffs to provide a unified API for operations
> that
> > > is simpler for a user or operator to use and learn.
> >
> > I agree that we should provide a unified API that does not distinguish
> between single-node and cluster-wide operations. I think the benefit of API
> simplicity from a development and client perspective outweighs the cost of
> ergonomics.
> >
> > > A small question from my side: I see that the underlying assumption is
> that
> > > Sidecar is able to query Cassandra instances before
> bouncing/recognizing
> > > the bounce. What if it could not communicate with the Cassandra
> instance
> > > (e.g., binary protocol disabled, C* process experiencing issues, or C*
> > > process starting as part of a new DC)?
> >
> > This would fall under scenario #2 in the Error Handling section of the
> CEP. If a Sidecar instance can’t communicate with Cassandra, after a
> configurable timeout and amount of retries, the Sidecar instance should
> mark the job as failed.
> >
> > > 1. Have we considered introducing the concept of a datacenter
> alongside cluster?
> > > I imagine there will be cases where a user wants to perform a rolling
> restart on a
> > > single datacenter rather than across the entire cluster.
> >
> > I think this could be added in the future, but for this initial
> implementation an operator would submit the nodes part of a datacenter to
> restart a datacenter. I prefer providing a unified API that can handle
> single node and cluster (or datacenter) wide operations over separate APIs
> which might be easier to use in isolation but complicate development and
> discoverability.
> >
> > >2. Do we see this framework extending to other cluster- or
> datacenter-wide operations,
> > > such as scale-up/scale-down operations, or backups/restores, or
> nodetool rebuilds
> > > run as part of adding a new datacenter?
> >
> > Yes, our goal with this design is that it is extensible for future
> operations, as well as currently supported operations (such as node
> decommissions) that already exist in Sidecar. In the initial Cassandra
> storage implementation, all inter-sidecar communication and operation
> tracking could occur in the proposed cluster_ops_node_state table.
> >
> >
> > > The design seems focused on cluster/availability signals (ring stable,
> > > peers up), which is a great start, but doesn’t mention pluggable
> workload
> > > signals like: 1) compaction load (nodetool compactionstats) 2) netstats
> > > activity (nodetool netstats) 3) hints backlog / streaming pending
> flushes
> > > or memtable pressure.
> > > Since restarting during heavy compaction/hints can add risk, are these
> > > kinds of workload-aware checks in scope for the MVP, or considered
> future
> > > work?
> >
> > I agree that the health check should be pluggable as well— this was also
> proposed in CEP-1. For the first iteration of rolling restarts, we are
> thinking of providing a health check implementation that checks for all
> other Cassandra peers being up, and future work can add more robust health
> checks.
> >
> > Best,
> > Andrés
> >
> > On Fri, Aug 29, 2025 at 3:56 PM Andrés Beck-Ruiz <
> andresbeckruiz@gmail.com> wrote:
> > Hello everyone,
> >
> > We would like to propose CEP 53: Cassandra Rolling Restarts via Sidecar (
> https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-53%3A+Cassandra+Rolling+Restarts+via+Sidecar
> )
> >
> > This CEP builds off of CEP-1 and proposes a design for safe, efficient,
> and operator friendly rolling restarts on Cassandra clusters, as well as an
> extensible approach for persisting future cluster-wide operations in
> Cassandra Sidecar. We hope to leverage this infrastructure in the future to
> implement upgrade automation.
> >
> > We welcome all feedback and discussion. Thank you in advance for your
> time and consideration of this proposal!
> >
> > Best,
> > Andrés and Paulo
>
>

Re: [DISCUSS] CEP 53: Cassandra Rolling Restarts via Sidecar

[Sam Tunnicliffe <sa...@beobal.com>]

Hi Andrés, thanks for this comprehensive CEP. 

I have a query about the representation of nodes in the various tables and responses. 

In the sidecar_internal.cluster_ops_node_state table, "We store the node UUID instead of IP address to guarantee that the correct Cassandra nodes are restarted in case any node moves to another host.". However, in the main sidecar_internal.cluster_ops table the nodes participating in the operation are represented as a list of IP addresses. Likewise, in the sample HTTP responses nodes always appear to be identified by their address, not ID. 

It's true that operators are more accustomed to dealing with addresses than IDs but it's equally the case that the address is not a persistent node identifier, as noted in this CEP. For that reason, in C* trunk the emphasis is shifting to lean more on node IDs, so I feel it would be a retrograde step to introduce new APIs which include only addresses. Could the schema and API responses in this CEP be unified in some way, either by using IDs exclusively or by extending the node representation to something that can incorporate both an ID and address? 

Among other things, Accord relies on persistent node IDs for correctness and a unique and persistent identifier is now assigned to every node as a prerequisite for it joining the cluster. This ID is a simple integer and is encoded into the node's Host ID which is the UUID available in various system tables, gossip state and nodetool commands. The initial thinking behind encoding in the Host ID was to maintain compatibility with existing tooling but at some point we will start to expose the ID directly in more places. Right now there is a vtable which shows the IDs directly, system_views.cluster_metadata_directory.     

Thanks, 
Sam

> On 8 Sep 2025, at 02:36, Andrés Beck-Ruiz <an...@gmail.com> wrote:
> 
> Hello all, 
> 
> Thanks for the feedback. I agree with the suggestions that operation state storage should be pluggable, with an initial implementation leveraging Cassandra as we have proposed. I have made edits to the Distributed Restart Design section in the CEP to reflect this.
> 
> > As for the API, I think the question that needs to be answered is if it is
> > worthwhile to have a distinction between single-node operations and
> > cluster-wide operations. For example, if I wanted to restart a single node
> > using the API proposed in CEP-53, I could submit a restart job with a
> > single node in the “nodes” list. This provides API simplicity at the cost
> > of ergonomics. It also means that all inter-sidecar communication would go
> > through the proposed cluster_ops_node_state table. Personally, I think
> > these are acceptable tradeoffs to provide a unified API for operations that
> > is simpler for a user or operator to use and learn.
> 
> I agree that we should provide a unified API that does not distinguish between single-node and cluster-wide operations. I think the benefit of API simplicity from a development and client perspective outweighs the cost of ergonomics.
> 
> > A small question from my side: I see that the underlying assumption is that
> > Sidecar is able to query Cassandra instances before bouncing/recognizing
> > the bounce. What if it could not communicate with the Cassandra instance
> > (e.g., binary protocol disabled, C* process experiencing issues, or C*
> > process starting as part of a new DC)?
> 
> This would fall under scenario #2 in the Error Handling section of the CEP. If a Sidecar instance can’t communicate with Cassandra, after a configurable timeout and amount of retries, the Sidecar instance should mark the job as failed. 
> 
> > 1. Have we considered introducing the concept of a datacenter alongside cluster?
> > I imagine there will be cases where a user wants to perform a rolling restart on a
> > single datacenter rather than across the entire cluster. 
> 
> I think this could be added in the future, but for this initial implementation an operator would submit the nodes part of a datacenter to restart a datacenter. I prefer providing a unified API that can handle single node and cluster (or datacenter) wide operations over separate APIs which might be easier to use in isolation but complicate development and discoverability.
> 
> >2. Do we see this framework extending to other cluster- or datacenter-wide operations,
> > such as scale-up/scale-down operations, or backups/restores, or nodetool rebuilds
> > run as part of adding a new datacenter?
> 
> Yes, our goal with this design is that it is extensible for future operations, as well as currently supported operations (such as node decommissions) that already exist in Sidecar. In the initial Cassandra storage implementation, all inter-sidecar communication and operation tracking could occur in the proposed cluster_ops_node_state table.
> 
> 
> > The design seems focused on cluster/availability signals (ring stable,
> > peers up), which is a great start, but doesn’t mention pluggable workload
> > signals like: 1) compaction load (nodetool compactionstats) 2) netstats
> > activity (nodetool netstats) 3) hints backlog / streaming pending flushes
> > or memtable pressure.
> > Since restarting during heavy compaction/hints can add risk, are these
> > kinds of workload-aware checks in scope for the MVP, or considered future
> > work?
> 
> I agree that the health check should be pluggable as well— this was also proposed in CEP-1. For the first iteration of rolling restarts, we are thinking of providing a health check implementation that checks for all other Cassandra peers being up, and future work can add more robust health checks.
> 
> Best,
> Andrés
> 
> On Fri, Aug 29, 2025 at 3:56 PM Andrés Beck-Ruiz <an...@gmail.com> wrote:
> Hello everyone,
> 
> We would like to propose CEP 53: Cassandra Rolling Restarts via Sidecar (https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-53%3A+Cassandra+Rolling+Restarts+via+Sidecar)
> 
> This CEP builds off of CEP-1 and proposes a design for safe, efficient, and operator friendly rolling restarts on Cassandra clusters, as well as an extensible approach for persisting future cluster-wide operations in Cassandra Sidecar. We hope to leverage this infrastructure in the future to implement upgrade automation.
> 
> We welcome all feedback and discussion. Thank you in advance for your time and consideration of this proposal!
> 
> Best,
> Andrés and Paulo

Re: [DISCUSS] CEP 53: Cassandra Rolling Restarts via Sidecar

[Andrés Beck-Ruiz <an...@gmail.com>]

Hello all,


Thanks for the feedback. I agree with the suggestions that operation state
storage should be pluggable, with an initial implementation leveraging
Cassandra as we have proposed. I have made edits to the Distributed Restart
Design
<https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-53%3A+Cassandra+Rolling+Restarts+via+Sidecar#CEP53:CassandraRollingRestartsviaSidecar-DistributedRestartDesign>
section
in the CEP to reflect this.

> As for the API, I think the question that needs to be answered is if it is

> worthwhile to have a distinction between single-node operations and

> cluster-wide operations. For example, if I wanted to restart a single node

> using the API proposed in CEP-53, I could submit a restart job with a

> single node in the “nodes” list. This provides API simplicity at the cost

> of ergonomics. It also means that all inter-sidecar communication would go

> through the proposed cluster_ops_node_state table. Personally, I think

> these are acceptable tradeoffs to provide a unified API for operations
that

> is simpler for a user or operator to use and learn.

I agree that we should provide a unified API that does not distinguish
between single-node and cluster-wide operations. I think the benefit of API
simplicity from a development and client perspective outweighs the cost of
ergonomics.

> A small question from my side: I see that the underlying assumption is
that

> Sidecar is able to query Cassandra instances before bouncing/recognizing

> the bounce. What if it could not communicate with the Cassandra instance

> (e.g., binary protocol disabled, C* process experiencing issues, or C*

> process starting as part of a new DC)?

This would fall under scenario #2 in the Error Handling
<https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-53%3A+Cassandra+Rolling+Restarts+via+Sidecar#CEP53:CassandraRollingRestartsviaSidecar-ErrorHandling>
section
of the CEP. If a Sidecar instance can’t communicate with Cassandra, after a
configurable timeout and amount of retries, the Sidecar instance should
mark the job as failed.


> 1. Have we considered introducing the concept of a datacenter alongside
cluster?
> I imagine there will be cases where a user wants to perform a rolling
restart on a
> single datacenter rather than across the entire cluster.


I think this could be added in the future, but for this initial
implementation an operator would submit the nodes part of a datacenter to
restart a datacenter. I prefer providing a unified API that can handle
single node and cluster (or datacenter) wide operations over separate APIs
which might be easier to use in isolation but complicate development and
discoverability.

>2. Do we see this framework extending to other cluster- or datacenter-wide
operations,
> such as scale-up/scale-down operations, or backups/restores, or nodetool
rebuilds

> run as part of adding a new datacenter?


Yes, our goal with this design is that it is extensible for future
operations, as well as currently supported operations (such as node
decommissions) that already exist in Sidecar. In the initial Cassandra
storage implementation, all inter-sidecar communication and operation
tracking could occur in the proposed cluster_ops_node_state table.


> The design seems focused on cluster/availability signals (ring stable,

> peers up), which is a great start, but doesn’t mention pluggable workload

> signals like: 1) compaction load (nodetool compactionstats) 2) netstats

> activity (nodetool netstats) 3) hints backlog / streaming pending flushes

> or memtable pressure.

> Since restarting during heavy compaction/hints can add risk, are these

> kinds of workload-aware checks in scope for the MVP, or considered future

> work?

I agree that the health check should be pluggable as well— this was also
proposed in CEP-1
<https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=95652224#CEP1:ApacheCassandraManagementProcess(es)Deprecated-ProposedScope.2>.
For the first iteration of rolling restarts, we are thinking of providing a
health check implementation that checks for all other Cassandra peers being
up, and future work can add more robust health checks.

Best,
Andrés

On Fri, Aug 29, 2025 at 3:56 PM Andrés Beck-Ruiz <an...@gmail.com>
wrote:

> Hello everyone,
>
> We would like to propose CEP 53: Cassandra Rolling Restarts via Sidecar (
> https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-53%3A+Cassandra+Rolling+Restarts+via+Sidecar
> )
>
> This CEP builds off of CEP-1
> <https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-1%3A+Apache+Cassandra+Management+Process%28es%29+-+Deprecated>
> and proposes a design for safe, efficient, and operator friendly rolling
> restarts on Cassandra clusters, as well as an extensible approach for
> persisting future cluster-wide operations in Cassandra Sidecar. We hope to
> leverage this infrastructure in the future to implement upgrade automation.
>
> We welcome all feedback and discussion. Thank you in advance for your time
> and consideration of this proposal!
>
> Best,
> Andrés and Paulo
>

Re: [DISCUSS] CEP 53: Cassandra Rolling Restarts via Sidecar

[Dinesh Joshi <dj...@apache.org>]

Andrés and Paulo, thank you very much for your CEP. I skimmed it and at the
first glance it looked great. I am going to take some time and dive into it
over the weekend to understand it more deeply.

I really appreciate this contribution.

Thanks,

Dinesh

On Fri, Aug 29, 2025 at 12:58 PM Andrés Beck-Ruiz <an...@gmail.com>
wrote:

> Hello everyone,
>
> We would like to propose CEP 53: Cassandra Rolling Restarts via Sidecar (
> https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-53%3A+Cassandra+Rolling+Restarts+via+Sidecar
> )
>
> This CEP builds off of CEP-1
> <https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-1%3A+Apache+Cassandra+Management+Process%28es%29+-+Deprecated>
> and proposes a design for safe, efficient, and operator friendly rolling
> restarts on Cassandra clusters, as well as an extensible approach for
> persisting future cluster-wide operations in Cassandra Sidecar. We hope to
> leverage this infrastructure in the future to implement upgrade automation.
>
> We welcome all feedback and discussion. Thank you in advance for your time
> and consideration of this proposal!
>
> Best,
> Andrés and Paulo
>