You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@cassandra.apache.org by Patrick McFadin <pm...@gmail.com> on 2024/09/19 18:07:10 UTC
Re: [Discuss] Repair inside C*
Is this CEP ready for a VOTE thread?
https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-37+%28DRAFT%29+Apache+Cassandra+Unified+Repair+Solution
On Sun, Feb 25, 2024 at 12:25 PM Jaydeep Chovatia <
chovatia.jaydeep@gmail.com> wrote:
> Thanks, Josh. I've just updated the CEP
> <https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-37+%28DRAFT%29+Apache+Cassandra+Official+Repair+Solution>
> and included all the solutions you mentioned below.
>
> Jaydeep
>
> On Thu, Feb 22, 2024 at 9:33 AM Josh McKenzie <jm...@apache.org>
> wrote:
>
>> Very late response from me here (basically necro'ing this thread).
>>
>> I think it'd be useful to get this condensed into a CEP that we can then
>> discuss in that format. It's clearly something we all agree we need and
>> having an implementation that works, even if it's not in your preferred
>> execution domain, is vastly better than nothing IMO.
>>
>> I don't have cycles (nor background ;) ) to do that, but it sounds like
>> you do Jaydeep given the implementation you have on a private fork + design.
>>
>> A non-exhaustive list of things that might be useful incorporating into
>> or referencing from a CEP:
>> Slack thread:
>> https://the-asf.slack.com/archives/CK23JSY2K/p1690225062383619
>> Joey's old C* ticket:
>> https://issues.apache.org/jira/browse/CASSANDRA-14346
>> Even older automatic repair scheduling:
>> https://issues.apache.org/jira/browse/CASSANDRA-10070
>> Your design gdoc:
>> https://docs.google.com/document/d/1CJWxjEi-mBABPMZ3VWJ9w5KavWfJETAGxfUpsViPcPo/edit#heading=h.r112r46toau0
>> PR with automated repair:
>> https://github.com/jaydeepkumar1984/cassandra/commit/ef6456d652c0d07cf29d88dfea03b73704814c2c
>>
>> My intuition is that we're all basically in agreement that this is
>> something the DB needs, we're all willing to bikeshed for our personal
>> preference on where it lives and how it's implemented, and at the end of
>> the day, code talks. I don't think anyone's said they'll die on the hill of
>> implementation details, so that feels like CEP time to me.
>>
>> If you were willing and able to get a CEP together for automated repair
>> based on the above material, given you've done the work and have the proof
>> points it's working at scale, I think this would be a *huge contribution*
>> to the community.
>>
>> On Thu, Aug 24, 2023, at 7:26 PM, Jaydeep Chovatia wrote:
>>
>> Is anyone going to file an official CEP for this?
>> As mentioned in this email thread, here is one of the solution's design
>> doc
>> <https://docs.google.com/document/d/1CJWxjEi-mBABPMZ3VWJ9w5KavWfJETAGxfUpsViPcPo/edit#heading=h.r112r46toau0>
>> and source code on a private Apache Cassandra patch. Could you go through
>> it and let me know what you think?
>>
>> Jaydeep
>>
>> On Wed, Aug 2, 2023 at 3:54 PM Jon Haddad <ru...@apache.org>
>> wrote:
>>
>> > That said I would happily support an effort to bring repair scheduling
>> to the sidecar immediately. This has nothing blocking it, and would
>> potentially enable the sidecar to provide an official repair scheduling
>> solution that is compatible with current or even previous versions of the
>> database.
>>
>> This is something I hadn't thought much about, and is a pretty good
>> argument for using the sidecar initially. There's a lot of deployments out
>> there and having an official repair option would be a big win.
>>
>>
>> On 2023/07/26 23:20:07 "C. Scott Andreas" wrote:
>> > I agree that it would be ideal for Cassandra to have a repair scheduler
>> in-DB.
>> >
>> > That said I would happily support an effort to bring repair scheduling
>> to the sidecar immediately. This has nothing blocking it, and would
>> potentially enable the sidecar to provide an official repair scheduling
>> solution that is compatible with current or even previous versions of the
>> database.
>> >
>> > Once TCM has landed, we’ll have much stronger primitives for repair
>> orchestration in the database itself. But I don’t think that should block
>> progress on a repair scheduling solution in the sidecar, and there is
>> nothing that would prevent someone from continuing to use a sidecar-based
>> solution in perpetuity if they preferred.
>> >
>> > - Scott
>> >
>> > > On Jul 26, 2023, at 3:25 PM, Jon Haddad <ru...@apache.org>
>> wrote:
>> > >
>> > > I'm 100% in favor of repair being part of the core DB, not the
>> sidecar. The current (and past) state of things where running the DB
>> correctly *requires* running a separate process (either community
>> maintained or official C* sidecar) is incredibly painful for folks. The
>> idea that your data integrity needs to be opt-in has never made sense to me
>> from the perspective of either the product or the end user.
>> > >
>> > > I've worked with way too many teams that have either configured this
>> incorrectly or not at all.
>> > >
>> > > Ideally Cassandra would ship with repair built in and on by default.
>> Power users can disable if they want to continue to maintain their own
>> repair tooling for some reason.
>> > >
>> > > Jon
>> > >
>> > >> On 2023/07/24 20:44:14 German Eichberger via dev wrote:
>> > >> All,
>> > >> We had a brief discussion in [2] about the Uber article [1] where
>> they talk about having integrated repair into Cassandra and how great that
>> is. I expressed my disappointment that they didn't work with the community
>> on that (Uber, if you are listening time to make amends 🙂) and it turns
>> out Joey already had the idea and wrote the code [3] - so I wanted to start
>> a discussion to gauge interest and maybe how to revive that effort.
>> > >> Thanks,
>> > >> German
>> > >> [1]
>> https://www.uber.com/blog/how-uber-optimized-cassandra-operations-at-scale/
>> > >> [2] https://the-asf.slack.com/archives/CK23JSY2K/p1690225062383619
>> > >> [3] https://issues.apache.org/jira/browse/CASSANDRA-14346
>> >
>>
>>
>>
Re: [Discuss] Repair inside C*
Posted by Jaydeep Chovatia <ch...@gmail.com>.
FYI..I've updated the CEP-37 content to include all the improvements we
have discussed over this ML, ongoing work, and future work. I've also
tagged JIRAs for each ongoing/future work and assigned to some individuals
with a tentative timeline.
Thanks, Mick, for suggesting capturing the ML discussion in the CEP-37!
Jaydeep
Re: [Discuss] Repair inside C*
Posted by Jaydeep Chovatia <ch...@gmail.com>.
Thanks for the kind words, Mick!
Jaydeep
On Wed, Oct 30, 2024 at 1:35 AM Mick Semb Wever <mc...@apache.org> wrote:
> Thanks Jaydeep. I've exhausted my lines on enquiry and am happy that
> thought has gone into them.
>
>
>
>> On top of the above list, here is my recommendation (this is just a pure
>> thought only and subject to change depending on how all the community
>> members see it):
>>
>> - Nov-Dec: We can definitely prioritize the table-level priority
>> feature, which would address many concerns - Jaydeep (I can take the lead
>> for a small discussion followed by implementation)
>> - Nov-Feb: For table-level tracking, we can divide it into two parts:
>> - (Part-1) Nov-Dec: A video meeting discussion among a few of us
>> and see how we want to design, etc. - Jaydeep
>> - (Part-2) Dec-Feb: Based on the above design, implement
>> accordingly - *TODO*
>>
>>
>
> It's the discussion that's important to me, that everyone involved has
> taken onboard the input and has a PoV on it.
>
>
Re: [Discuss] Repair inside C*
Posted by Mick Semb Wever <mc...@apache.org>.
Thanks Jaydeep. I've exhausted my lines on enquiry and am happy that
thought has gone into them.
> On top of the above list, here is my recommendation (this is just a pure
> thought only and subject to change depending on how all the community
> members see it):
>
> - Nov-Dec: We can definitely prioritize the table-level priority
> feature, which would address many concerns - Jaydeep (I can take the lead
> for a small discussion followed by implementation)
> - Nov-Feb: For table-level tracking, we can divide it into two parts:
> - (Part-1) Nov-Dec: A video meeting discussion among a few of us
> and see how we want to design, etc. - Jaydeep
> - (Part-2) Dec-Feb: Based on the above design, implement
> accordingly - *TODO*
>
>
It's the discussion that's important to me, that everyone involved has
taken onboard the input and has a PoV on it.
Re: [Discuss] Repair inside C*
Posted by Jaydeep Chovatia <ch...@gmail.com>.
>Repairs causing a node to OOM is not unusual. I've been working with a
customer in this situation the past few weeks. Getting fixes out, or
mitigating the problem, is not always as quick as one hopes (see my
previous comment about how the repair_session_size setting gets easily
clobbered today). This situation would be much improved with table
priority and tracking is added to the system_distributed table(s).
I agree we would need to tackle this OOM / JVM crashing scenario
eventually, but on the other hand, adding table-level tracking looks easy
but to perfect it, it would take some effort, say we would have to handle
all corner-case scenarios, such as cleaning the state metadata, what would
happen if there is a race condition that table was dropped but metadata
could not. Architecture extension is simple, but making it bug-free and
robust is a bit complex.
>Does this emergency list imply then not doing --partitioner-range ?
The emergency list is to prioritize a few nodes over others, but those
nodes will continue to honor the same repair configuration that has been
provided. The default configuration is to repair primary token ranges only.
>For per-table custom-priorities and tracking it sounds like adding a
clustering column. So the number of records would go from ~number of nodes
in the cluster, to ~number of nodes multiplied by up to the number of
tables in the cluster. We do see clusters too often with up to a thousand
tables, despite strong recommendations not to go over two hundred. Do you
see here any concern ?
My initial thoughts are to add this as a CQL table property, something like
"repair_pririty=0.0", with all tables having the same priority. But the
user can change the priority through ALTER, say, "ALTER TABLE T1 WITH
repair_pririty=0.1", then T1 will be prioritized over other tables. Again,
I need to give more thought to it and need to do a small discussion either
in a bi-weekly meeting or on a ticket to ensure all folks are on the same
page. If we go with this approach, we do not need to add any additional
columns to the repair metadata tables, so that way the design continues to
remain lightweight, etc.
For a moment, let's just assume we add a new clustering column to track
tables. After that, the number of rows will be = <total_nodes> *
<total_tables>, which is still not an issue. As I mentioned above, the
bigger problem for table-tracking is not the architecture extension, but
perfecting with all race conditions is a bit complex.
>Also, in what versions will we be able to introduce such improvements ? We
will be waiting until the next major release ? Playing around with the
schema of system tables in release branches is not much fun.
There are a few items on the priority beyond the CEP-37 MVP scope that some
of us are working on:
1. Extend disk-capacity check for full repair - Jaydeep
2. Making the incremental repair more reliable by having an unrepaired
size-based token splitter - Andy T, Chris L
3. Add support for the Preview Repair - Kristijonas
4. Start a new ML discussion gauge consensus on whether repairs should be
backward/forwards compatible between major versions in the future - Andy T
On top of the above list, here is my recommendation (this is just a pure
thought only and subject to change depending on how all the community
members see it):
- Nov-Dec: We can definitely prioritize the table-level priority
feature, which would address many concerns - Jaydeep (I can take the lead
for a small discussion followed by implementation)
- Nov-Feb: For table-level tracking, we can divide it into two parts:
- (Part-1) Nov-Dec: A video meeting discussion among a few of us and
see how we want to design, etc. - Jaydeep
- (Part-2) Dec-Feb: Based on the above design, implement accordingly
- *TODO*
Jaydeep
On Tue, Oct 29, 2024 at 12:06 PM Mick Semb Wever <mc...@apache.org> wrote:
>
> Jaydeep,
> your replies address my main concerns, there's a few questions of
> curiosity as replies inline below…
>
>
>
>
>
>> >Without any per-table scheduling and history (IIUC) a node would have
>> to restart the repairs for all keyspaces and tables.
>>
>> The above-mentioned quote should work fine and will make sure the bad
>> tables/keyspaces are skipped, allowing the good keyspaces/tables to proceed
>> on a node as long as the Cassandra JVM itself keeps crashing. If a JVM
>> keeps crashing, then it will restart all over again, but then fixing the
>> JVM crashing might be a more significant issue and does not happen
>> regularly, IMO.
>>
>
>
> Repairs causing a node to OOM is not unusual. I've been working with a
> customer in this situation the past few weeks. Getting fixes out, or
> mitigating the problem, is not always as quick as one hopes (see my
> previous comment about how the repair_session_size setting gets easily
> clobbered today). This situation would be much improved with table
> priority and tracking is added to the system_distributed table(s).
>
>
>
>> If an admin sets some nodes on a priority queue, those nodes will be
>> repaired over the scheduler's own list. If an admin tags some nodes on the
>> emergency list, then those nodes will repair immediately. Basically, an
>> admin tells the scheduler, "*Just do what I say instead of using your
>> list of nodes*".
>>
>
>
> Does this emergency list imply then not doing --partitioner-range ?
>
>
> >I am also curious as to how the impact of these tables changes as we
>> address (1) and (2).
>>
>> Quite a lot of (1) & (2) can be addressed by just adding a new CQL
>> property, which won't even touch these metadata tables. In case we need to,
>> depending on the design for (1) & (2), it can be either addressed by adding
>> new columns and/or adding a new metadata table.
>>
>
> For per-table custom-priorities and tracking it sounds like adding a
> clustering column. So the number of records would go from ~number of nodes
> in the cluster, to ~number of nodes multiplied by up to the number of
> tables in the cluster. We do see clusters too often with up to a thousand
> tables, despite strong recommendations not to go over two hundred. Do you
> see here any concern ?
>
> Also, in what versions will we be able to introduce such improvements ? We
> will be waiting until the next major release ? Playing around with the
> schema of system tables in release branches is not much fun.
>
>
>
Re: [Discuss] Repair inside C*
Posted by Mick Semb Wever <mc...@apache.org>.
Jaydeep,
your replies address my main concerns, there's a few questions of
curiosity as replies inline below…
> >Without any per-table scheduling and history (IIUC) a node would have
> to restart the repairs for all keyspaces and tables.
>
> The above-mentioned quote should work fine and will make sure the bad
> tables/keyspaces are skipped, allowing the good keyspaces/tables to proceed
> on a node as long as the Cassandra JVM itself keeps crashing. If a JVM
> keeps crashing, then it will restart all over again, but then fixing the
> JVM crashing might be a more significant issue and does not happen
> regularly, IMO.
>
Repairs causing a node to OOM is not unusual. I've been working with a
customer in this situation the past few weeks. Getting fixes out, or
mitigating the problem, is not always as quick as one hopes (see my
previous comment about how the repair_session_size setting gets easily
clobbered today). This situation would be much improved with table
priority and tracking is added to the system_distributed table(s).
> If an admin sets some nodes on a priority queue, those nodes will be
> repaired over the scheduler's own list. If an admin tags some nodes on the
> emergency list, then those nodes will repair immediately. Basically, an
> admin tells the scheduler, "*Just do what I say instead of using your
> list of nodes*".
>
Does this emergency list imply then not doing --partitioner-range ?
>I am also curious as to how the impact of these tables changes as we
> address (1) and (2).
>
> Quite a lot of (1) & (2) can be addressed by just adding a new CQL
> property, which won't even touch these metadata tables. In case we need to,
> depending on the design for (1) & (2), it can be either addressed by adding
> new columns and/or adding a new metadata table.
>
For per-table custom-priorities and tracking it sounds like adding a
clustering column. So the number of records would go from ~number of nodes
in the cluster, to ~number of nodes multiplied by up to the number of
tables in the cluster. We do see clusters too often with up to a thousand
tables, despite strong recommendations not to go over two hundred. Do you
see here any concern ?
Also, in what versions will we be able to introduce such improvements ? We
will be waiting until the next major release ? Playing around with the
schema of system tables in release branches is not much fun.
Re: [Discuss] Repair inside C*
Posted by Jaydeep Chovatia <ch...@gmail.com>.
Thanks, Mick, for the comment, please find my response below.
>(1)
I think I covered most of the points in my response to Alexander (except
one, which I am responding to below separately). Tl;dr is the MVP that can
be easily extended to do a table-level schedule; it is just going to be
another CQL table property as opposed to a yaml config (currently in MVP).
I had already added this as a near-term feature here and added that when we
add repair priority table-wise, we need to ensure the table-level
scheduling is also taken care of. Please visit my latest few comments to
the ticket https://issues.apache.org/jira/browse/CASSANDRA-20013
>You may also want to do repairs in different DCs differently.
Currently, the MVP allows one to skip one or more DCs if they wish to do so
by defaulting all DCs. This again points to the similar theme of allowing
schedule (or priority) at a table level followed by a DC level. The MVP can
be easily extended at whatever granularity we want scheduling to be without
many architectural changes. We all just have to finalize the granularity we
want. I've also added to the ticket above that scheduling support at a
table-level followed by DC-level granularity.
>I'm curious as to how crashed repairs are handled and resumed
The MVP has a max allowed quota at a keyspace level and at a table level.
So, if a repair and/or keyspace takes much longer than the timeout due to
failures/more data it needs to repair, etc., then it will skip to the next
table/keyspace.
>Without any per-table scheduling and history (IIUC) a node would have to
restart the repairs for all keyspaces and tables.
The above-mentioned quote should work fine and will make sure the bad
tables/keyspaces are skipped, allowing the good keyspaces/tables to proceed
on a node as long as the Cassandra JVM itself keeps crashing. If a JVM
keeps crashing, then it will restart all over again, but then fixing the
JVM crashing might be a more significant issue and does not happen
regularly, IMO.
>And without such per-table tracking, I'm also kinda curious as to how we
interact with manual repair invocations the user makes. There are
operational requirements to do manual repairs, e.g. node replacement or if
a node has been down for too long, and consistency breakages until such
repair is complete. Leaving such operational requirements to this CEP's
in-built scheduler is a limited approach, it may be many days before it
gets to doing it, and even with node priority will it appropriately switch
from primary-range to all-replica-ranges?
To alleviate some of this, the MVP has two options one can configure
dynamically through *nodetool*: 1) Setting priority for nodes, 2) Telling
the scheduler to repair one or more nodes immediately
If an admin sets some nodes on a priority queue, those nodes will be
repaired over the scheduler's own list. If an admin tags some nodes on the
emergency list, then those nodes will repair immediately. Basically, an
admin tells the scheduler, "*Just do what I say instead of using your list
of nodes*".
Even with this, if an admin decides to trigger repair manually directly
through *nodetool repair*, then the scheduler should not interfere with
that manually triggered operation - they can progress independently. The
MVP has options to disable the scheduler's repair dynamically without any
cluster restart, etc., so the admin can use some of the combinations and
decide what to do when they invoke any manual repair operation.
>What if the user accidentally invokes an incremental repair when the
in-built scheduler is expecting only to ever perform full repairs? Does it
know how to detect/remedy that?
The user invocation and the scheduler invocations go through two different
Repair sessions. If the MVP scheduler has been configured only to perform
FR, then the scheduler will never fire IR, but it does not prohibit the
user from firing IR through *nodetool repair*. As an enhancement to the
MVP, in the future, we must warn the user that it might not be safe to run
IR as the in-built scheduler has been configured not to do IR, etc., so be
careful, etc.
>Having read the design doc and PR, I am impressed how lightweight the
design of the tables are.
Thanks. To reiterate, the number of records in the system_distributed will
be equivalent to the number of nodes in the Cluster.
>But I do still think we deserve some numbers, and a further line of
questioning: what consistency guarantees do we need, how does this work
cross-dc, during topology changes, does an event that introduces
data-at-rest inconsistencies in the cluster then become
confused/inefficient when the mechanism to repair it also now has its
metadata inconsistent. For the most part this is a problem not unique to
any table in system_distributed and otherwise handled, but how does the
system_distributed keyspace handling of such failures impact repairs.
Keeping practicality in mind, the record count to the table should be as
small as three rows and as big as a couple of thousands, considering
production clusters worldwide, IMO.
Each node is responsible for updating its record only, i.e., each node
touches, at most, one record with consistency, and it uses LWT
(LOCAL_QUORUM+LOCAL_SERIAL) to update the data to preserve consistency.
This means, by and large, the metadata data would remain consistent for the
majority of cases. Let's assume, in the worst case, say, the data becomes
inconsistent between two nodes, which means these two nodes have some sort
of split view. In this case, the scheduler might trigger repair on more
nodes simultaneously than the parallelism is configured to do. But this
worst case is possible even with the data stored outside of
*system_distributed* unless we choose to store the repair metadata outside
Cassandra, say, in ETCD. I believe CEP-15 should strengthen our metadata
story further, and we can tweak it once CEP-15 is certified.
In summary, I personally do not believe this is a massive problem as it
does not break any, and with CEP-15 on the horizon, it will further
strengthen the repair metadata story. We should not store this metadata
outside of Cassandra because it gives a different impression to others
about the Apache Cassandra project :) - we should rather solidify
eventually on top of CEP-15 or something similar.
>I am also curious as to how the impact of these tables changes as we
address (1) and (2).
Quite a lot of (1) & (2) can be addressed by just adding a new CQL
property, which won't even touch these metadata tables. In case we need to,
depending on the design for (1) & (2), it can be either addressed by adding
new columns and/or adding a new metadata table.
> I can see how the CEP's design works well for the biggest clusters, and
those with heterogeneous data-models (which often comes with larger
deployment sets). But I don't think we can use this as the bar to quality
or acceptance. Many smaller clusters that come with lots of keyspaces and
tables have real troubles trying to get repairs to run weekly. We can't
simply blame users for not having optimal data models and deployments.
The MVP implementation/design should work for most cases—small, medium, and
enormous clusters. As I mentioned to Alexander, the scheduling is
available; the default is "1d," but the config can be changed to "7d" and
that cluster will be scheduled for repair only once every 7d. One can also
have a different scheduling for IR and FR, say IR every 1d and the FR every
7d.
My personal recommendation is to keep running the repair
continuously because if we delay the repair, the amount of inconsistencies
amplifies, and we have to pay a hefty price later. But the MVP does have an
option if one wants to run it every 7d or say. I am even fine if we want to
increase the default from 1d --> 7d. Just sharing another personal
experience that we have all different cluster size combinations we are
running in production, and the same design has been flawlessly working so
far. 🤞
>No one expects the CEP to initially handle and solve every situation,
especially poor data-models and over-capacity clusters. Hope here is just a
bit of discussion that can help us be informative about our limitations,
and possibly save some users from thinking this is their silver bullet.
Totally agree with you that it is almost impossible to come up with a
solution and default configuration that will satisfy 100% of the use cases
across the world, but the idea here is to start with the most common ones,
and as we learn, keep building on top of the MVP, e.g. table-level
priority, etc. to make it a richer solution that might increase the
coverage. That's why, if we look at the approach, then it has been
incremental so far.
a) It has been validated with the production use cases I have. b) Then
Chris L started incorporating, and along the way, he suggested some tweaks,
changes, etc., and now it works for most cases on his side. c) In the same
way, we will continue engaging more users - keep tweaking, and then
eventually, all of us will make this even richer solution.
>The biggest aspect to this I believe is (1), but operational stability and
tuning is also critical. Alex mentions the range-centric approach, which
helps balance load, which in turn gives you more head room. But there's
also stuff like parallel repairs, handling (dividing) repair_session_size,
256 vnode times 16 subranges on many empty tables saturating throughput,
etc. I think most of these are minor and will fit into the design ok.
WRT 256 vnode multiplied by 16 subranges running on tiny tables, I see the
implementation of splitting repairs by partition count rather than token
range as pretty crucial tbh.
This has already been answered as part of Alexander's response and above.
>Also curious (maybe I missed it in the PR) how incremental repairs are
getting token ranged, as this has a noticeable duplicating impact on the
cost of anti-compaction.
The IR will split the token ranges evenly - this approach is ok for FR, but
it is not optimal for IR. As I mentioned above, Andy T and Chris L have a
working POC to calculate the token ranges based on unrepaired data size on
top of the MVP, and they are working on adding test cases, etc., and then
the MVP will be extended with that solution for IR.
>Also, is node load ever taken into account, e.g. avoiding starting
repairs on nodes where too many pending compactions, hints being received,
etc.
Currently, it does not, but this is altogether a different scope of work in
Cassandra, in that we should assign priority to every work item, say
Compaction, Streaming, Repair, Memtable flush, Snapshot, etc., and then
monitor resource utilization and then only honor the top priority tasks
when under resource crunch. I have proposed a glimpse of it in the CEP-41
<https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-41+%28DRAFT%29+Apache+Cassandra+Unified+Rate+Limiter>
(Draft), a lot more to be done here. Alex Petrov mentioned to me that he is
also thinking along these lines, and will initiate a detailed discussion on
this topic soon.
Jaydeep
On Mon, Oct 28, 2024 at 9:21 AM Mick Semb Wever <mc...@apache.org> wrote:
> any name works for me, Jaydeep :-)
>
> I've taken a run through of the CEP, design doc, and current PR. Below
> are my four (rough categories of) questions.
> I am keen to see a MVP land, so I'm more looking at what the CEP's design
> might not be able to do, rather than what may or may not land in an initial
> implementation. There's a bit below, and some of it really would be better
> in the PR, feel free to take it there if deemed more constructive.
>
>
> 1) The need for different schedules for different tables
> 2) Failure mode: repairs failing and thrashing repairs for all
> keyspaces+tables
> 3) Small concerns on relying on system tables
> 4) Small concerns on tuning requirements
>
>
> (1)
> Alex also touched on this. I'm aware of too many reasons where this is a
> must-have. Many users cannot repair their clusters without tuning
> per-table schedules. Different gc_grace_seconds is the biggest reason.
> But there's also running full repairs infrequently for disk rot (or similar
> reason) on a table that's otherwise frequently incremental repaired (also
> means an incremental repair could be skipped if the full repair was
> currently running). Or TWCS tables where you benefit from higher frequency
> of incremental repair (and/or want to minimise repairs older than the
> current time_window). You may also want to do repairs in different DCs
> differently.
>
> (2)
> I'm curious as to how crashed repairs are handled and resumed…
> A problem a lot of users struggle with is where the repair on one table is
> enigmatically problematic, crashing or timing out, and it takes a long time
> to figure it out.
> Without any per-table scheduling and history (IIUC) a node would have to
> restart the repairs for all keyspaces and tables. This will lead to
> over-repairing some tables and never repairing others.
>
> And without such per-table tracking, I'm also kinda curious as to how we
> interact with manual repair invocations the user makes.
>
> There are operational requirements to do manual repairs, e.g. node
> replacement or if a node has been down for too long, and consistency
> breakages until such repair is complete. Leaving such operational
> requirements to this CEP's in-built scheduler is a limited approach, it may
> be many days before it gets to doing it, and even with node priority will
> it appropriately switch from primary-range to all-replica-ranges?
>
> What if the user accidently invokes an incremental repair when the
> in-built scheduler is expecting only to ever perform full repairs, does it
> know how to detect/remedy that?
>
>
> (3)
> Having stuff in system tables is brittle and a write-amplification, we
> have plenty of experience of this from DSE NodeSync and Reaper. Reaper's
> ability to store its metadata out-of-cluster is a huge benefit. Having
> read the design doc and PR, I am impressed how lightweight the design of
> the tables are. But I do still think we deserve some numbers, and a
> further line of questioning: what consistency guarantees do we need, how
> does this work cross-dc, during topology changes, does an event that
> introduces data-at-rest inconsistencies in the cluster then become
> confused/inefficient when the mechanism to repair it also now has its
> metadata inconsistent. For the most part this is a problem not unique to
> any table in system_distributed and otherwise handled, but how does the
> system_distributed keyspace handling of such failures impact repairs.
>
> Even with strong consistency, I would assume the design needs to be
> pessimistic, e.g. multiple node repairs can be started at the time. Is
> this true, if so how is it handled ?
>
> I am also curious as to how the impact of these tables changes as we
> address (1) and (2).
>
> (4)
> I can see how the CEP's design works well for the biggest clusters, and
> those with heterogeneous data-models (which often comes with larger
> deployment sets). But I don't think we can use this as the bar to quality
> or acceptance. Many smaller clusters that come with lots of keyspaces and
> tables have real troubles trying to get repairs to run weekly. We can't
> simply blame users for not having optimal data models and deployments.
>
> Carefully tuning the schedules of tables, and the cluster itself, is often
> a requirement – time-consuming and a real pain point. The CEP as it stands
> today I can, with confidence, say will simply not work for many users.
> Worse than that it will provide false hope, and take time and effort for
> users until they realise it won't work, leaving them having to revert to
> their previous solution. No one expects the CEP to initially handle and
> solve every situation, especially poor data-models and over-capacity
> clusters. Hope here is just a bit of discussion that can help us be
> informative about our limitations, and possibly save some users from
> thinking this is their silver bullet.
>
> The biggest aspect to this I believe is (1), but operational stability and
> tuning is also critical. Alex mentions the range-centric approach, which
> helps balance load, which in turn gives you more head room. But there's
> also stuff like parallel repairs, handling (dividing) repair_session_size,
> 256 vnode times 16 subranges on many empty tables saturating throughput,
> etc. I think most of these are minor and will fit into the design ok.
> WRT 256 vnode multiplied by 16 subranges running on tiny tables, I see the
> implementation of splitting repairs by partition count rather than token
> range as pretty crucial tbh. Also curious (maybe I missed it in the PR)
> how incremental repairs are getting token ranged, as this has a noticeable
> duplicating impact on the cost of anti-compaction.
>
> Also, is node load ever taken into account, e.g. avoiding starting
> repairs on nodes where too many pending compactions, hints being received,
> etc.
>
>
Re: [Discuss] Repair inside C*
Posted by Mick Semb Wever <mc...@apache.org>.
any name works for me, Jaydeep :-)
I've taken a run through of the CEP, design doc, and current PR. Below are
my four (rough categories of) questions.
I am keen to see a MVP land, so I'm more looking at what the CEP's design
might not be able to do, rather than what may or may not land in an initial
implementation. There's a bit below, and some of it really would be better
in the PR, feel free to take it there if deemed more constructive.
1) The need for different schedules for different tables
2) Failure mode: repairs failing and thrashing repairs for all
keyspaces+tables
3) Small concerns on relying on system tables
4) Small concerns on tuning requirements
(1)
Alex also touched on this. I'm aware of too many reasons where this is a
must-have. Many users cannot repair their clusters without tuning
per-table schedules. Different gc_grace_seconds is the biggest reason.
But there's also running full repairs infrequently for disk rot (or similar
reason) on a table that's otherwise frequently incremental repaired (also
means an incremental repair could be skipped if the full repair was
currently running). Or TWCS tables where you benefit from higher frequency
of incremental repair (and/or want to minimise repairs older than the
current time_window). You may also want to do repairs in different DCs
differently.
(2)
I'm curious as to how crashed repairs are handled and resumed…
A problem a lot of users struggle with is where the repair on one table is
enigmatically problematic, crashing or timing out, and it takes a long time
to figure it out.
Without any per-table scheduling and history (IIUC) a node would have to
restart the repairs for all keyspaces and tables. This will lead to
over-repairing some tables and never repairing others.
And without such per-table tracking, I'm also kinda curious as to how we
interact with manual repair invocations the user makes.
There are operational requirements to do manual repairs, e.g. node
replacement or if a node has been down for too long, and consistency
breakages until such repair is complete. Leaving such operational
requirements to this CEP's in-built scheduler is a limited approach, it may
be many days before it gets to doing it, and even with node priority will
it appropriately switch from primary-range to all-replica-ranges?
What if the user accidently invokes an incremental repair when the in-built
scheduler is expecting only to ever perform full repairs, does it know how
to detect/remedy that?
(3)
Having stuff in system tables is brittle and a write-amplification, we have
plenty of experience of this from DSE NodeSync and Reaper. Reaper's
ability to store its metadata out-of-cluster is a huge benefit. Having
read the design doc and PR, I am impressed how lightweight the design of
the tables are. But I do still think we deserve some numbers, and a
further line of questioning: what consistency guarantees do we need, how
does this work cross-dc, during topology changes, does an event that
introduces data-at-rest inconsistencies in the cluster then become
confused/inefficient when the mechanism to repair it also now has its
metadata inconsistent. For the most part this is a problem not unique to
any table in system_distributed and otherwise handled, but how does the
system_distributed keyspace handling of such failures impact repairs.
Even with strong consistency, I would assume the design needs to be
pessimistic, e.g. multiple node repairs can be started at the time. Is
this true, if so how is it handled ?
I am also curious as to how the impact of these tables changes as we
address (1) and (2).
(4)
I can see how the CEP's design works well for the biggest clusters, and
those with heterogeneous data-models (which often comes with larger
deployment sets). But I don't think we can use this as the bar to quality
or acceptance. Many smaller clusters that come with lots of keyspaces and
tables have real troubles trying to get repairs to run weekly. We can't
simply blame users for not having optimal data models and deployments.
Carefully tuning the schedules of tables, and the cluster itself, is often
a requirement – time-consuming and a real pain point. The CEP as it stands
today I can, with confidence, say will simply not work for many users.
Worse than that it will provide false hope, and take time and effort for
users until they realise it won't work, leaving them having to revert to
their previous solution. No one expects the CEP to initially handle and
solve every situation, especially poor data-models and over-capacity
clusters. Hope here is just a bit of discussion that can help us be
informative about our limitations, and possibly save some users from
thinking this is their silver bullet.
The biggest aspect to this I believe is (1), but operational stability and
tuning is also critical. Alex mentions the range-centric approach, which
helps balance load, which in turn gives you more head room. But there's
also stuff like parallel repairs, handling (dividing) repair_session_size,
256 vnode times 16 subranges on many empty tables saturating throughput,
etc. I think most of these are minor and will fit into the design ok.
WRT 256 vnode multiplied by 16 subranges running on tiny tables, I see the
implementation of splitting repairs by partition count rather than token
range as pretty crucial tbh. Also curious (maybe I missed it in the PR)
how incremental repairs are getting token ranged, as this has a noticeable
duplicating impact on the cost of anti-compaction.
Also, is node load ever taken into account, e.g. avoiding starting repairs
on nodes where too many pending compactions, hints being received, etc.
Re: [Discuss] Repair inside C*
Posted by Jaydeep Chovatia <ch...@gmail.com>.
Mick, I am highly sorry to mispronounce your name.
Indeed, Mick - two additional weeks is not an issue at all.
Jaydeep
On Fri, Oct 18, 2024 at 1:41 PM Jaydeep Chovatia <ch...@gmail.com>
wrote:
> Indeed, Mike - two additional weeks is not an issue at all.
> Thanks!
>
> Jaydeep
>
Re: [Discuss] Repair inside C*
Posted by Jaydeep Chovatia <ch...@gmail.com>.
Indeed, Mike - two additional weeks is not an issue at all.
Thanks!
Jaydeep
Re: [Discuss] Repair inside C*
Posted by Jaydeep Chovatia <ch...@gmail.com>.
>Since the auto repair is running from within Cassandra, we might have more
control over this and implement a proper cleanup of such snapshots.
Rightly said, Alexander. Having internal knowledge of Cassandra, we can do
a lot more. For example, for better Incremental Reliability reliability,
Andy T and Chris L have developed a new token-split algorithm on top of the
MVP based on unrepaired data in SSTables (soon it will be added to the MVP
as they are working on writing test cases, etc.), and that
requires internal SSTable data-structure access, etc.
Jaydeep
On Mon, Oct 28, 2024 at 10:51 PM Jaydeep Chovatia <
chovatia.jaydeep@gmail.com> wrote:
>
>> > That's inaccurate, we can check the replica set for the subrange we're
> about to run and see if it overlaps with the replica set of other ranges
> which are being processed already.
> We can definitely check the replicas for the subrange we plan to run and
> see if they overlap with the ongoing one. I am saying that for a smaller
> cluster if we want to repair multiple token ranges in parallel, it is tough
> to guarantee that replica sets won't overlap.
>
> >Jira to auto-delete snapshots at X% disk full ?
> Sure, just created a new JIRA
> https://issues.apache.org/jira/browse/CASSANDRA-20035
>
> Jaydeep
>
Re: [Discuss] Repair inside C*
Posted by Jaydeep Chovatia <ch...@gmail.com>.
>
>
> > That's inaccurate, we can check the replica set for the subrange we're
about to run and see if it overlaps with the replica set of other ranges
which are being processed already.
We can definitely check the replicas for the subrange we plan to run and
see if they overlap with the ongoing one. I am saying that for a smaller
cluster if we want to repair multiple token ranges in parallel, it is tough
to guarantee that replica sets won't overlap.
>Jira to auto-delete snapshots at X% disk full ?
Sure, just created a new JIRA
https://issues.apache.org/jira/browse/CASSANDRA-20035
Jaydeep
Re: [Discuss] Repair inside C*
Posted by Jeff Jirsa <jj...@gmail.com>.
> On Oct 28, 2024, at 9:52 PM, Alexander Dejanovski <al...@datastax.com> wrote:
>
>
>
>> If a repair session finishes gracefully, then this timeout is not applicable. Anyway, I do not have any strong opinion on the value. I am open to lowering it to 1h or something.
> True, it will only delay killing hanging repairs.
> One thing that we cannot solve in Reaper at the moment is that sequential and dc aware repair sessions that get terminated due to the timeout leave ephemeral snapshots behind. Since they're only reclaimed on restart, having a lot of timeouts can end up filling the the disks if the snapshots get materialized.
> Since the auto repair is running from within Cassandra, we might have more control over this and implement a proper cleanup of such snapshots.
Jira to auto-delete snapshots at X% disk full ?
Re: [Discuss] Repair inside C*
Posted by Alexander Dejanovski <al...@datastax.com>.
>
> The scheduler repairs, by default, the primary ranges for all the nodes
> going through the repair. Since it uses the primary ranges, all the nodes
> repairing parallelly would not overlap in any form for the primary ranges.
> However, the replica set for the nodes going through repair may or may not
> overlap, but it totally depends on the cluster size and parallelism used.
> If a cluster is small, there is a possibility, but if it is large, the
> possibility reduces. Even if we go with a range-centric approach and if we
> repair N token ranges in parallel, there is no guarantee that their replica
> sets won't overlap for smaller clusters.
That's inaccurate, we can check the replica set for the subrange we're
about to run and see if it overlaps with the replica set of other ranges
which are being processed already.
The only solution is to reduce the repair parallelism to one. node at a
> time.
Yes, I agree.
This is supported with the MVP, we can set "min_repair_interval: 7d" (the
> default is 24h) and the nodes will repair only once every 7 days.
>
The MVP implementation allows running full and incremental repairs (and
> Preview repair code changes are done and it is coming soon) independently
> and in parallel. One can set the above config for each repair type with
> their preferred schedule.
Nice, sorry I missed these in the CEP doc.
> I have already created a ticket to add this as an enhancement
> https://issues.apache.org/jira/browse/CASSANDRA-20013
Thanks, table level repair priority could be a very interesting
improvement, that's something Reaper lacks as well at the moment.
If a repair session finishes gracefully, then this timeout is not
> applicable. Anyway, I do not have any strong opinion on the value. I am
> open to lowering it to *1h* or something.
True, it will only delay killing hanging repairs.
One thing that we cannot solve in Reaper at the moment is that sequential
and dc aware repair sessions that get terminated due to the timeout leave
ephemeral snapshots behind. Since they're only reclaimed on restart, having
a lot of timeouts can end up filling the the disks if the snapshots get
materialized.
Since the auto repair is running from within Cassandra, we might have more
control over this and implement a proper cleanup of such snapshots.
Alexander Dejanovski
Astra Managed Clusters / Mission Control
w. www.datastax.com
<https://www.datastax.com/lp/astra-registration>
On Mon, Oct 28, 2024 at 7:01 PM Jaydeep Chovatia <ch...@gmail.com>
wrote:
> Thanks a lot, Alexander, for the review! Please find my response below:
>
> > making these replicas process 3 concurrent repairs while others could
> be left uninvolved in any repair at all...Taking a range centric approach
> (we're not repairing nodes, we're repairing the token ranges) allows to
> spread the load evenly without overlap in the replica sets.
> The scheduler repairs, by default, the primary ranges for all the nodes
> going through the repair. Since it uses the primary ranges, all the nodes
> repairing parallelly would not overlap in any form for the primary ranges.
> However, the replica set for the nodes going through repair may or may not
> overlap, but it totally depends on the cluster size and parallelism used.
> If a cluster is small, there is a possibility, but if it is large, the
> possibility reduces. Even if we go with a range-centric approach and if we
> repair N token ranges in parallel, there is no guarantee that their replica
> sets won't overlap for smaller clusters.
>
> > I'm more worried even with incremental repair here, because you might
> end up with some conflicts around sstables which would be in the pending
> repair pool but would be needed by a competing repair job.
> This can happen regardless of whether we go by "node-centric" vs.
> "range-centric" if we run multiple parallel repair sessions. The reason is
> that SSTables for all the nodes going through repair may not be physically
> isolated 1:1 as per the token ranges being repaired. We just had a detailed
> discussion about the SSTable overlap for incremental repair (IR) last week
> in Slack (#cassandra-repair-scheduling-cep37), and the general consensus
> was that there is no better way to address it than just to retry a few
> times. The only solution is to reduce the repair parallelism to one. node
> at a time.
> The ideal and reliable way to repair IR is to calculate the token ranges
> based on the unrepaired data size and also apply the upper cap on the data
> size being repaired. The good news is that Andy Tolbert already extended
> the CEP-37 MVP for this, and he is working on making it perfect by adding
> necessary tests, etc., so it can be landed on top of this MVP. tl;dr Andy T
> and Chris L are already on top of this and soon it will be available on top
> of CEP-37 MVP.
>
> >I don't know if in the latest versions such sstables would be totally
> ignored or if the competing repair job would fail.
> The competing IR session would be aborted, and the scheduler would retry a
> few times.
>
> >Continuous repair might create a lot of overhead for full repairs which
> often don't require more than 1 run per week.
> This is supported with the MVP, we can set "min_repair_interval: 7d"
> (the default is 24h) and the nodes will repair only once every 7 days.
>
> >It also will not allow running a mix of scheduled full/incremental repairs
> The MVP implementation allows running full and incremental repairs (and
> Preview repair code changes are done and it is coming soon) independently
> and in parallel. One can set the above config for each repair type with
> their preferred schedule.
>
> >Here, nodes will be processed sequentially and each node will process the
> keyspaces sequentially, tying the repair cycle of all keyspaces together.
> The keyspaces and tables on each node will be randomly shuffled to avoid
> multiple nodes working on the same table/keyspaces.
>
> >There are many cases where one might have differentiated gc_grace_seconds
> settings to optimize reclaiming tombstones when applicable. That requires
> having some fine control over the repair cycle for a given keyspace/set of
> tables.
> As I mentioned, there is already a way to schedule a frequency of repair
> cycle, but the frequency is currently a global config on a node; hence
> applicable to all the tables on a node. However, the MVP design is flexible
> enough to be easily extended to add the schedule as a new CQL table-level
> property, which will then honor the table-level schedule as opposed to a
> global schedule. There was another suggestion from @masokol (from
> Ecchronos) to maybe assign a repair priority on a table level to prioritize
> one table over the other, and that can also solve this problem, which is
> also feasible on top of the MVP. I have already created a ticket to add
> this as an enhancement
> https://issues.apache.org/jira/browse/CASSANDRA-20013
>
> >I think the 3 hours timeout might be quite large and probably means a lot
> of data is being repaired for each split. That usually involves some level
> of overstreaming
> This timeout applies to unstuck stuck repair sessions due to some bug in
> the repair code path. e.g.
> https://issues.apache.org/jira/browse/CASSANDRA-14674
> If a repair session finishes gracefully, then this timeout is not
> applicable. Anyway, I do not have any strong opinion on the value. I am
> open to lowering it to *1h* or something.
>
> Jaydeep
>
> On Mon, Oct 28, 2024 at 4:45 AM Alexander DEJANOVSKI <
> adejanovski@gmail.com> wrote:
>
>> Hi Jaydeep,
>>
>> I've taken a look at the proposed design and have a few
>> comments/questions.
>> As one of the maintainers of Reaper, I'm looking this through the lens of
>> how Reaper does things.
>>
>>
>> *The approach taken in the CEP-37 design is "node-centric" vs a "range
>> centric" approach (which is the one Reaper takes).*I'm worried that this
>> will not allow spreading the repair load evenly across the cluster, since
>> nodes are the concurrency unit. You could allow running repair on 3 nodes
>> concurrently for example, but these 3 nodes could all involve the same
>> replicas, making these replicas process 3 concurrent repairs while others
>> could be left uninvolved in any repair at all.
>> Taking a range centric approach (we're not repairing nodes, we're
>> repairing the token ranges) allows to spread the load evenly without
>> overlap in the replica sets.
>> I'm more worried even with incremental repair here, because you might end
>> up with some conflicts around sstables which would be in the pending repair
>> pool but would be needed by a competing repair job.
>> I don't know if in the latest versions such sstables would be totally
>> ignored or if the competing repair job would fail.
>>
>> *Each repair command will repair all keyspaces (with the ability to fully
>> exclude some tables) and **I haven't seen a notion of schedule which
>> seems to suggest repairs are running continuously (unless I missed
>> something?).*
>> There are many cases where one might have differentiated gc_grace_seconds
>> settings to optimize reclaiming tombstones when applicable. That requires
>> having some fine control over the repair cycle for a given keyspace/set of
>> tables.
>> Here, nodes will be processed sequentially and each node will process the
>> keyspaces sequentially, tying the repair cycle of all keyspaces together.
>> If one of the ranges for a specific keyspace cannot be repaired within
>> the 3 hours timeout, it could block all the other keyspaces repairs.
>> Continuous repair might create a lot of overhead for full repairs which
>> often don't require more than 1 run per week.
>> It also will not allow running a mix of scheduled full/incremental
>> repairs (I'm unsure if that is still a recommendation, but it was still
>> recommended not so long ago)
>>
>> *The timeout base duration is large*
>> I think the 3 hours timeout might be quite large and probably means a lot
>> of data is being repaired for each split. That usually involves some level
>> of overstreaming. I don't have numbers to support this, it's more about my
>> own experience on sizing splits in production with Reaper to reduce the
>> impact as much as possible on cluster performance.
>> We use 30 minutes as default in Reaper with subsequent attempts growing
>> the timeout dynamically for challenging splits.
>>
>> Finally thanks for picking this up, I'm eager to see Reaper not being
>> needed anymore and having the database manage its own repairs!
>>
>>
>> Le mar. 22 oct. 2024 à 21:10, Benedict <be...@apache.org> a écrit :
>>
>>> I realise it’s out of scope, but to counterbalance all of the
>>> pro-decomposition messages I wanted to chime in with a strong -1. But we
>>> can debate that in a suitable context later.
>>>
>>> On 22 Oct 2024, at 16:36, Jordan West <jw...@apache.org> wrote:
>>>
>>>
>>> Agreed with the sentiment that decomposition is a good target but out of
>>> scope here. I’m personally excited to see an in-tree repair scheduler and
>>> am supportive of the approach shared here.
>>>
>>> Jordan
>>>
>>> On Tue, Oct 22, 2024 at 08:12 Dinesh Joshi <dj...@apache.org> wrote:
>>>
>>>> Decomposing Cassandra may be architecturally desirable but that is not
>>>> the goal of this CEP. This CEP brings value to operators today so it should
>>>> be considered on that merit. We definitely need to have a separate
>>>> conversation on Cassandra's architectural direction.
>>>>
>>>> On Tue, Oct 22, 2024 at 7:51 AM Joseph Lynch <jo...@gmail.com>
>>>> wrote:
>>>>
>>>>> Definitely like this in C* itself. We only changed our proposal to
>>>>> putting repair scheduling in the sidecar before because trunk was frozen
>>>>> for the foreseeable future at that time. With trunk unfrozen and
>>>>> development on the main process going at a fast pace I think it makes way
>>>>> more sense to integrate natively as table properties as this CEP proposes.
>>>>> Completely agree the scheduling overhead should be minimal.
>>>>>
>>>>> Moving the actual repair operation (comparing data and streaming
>>>>> mismatches) along with compaction operations to a separate process long
>>>>> term makes a lot of sense but imo only once we both have a release of
>>>>> sidecar and a contract figured out between them on communication. I'm
>>>>> watching CEP-38 there as I think CQL and virtual tables are looking much
>>>>> stronger than when we wrote CEP-1 and chose HTTP but that's for that
>>>>> discussion and not this one.
>>>>>
>>>>> -Joey
>>>>>
>>>>> On Mon, Oct 21, 2024 at 3:25 PM Francisco Guerrero <fr...@apache.org>
>>>>> wrote:
>>>>>
>>>>>> Like others have said, I was expecting the scheduling portion of
>>>>>> repair is
>>>>>> negligible. I was mostly curious if you had something handy that you
>>>>>> can
>>>>>> quickly share.
>>>>>>
>>>>>> On 2024/10/21 18:59:41 Jaydeep Chovatia wrote:
>>>>>> > >Jaydeep, do you have any metrics on your clusters comparing them
>>>>>> before
>>>>>> > and after introducing repair scheduling into the Cassandra process?
>>>>>> >
>>>>>> > Yes, I had made some comparisons when I started rolling this
>>>>>> feature out to
>>>>>> > our production five years ago :) Here are the details:
>>>>>> > *The Scheduling*
>>>>>> > The scheduling itself is exceptionally lightweight, as only one
>>>>>> additional
>>>>>> > thread monitors the repair activity, updating the status to a
>>>>>> system table
>>>>>> > once every few minutes or so. So, it does not appear anywhere in
>>>>>> the CPU
>>>>>> > charts, etc. Unfortunately, I do not have those graphs now, but I
>>>>>> can do a
>>>>>> > quick comparison if it helps!
>>>>>> >
>>>>>> > *The Repair Itself*
>>>>>> > As we all know, the Cassandra repair algorithm is a heavy-weight
>>>>>> process
>>>>>> > due to Merkle tree/streaming, etc., no matter how we schedule it.
>>>>>> But it is
>>>>>> > an orthogonal topic and folks are already discussing creating a new
>>>>>> CEP.
>>>>>> >
>>>>>> > Jaydeep
>>>>>> >
>>>>>> >
>>>>>> > On Mon, Oct 21, 2024 at 10:02 AM Francisco Guerrero <
>>>>>> frankgh@apache.org>
>>>>>> > wrote:
>>>>>> >
>>>>>> > > Jaydeep, do you have any metrics on your clusters comparing them
>>>>>> before
>>>>>> > > and after introducing repair scheduling into the Cassandra
>>>>>> process?
>>>>>> > >
>>>>>> > > On 2024/10/21 16:57:57 "J. D. Jordan" wrote:
>>>>>> > > > Sounds good. Just wanted to bring it up. I agree that the
>>>>>> scheduling bit
>>>>>> > > is
>>>>>> > > > pretty light weight and the ideal would be to bring the whole
>>>>>> of the
>>>>>> > > repair
>>>>>> > > > external, which is a much bigger can of worms to open.
>>>>>> > > >
>>>>>> > > >
>>>>>> > > >
>>>>>> > > > -Jeremiah
>>>>>> > > >
>>>>>> > > >
>>>>>> > > >
>>>>>> > > > > On Oct 21, 2024, at 11:21 AM, Chris Lohfink <
>>>>>> clohfink85@gmail.com>
>>>>>> > > wrote:
>>>>>> > > > >
>>>>>> > > > >
>>>>>> > > >
>>>>>> > > > >
>>>>>> > > > >
>>>>>> > > > > > I actually think we should be looking at how we can move
>>>>>> things out
>>>>>> > > of the
>>>>>> > > > > database process.
>>>>>> > > > >
>>>>>> > > > >
>>>>>> > > > >
>>>>>> > > > >
>>>>>> > > > >
>>>>>> > > > > While worth pursuing, I think we would need a different CEP
>>>>>> just to
>>>>>> > > figure
>>>>>> > > > > out how to do that. Not only is there a lot of infrastructure
>>>>>> > > difficulty in
>>>>>> > > > > running multi process, the inter app communication needs to
>>>>>> be figured
>>>>>> > > out
>>>>>> > > > > better then JMX. Even the sidecar we dont have a solid story
>>>>>> on how to
>>>>>> > > > > ensure both are running or anything yet. It's up to each app
>>>>>> owner to
>>>>>> > > figure
>>>>>> > > > > it out. Once we have a good thing in place I think we can
>>>>>> start moving
>>>>>> > > > > compactions, repairs, etc out of the database. Even then it's
>>>>>> the
>>>>>> > > _repairs_
>>>>>> > > > > that is expensive, not the scheduling.
>>>>>> > > > >
>>>>>> > > > >
>>>>>> > > > >
>>>>>> > > > >
>>>>>> > > > > On Mon, Oct 21, 2024 at 9:45 AM Jeremiah Jordan
>>>>>> > > > > <[jeremiah.jordan@gmail.com](mailto:jeremiah.jordan@gmail.com
>>>>>> )>
>>>>>> > > wrote:
>>>>>> > > > >
>>>>>> > > > >
>>>>>> > > >
>>>>>> > > > >> I love the idea of a repair service being there by default
>>>>>> for an
>>>>>> > > install
>>>>>> > > > of C*. My main concern here is that it is putting more
>>>>>> services into
>>>>>> > > the main
>>>>>> > > > database process. I actually think we should be looking at how
>>>>>> we can
>>>>>> > > move
>>>>>> > > > things out of the database process. The C* process being a
>>>>>> giant
>>>>>> > > monolith has
>>>>>> > > > always been a pain point. Is there anyway it makes sense for
>>>>>> this to be
>>>>>> > > an
>>>>>> > > > external process rather than a new thread pool inside the C*
>>>>>> process?
>>>>>> > > >
>>>>>> > > > >>
>>>>>> > > >
>>>>>> > > > >>
>>>>>> > > > >
>>>>>> > > > >>
>>>>>> > > >
>>>>>> > > > >> -Jeremiah Jordan
>>>>>> > > >
>>>>>> > > > >>
>>>>>> > > >
>>>>>> > > > >>
>>>>>> > > > >
>>>>>> > > > >>
>>>>>> > > >
>>>>>> > > > >> On Oct 18, 2024 at 2:58:15 PM, Mick Semb Wever
>>>>>> > > > <[mck@apache.org](mailto:mck@apache.org)> wrote:
>>>>>> > > > >
>>>>>> > > > >>
>>>>>> > > >
>>>>>> > > > >>>
>>>>>> > > > >
>>>>>> > > > >>>
>>>>>> > > >
>>>>>> > > > >>> This is looking strong, thanks Jaydeep.
>>>>>> > > >
>>>>>> > > > >>>
>>>>>> > > >
>>>>>> > > > >>>
>>>>>> > > > >
>>>>>> > > > >>>
>>>>>> > > >
>>>>>> > > > >>> I would suggest folk take a look at the design doc and the
>>>>>> PR in the
>>>>>> > > CEP.
>>>>>> > > > A lot is there (that I have completely missed).
>>>>>> > > >
>>>>>> > > > >>>
>>>>>> > > >
>>>>>> > > > >>>
>>>>>> > > > >
>>>>>> > > > >>>
>>>>>> > > >
>>>>>> > > > >>> I would especially ask all authors of prior art (Reaper, DSE
>>>>>> > > nodesync,
>>>>>> > > > ecchronos) to take a final review of the proposal
>>>>>> > > > >
>>>>>> > > > >>>
>>>>>> > > >
>>>>>> > > > >>>
>>>>>> > > > >
>>>>>> > > > >>>
>>>>>> > > >
>>>>>> > > > >>> Jaydeep, can we ask for a two week window while we reach
>>>>>> out to these
>>>>>> > > > people ? There's a lot of prior art in this space, and it
>>>>>> feels like
>>>>>> > > we're in
>>>>>> > > > a good place now where it's clear this has legs and we can use
>>>>>> that to
>>>>>> > > bring
>>>>>> > > > folk in and make sure there's no remaining blindspots.
>>>>>> > > >
>>>>>> > > > >>>
>>>>>> > > >
>>>>>> > > > >>>
>>>>>> > > > >
>>>>>> > > > >>>
>>>>>> > > >
>>>>>> > > > >>>
>>>>>> > > > >
>>>>>> > > > >>>
>>>>>> > > >
>>>>>> > > > >>> On Fri, 18 Oct 2024 at 01:40, Jaydeep Chovatia
>>>>>> > > > <[chovatia.jaydeep@gmail.com](mailto:chovatia.jaydeep@gmail.com
>>>>>> )>
>>>>>> > > wrote:
>>>>>> > > > >
>>>>>> > > > >>>
>>>>>> > > >
>>>>>> > > > >>>> Sorry, there is a typo in the CEP-37 link; here is the
>>>>>> correct
>>>>>> > > > [link](
>>>>>> > >
>>>>>> https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-37+Apache+Cassandra+Unified+Repair+Solution
>>>>>> > > )
>>>>>> > > >
>>>>>> > > > >>>>
>>>>>> > > >
>>>>>> > > > >>>>
>>>>>> > > > >
>>>>>> > > > >>>>
>>>>>> > > >
>>>>>> > > > >>>>
>>>>>> > > > >
>>>>>> > > > >>>>
>>>>>> > > >
>>>>>> > > > >>>> On Thu, Oct 17, 2024 at 4:36 PM Jaydeep Chovatia
>>>>>> > > > <[chovatia.jaydeep@gmail.com](mailto:chovatia.jaydeep@gmail.com
>>>>>> )>
>>>>>> > > wrote:
>>>>>> > > > >
>>>>>> > > > >>>>
>>>>>> > > >
>>>>>> > > > >>>>> First, thank you for your patience while we strengthened
>>>>>> the
>>>>>> > > CEP-37.
>>>>>> > > >
>>>>>> > > > >>>>>
>>>>>> > > >
>>>>>> > > > >>>>>
>>>>>> > > > >
>>>>>> > > > >>>>>
>>>>>> > > >
>>>>>> > > > >>>>> Over the last eight months, Chris Lohfink, Andy Tolbert,
>>>>>> Josh
>>>>>> > > McKenzie,
>>>>>> > > > Dinesh Joshi, Kristijonas Zalys, and I have done tons of work
>>>>>> (online
>>>>>> > > > discussions/a dedicated Slack channel
>>>>>> > > #cassandra-repair-scheduling-cep37) to
>>>>>> > > > come up with the best possible design that not only
>>>>>> significantly
>>>>>> > > simplifies
>>>>>> > > > repair operations but also includes the most common features
>>>>>> that
>>>>>> > > everyone
>>>>>> > > > will benefit from running at Scale.
>>>>>> > > >
>>>>>> > > > >>>>>
>>>>>> > > >
>>>>>> > > > >>>>> For example,
>>>>>> > > >
>>>>>> > > > >>>>>
>>>>>> > > >
>>>>>> > > > >>>>> * Apache Cassandra must be capable of running multiple
>>>>>> repair
>>>>>> > > types,
>>>>>> > > > such as Full, Incremental, Paxos, and Preview - so the
>>>>>> framework should
>>>>>> > > be
>>>>>> > > > easily extendable with no additional overhead from the
>>>>>> operator’s point
>>>>>> > > of
>>>>>> > > > view.
>>>>>> > > >
>>>>>> > > > >>>>>
>>>>>> > > >
>>>>>> > > > >>>>> * An easy way to extend the token-split calculation
>>>>>> algorithm
>>>>>> > > with a
>>>>>> > > > default implementation should exist.
>>>>>> > > >
>>>>>> > > > >>>>>
>>>>>> > > >
>>>>>> > > > >>>>> * Running incremental repair reliably at Scale is pretty
>>>>>> > > challenging,
>>>>>> > > > so we need to place safeguards, such as migration/rollback w/o
>>>>>> restart
>>>>>> > > and
>>>>>> > > > stopping incremental repair automatically if the disk is about
>>>>>> to get
>>>>>> > > full.
>>>>>> > > >
>>>>>> > > > >>>>>
>>>>>> > > >
>>>>>> > > > >>>>>
>>>>>> > > >
>>>>>> > > > >>>>>
>>>>>> > > >
>>>>>> > > > >>>>> We are glad to inform you that CEP-37 (i.e., Repair inside
>>>>>> > > Cassandra) is
>>>>>> > > > now officially ready for review after multiple rounds of design,
>>>>>> > > testing, code
>>>>>> > > > reviews, documentation reviews, and, more importantly,
>>>>>> validation that
>>>>>> > > it runs
>>>>>> > > > at Scale!
>>>>>> > > >
>>>>>> > > > >>>>>
>>>>>> > > >
>>>>>> > > > >>>>>
>>>>>> > > > >
>>>>>> > > > >>>>>
>>>>>> > > >
>>>>>> > > > >>>>> Some facts about CEP-37.
>>>>>> > > >
>>>>>> > > > >>>>>
>>>>>> > > >
>>>>>> > > > >>>>> * Multiple members have verified all aspects of CEP-37
>>>>>> numerous
>>>>>> > > times.
>>>>>> > > >
>>>>>> > > > >>>>>
>>>>>> > > >
>>>>>> > > > >>>>> * The design proposed in CEP-37 has been thoroughly
>>>>>> tried and
>>>>>> > > tested
>>>>>> > > > on an immense scale (hundreds of unique Cassandra clusters,
>>>>>> tens of
>>>>>> > > thousands
>>>>>> > > > of Cassandra nodes, with tens of millions of QPS) on top of 4.1
>>>>>> > > open-source
>>>>>> > > > for more than five years; please see more details[
>>>>>> > > > here](
>>>>>> > >
>>>>>> https://www.uber.com/en-US/blog/how-uber-optimized-cassandra-operations-
>>>>>> > > > at-scale/).
>>>>>> > > >
>>>>>> > > > >>>>>
>>>>>> > > >
>>>>>> > > > >>>>> * The following
>>>>>> > > > [presentation](
>>>>>> > >
>>>>>> https://docs.google.com/presentation/d/1Zilww9c7LihHULk_ckErI2s4XbObxjWknKqRtbvHyZc/edit#slide=id.g30a4fd4fcf7_0_13
>>>>>> > > )
>>>>>> > > > highlights the rigorous applied to CEP-37, which was given
>>>>>> during last
>>>>>> > > week’s
>>>>>> > > > Apache Cassandra Bay Area [Meetup](
>>>>>> > > https://www.meetup.com/apache-cassandra-
>>>>>> > > > bay-area/events/303469006/),
>>>>>> > > >
>>>>>> > > > >>>>>
>>>>>> > > >
>>>>>> > > > >>>>>
>>>>>> > > >
>>>>>> > > >
>>>>>> > > > >
>>>>>> > > > >>>>>
>>>>>> > > >
>>>>>> > > > >>>>> Since things are massively overhauled, we believe it is
>>>>>> almost
>>>>>> > > ready for
>>>>>> > > > a final pass pre-VOTE. We would like you to please review the
>>>>>> > > > [CEP-37](
>>>>>> > >
>>>>>> https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-37+Apache+Cassandra+Unified+Repair+Solution\
>>>>>> <https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-37+Apache+Cassandra+Unified+Repair+Solution%5C>
>>>>>> > > <
>>>>>> https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-37+Apache+Cassandra+Unified+Repair+Solution%5C
>>>>>> >
>>>>>> > > ))
>>>>>> > > > and the associated detailed design
>>>>>> > > > [doc](https://docs.google.com/document/d/1CJWxjEi-
>>>>>> > > >
>>>>>> mBABPMZ3VWJ9w5KavWfJETAGxfUpsViPcPo/edit#heading=h.r112r46toau0).
>>>>>> > > >
>>>>>> > > > >>>>>
>>>>>> > > >
>>>>>> > > > >>>>>
>>>>>> > > > >
>>>>>> > > > >>>>>
>>>>>> > > >
>>>>>> > > > >>>>> Thank you everyone!
>>>>>> > > >
>>>>>> > > > >>>>>
>>>>>> > > >
>>>>>> > > > >>>>> Chris, Andy, Josh, Dinesh, Kristijonas, and Jaydeep
>>>>>> > > >
>>>>>> > > > >>>>>
>>>>>> > > >
>>>>>> > > > >>>>>
>>>>>> > > > >
>>>>>> > > > >
>>>>>> > > > >>>>>
>>>>>> > > >
>>>>>> > > > >>>>>
>>>>>> > > > >
>>>>>> > > > >>>>>
>>>>>> > > >
>>>>>> > > > >>>>> On Thu, Sep 19, 2024 at 11:26 AM Josh McKenzie
>>>>>> > > > <[jmckenzie@apache.org](mailto:jmckenzie@apache.org)> wrote:
>>>>>> > > > >
>>>>>> > > > >>>>>
>>>>>> > > >
>>>>>> > > > >>>>>> __
>>>>>> > > >
>>>>>> > > > >>>>>>
>>>>>> > > >
>>>>>> > > > >>>>>> Not quite; finishing touches on the CEP and design doc
>>>>>> are in
>>>>>> > > flight
>>>>>> > > > (as of last / this week).
>>>>>> > > > >
>>>>>> > > > >>>>>>
>>>>>> > > >
>>>>>> > > > >>>>>>
>>>>>> > > > >
>>>>>> > > > >>>>>>
>>>>>> > > >
>>>>>> > > > >>>>>> Soon(tm).
>>>>>> > > >
>>>>>> > > > >>>>>>
>>>>>> > > >
>>>>>> > > > >>>>>>
>>>>>> > > > >
>>>>>> > > > >>>>>>
>>>>>> > > >
>>>>>> > > > >>>>>> On Thu, Sep 19, 2024, at 2:07 PM, Patrick McFadin wrote:
>>>>>> > > > >
>>>>>> > > > >>>>>>
>>>>>> > > >
>>>>>> > > > >>>>>>> Is this CEP ready for a VOTE thread?
>>>>>> > > > <
>>>>>> > >
>>>>>> https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-37+%28DRAFT%29+Apache+Cassandra+Unified+Repair+Solution
>>>>>> >
>>>>>> > >
>>>>>> > > > >
>>>>>> > > > >>>>>>>
>>>>>> > > >
>>>>>> > > > >>>>>>>
>>>>>> > > > >
>>>>>> > > > >>>>>>>
>>>>>> > > >
>>>>>> > > > >>>>>>> On Sun, Feb 25, 2024 at 12:25 PM Jaydeep Chovatia
>>>>>> > > > <[chovatia.jaydeep@gmail.com](mailto:chovatia.jaydeep@gmail.com
>>>>>> )>
>>>>>> > > wrote:
>>>>>> > > > >
>>>>>> > > > >>>>>>>
>>>>>> > > >
>>>>>> > > > >>>>>>>> Thanks, Josh. I've just updated the
>>>>>> > > > [CEP](
>>>>>> > >
>>>>>> https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-37+%28DRAFT%29+Apache+Cassandra+Official+Repair+Solution
>>>>>> > > )
>>>>>> > > > and included all the solutions you mentioned below.
>>>>>> > > > >
>>>>>> > > > >>>>>>>>
>>>>>> > > >
>>>>>> > > > >>>>>>>>
>>>>>> > > > >
>>>>>> > > > >>>>>>>>
>>>>>> > > >
>>>>>> > > > >>>>>>>> Jaydeep
>>>>>> > > > >
>>>>>> > > > >>>>>>>>
>>>>>> > > >
>>>>>> > > > >>>>>>>>
>>>>>> > > > >
>>>>>> > > > >>>>>>>>
>>>>>> > > >
>>>>>> > > > >>>>>>>> On Thu, Feb 22, 2024 at 9:33 AM Josh McKenzie
>>>>>> > > > <[jmckenzie@apache.org](mailto:jmckenzie@apache.org)> wrote:
>>>>>> > > > >
>>>>>> > > > >>>>>>>>
>>>>>> > > >
>>>>>> > > > >>>>>>>>> __
>>>>>> > > > >
>>>>>> > > > >>>>>>>>>
>>>>>> > > >
>>>>>> > > > >>>>>>>>> Very late response from me here (basically necro'ing
>>>>>> this
>>>>>> > > thread).
>>>>>> > > > >
>>>>>> > > > >>>>>>>>>
>>>>>> > > >
>>>>>> > > > >>>>>>>>>
>>>>>> > > > >
>>>>>> > > > >>>>>>>>>
>>>>>> > > >
>>>>>> > > > >>>>>>>>> I think it'd be useful to get this condensed into a
>>>>>> CEP that
>>>>>> > > we can
>>>>>> > > > then discuss in that format. It's clearly something we all
>>>>>> agree we need
>>>>>> > > and
>>>>>> > > > having an implementation that works, even if it's not in your
>>>>>> preferred
>>>>>> > > > execution domain, is vastly better than nothing IMO.
>>>>>> > > > >
>>>>>> > > > >>>>>>>>>
>>>>>> > > >
>>>>>> > > > >>>>>>>>>
>>>>>> > > > >
>>>>>> > > > >>>>>>>>>
>>>>>> > > >
>>>>>> > > > >>>>>>>>> I don't have cycles (nor background ;) ) to do that,
>>>>>> but it
>>>>>> > > sounds
>>>>>> > > > like you do Jaydeep given the implementation you have on a
>>>>>> private fork +
>>>>>> > > > design.
>>>>>> > > > >
>>>>>> > > > >>>>>>>>>
>>>>>> > > >
>>>>>> > > > >>>>>>>>>
>>>>>> > > > >
>>>>>> > > > >>>>>>>>>
>>>>>> > > >
>>>>>> > > > >>>>>>>>> A non-exhaustive list of things that might be useful
>>>>>> > > incorporating
>>>>>> > > > into or referencing from a CEP:
>>>>>> > > > >
>>>>>> > > > >>>>>>>>>
>>>>>> > > >
>>>>>> > > > >>>>>>>>> Slack thread: <https://the-
>>>>>> > > > asf.slack.com/archives/CK23JSY2K/p1690225062383619>
>>>>>> > > > >
>>>>>> > > > >>>>>>>>>
>>>>>> > > >
>>>>>> > > > >>>>>>>>> Joey's old C* ticket:
>>>>>> > > > <https://issues.apache.org/jira/browse/CASSANDRA-14346>
>>>>>> > > > >
>>>>>> > > > >>>>>>>>>
>>>>>> > > >
>>>>>> > > > >>>>>>>>> Even older automatic repair scheduling:
>>>>>> > > > <https://issues.apache.org/jira/browse/CASSANDRA-10070>
>>>>>> > > > >
>>>>>> > > > >>>>>>>>>
>>>>>> > > >
>>>>>> > > > >>>>>>>>> Your design gdoc: <
>>>>>> > > https://docs.google.com/document/d/1CJWxjEi-
>>>>>> > > > mBABPMZ3VWJ9w5KavWfJETAGxfUpsViPcPo/edit#heading=h.r112r46toau0>
>>>>>> > > > >
>>>>>> > > > >>>>>>>>>
>>>>>> > > >
>>>>>> > > > >>>>>>>>> PR with automated repair:
>>>>>> > > > <
>>>>>> > >
>>>>>> https://github.com/jaydeepkumar1984/cassandra/commit/ef6456d652c0d07cf29d88dfea03b73704814c2c
>>>>>> >
>>>>>> > >
>>>>>> > > > >
>>>>>> > > > >>>>>>>>>
>>>>>> > > >
>>>>>> > > > >>>>>>>>>
>>>>>> > > > >
>>>>>> > > > >>>>>>>>>
>>>>>> > > >
>>>>>> > > > >>>>>>>>> My intuition is that we're all basically in agreement
>>>>>> that
>>>>>> > > this is
>>>>>> > > > something the DB needs, we're all willing to bikeshed for our
>>>>>> personal
>>>>>> > > > preference on where it lives and how it's implemented, and at
>>>>>> the end of
>>>>>> > > the
>>>>>> > > > day, code talks. I don't think anyone's said they'll die on the
>>>>>> hill of
>>>>>> > > > implementation details, so that feels like CEP time to me.
>>>>>> > > > >
>>>>>> > > > >>>>>>>>>
>>>>>> > > >
>>>>>> > > > >>>>>>>>>
>>>>>> > > > >
>>>>>> > > > >>>>>>>>>
>>>>>> > > >
>>>>>> > > > >>>>>>>>> If you were willing and able to get a CEP together for
>>>>>> > > automated
>>>>>> > > > repair based on the above material, given you've done the work
>>>>>> and have
>>>>>> > > the
>>>>>> > > > proof points it's working at scale, I think this would be a
>>>>>> _huge
>>>>>> > > > contribution_ to the community.
>>>>>> > > > >
>>>>>> > > > >>>>>>>>>
>>>>>> > > >
>>>>>> > > > >>>>>>>>>
>>>>>> > > > >
>>>>>> > > > >>>>>>>>>
>>>>>> > > >
>>>>>> > > > >>>>>>>>> On Thu, Aug 24, 2023, at 7:26 PM, Jaydeep Chovatia
>>>>>> wrote:
>>>>>> > > > >
>>>>>> > > > >>>>>>>>>
>>>>>> > > >
>>>>>> > > > >>>>>>>>>> Is anyone going to file an official CEP for this?
>>>>>> > > > >
>>>>>> > > > >>>>>>>>>>
>>>>>> > > >
>>>>>> > > > >>>>>>>>>> As mentioned in this email thread, here is one of the
>>>>>> > > solution's
>>>>>> > > > [design doc](https://docs.google.com/document/d/1CJWxjEi-
>>>>>> > > >
>>>>>> mBABPMZ3VWJ9w5KavWfJETAGxfUpsViPcPo/edit#heading=h.r112r46toau0) and
>>>>>> > > source
>>>>>> > > > code on a private Apache Cassandra patch. Could you go through
>>>>>> it and
>>>>>> > > let me
>>>>>> > > > know what you think?
>>>>>> > > > >
>>>>>> > > > >>>>>>>>>>
>>>>>> > > >
>>>>>> > > > >>>>>>>>>>
>>>>>> > > > >
>>>>>> > > > >>>>>>>>>>
>>>>>> > > >
>>>>>> > > > >>>>>>>>>> Jaydeep
>>>>>> > > > >
>>>>>> > > > >>>>>>>>>>
>>>>>> > > >
>>>>>> > > > >>>>>>>>>>
>>>>>> > > > >
>>>>>> > > > >>>>>>>>>>
>>>>>> > > >
>>>>>> > > > >>>>>>>>>> On Wed, Aug 2, 2023 at 3:54 PM Jon Haddad
>>>>>> > > > <[rustyrazorblade@apache.org](mailto:rustyrazorblade@apache.org
>>>>>> )>
>>>>>> > > wrote:
>>>>>> > > > >
>>>>>> > > > >>>>>>>>>>
>>>>>> > > >
>>>>>> > > > >>>>>>>>>>> > That said I would happily support an effort to
>>>>>> bring repair
>>>>>> > > > scheduling to the sidecar immediately. This has nothing
>>>>>> blocking it, and
>>>>>> > > would
>>>>>> > > > potentially enable the sidecar to provide an official repair
>>>>>> scheduling
>>>>>> > > > solution that is compatible with current or even previous
>>>>>> versions of the
>>>>>> > > > database.
>>>>>> > > > >
>>>>>> > > > >>>>>>>>>>>
>>>>>> > > >
>>>>>> > > > >>>>>>>>>>>
>>>>>> > > > >
>>>>>> > > > >>>>>>>>>>>
>>>>>> > > >
>>>>>> > > > >>>>>>>>>>> This is something I hadn't thought much about, and
>>>>>> is a
>>>>>> > > pretty
>>>>>> > > > good argument for using the sidecar initially. There's a lot of
>>>>>> > > deployments
>>>>>> > > > out there and having an official repair option would be a big
>>>>>> win.
>>>>>> > > > >
>>>>>> > > > >>>>>>>>>>>
>>>>>> > > >
>>>>>> > > > >>>>>>>>>>>
>>>>>> > > > >
>>>>>> > > > >>>>>>>>>>>
>>>>>> > > >
>>>>>> > > > >>>>>>>>>>>
>>>>>> > > > >
>>>>>> > > > >>>>>>>>>>>
>>>>>> > > >
>>>>>> > > > >>>>>>>>>>> On 2023/07/26 23:20:07 "C. Scott Andreas" wrote:
>>>>>> > > > >
>>>>>> > > > >>>>>>>>>>>
>>>>>> > > >
>>>>>> > > > >>>>>>>>>>> > I agree that it would be ideal for Cassandra to
>>>>>> have a
>>>>>> > > repair
>>>>>> > > > scheduler in-DB.
>>>>>> > > > >
>>>>>> > > > >>>>>>>>>>>
>>>>>> > > >
>>>>>> > > > >>>>>>>>>>> >
>>>>>> > > > >
>>>>>> > > > >>>>>>>>>>>
>>>>>> > > >
>>>>>> > > > >>>>>>>>>>> > That said I would happily support an effort to
>>>>>> bring repair
>>>>>> > > > scheduling to the sidecar immediately. This has nothing
>>>>>> blocking it, and
>>>>>> > > would
>>>>>> > > > potentially enable the sidecar to provide an official repair
>>>>>> scheduling
>>>>>> > > > solution that is compatible with current or even previous
>>>>>> versions of the
>>>>>> > > > database.
>>>>>> > > > >
>>>>>> > > > >>>>>>>>>>>
>>>>>> > > >
>>>>>> > > > >>>>>>>>>>> >
>>>>>> > > > >
>>>>>> > > > >>>>>>>>>>>
>>>>>> > > >
>>>>>> > > > >>>>>>>>>>> > Once TCM has landed, we’ll have much stronger
>>>>>> primitives
>>>>>> > > for
>>>>>> > > > repair orchestration in the database itself. But I don’t think
>>>>>> that
>>>>>> > > should
>>>>>> > > > block progress on a repair scheduling solution in the sidecar,
>>>>>> and there
>>>>>> > > is
>>>>>> > > > nothing that would prevent someone from continuing to use a
>>>>>> sidecar-based
>>>>>> > > > solution in perpetuity if they preferred.
>>>>>> > > > >
>>>>>> > > > >>>>>>>>>>>
>>>>>> > > >
>>>>>> > > > >>>>>>>>>>> >
>>>>>> > > > >
>>>>>> > > > >>>>>>>>>>>
>>>>>> > > >
>>>>>> > > > >>>>>>>>>>> > \- Scott
>>>>>> > > > >
>>>>>> > > > >>>>>>>>>>>
>>>>>> > > >
>>>>>> > > > >>>>>>>>>>> >
>>>>>> > > > >
>>>>>> > > > >>>>>>>>>>>
>>>>>> > > >
>>>>>> > > > >>>>>>>>>>> > > On Jul 26, 2023, at 3:25 PM, Jon Haddad
>>>>>> > > > <[rustyrazorblade@apache.org](mailto:rustyrazorblade@apache.org
>>>>>> )>
>>>>>> > > wrote:
>>>>>> > > > >
>>>>>> > > > >>>>>>>>>>>
>>>>>> > > >
>>>>>> > > > >>>>>>>>>>> > >
>>>>>> > > > >
>>>>>> > > > >>>>>>>>>>>
>>>>>> > > >
>>>>>> > > > >>>>>>>>>>> > > I'm 100% in favor of repair being part of the
>>>>>> core DB,
>>>>>> > > not
>>>>>> > > > the sidecar. The current (and past) state of things where
>>>>>> running the DB
>>>>>> > > > correctly *requires* running a separate process (either
>>>>>> community
>>>>>> > > maintained
>>>>>> > > > or official C* sidecar) is incredibly painful for folks. The
>>>>>> idea that
>>>>>> > > your
>>>>>> > > > data integrity needs to be opt-in has never made sense to me
>>>>>> from the
>>>>>> > > > perspective of either the product or the end user.
>>>>>> > > > >
>>>>>> > > > >>>>>>>>>>>
>>>>>> > > >
>>>>>> > > > >>>>>>>>>>> > >
>>>>>> > > > >
>>>>>> > > > >>>>>>>>>>>
>>>>>> > > >
>>>>>> > > > >>>>>>>>>>> > > I've worked with way too many teams that have
>>>>>> either
>>>>>> > > > configured this incorrectly or not at all.
>>>>>> > > > >
>>>>>> > > > >>>>>>>>>>>
>>>>>> > > >
>>>>>> > > > >>>>>>>>>>> > >
>>>>>> > > > >
>>>>>> > > > >>>>>>>>>>>
>>>>>> > > >
>>>>>> > > > >>>>>>>>>>> > > Ideally Cassandra would ship with repair built
>>>>>> in and on
>>>>>> > > by
>>>>>> > > > default. Power users can disable if they want to continue to
>>>>>> maintain
>>>>>> > > their
>>>>>> > > > own repair tooling for some reason.
>>>>>> > > > >
>>>>>> > > > >>>>>>>>>>>
>>>>>> > > >
>>>>>> > > > >>>>>>>>>>> > >
>>>>>> > > > >
>>>>>> > > > >>>>>>>>>>>
>>>>>> > > >
>>>>>> > > > >>>>>>>>>>> > > Jon
>>>>>> > > > >
>>>>>> > > > >>>>>>>>>>>
>>>>>> > > >
>>>>>> > > > >>>>>>>>>>> > >
>>>>>> > > > >
>>>>>> > > > >>>>>>>>>>>
>>>>>> > > >
>>>>>> > > > >>>>>>>>>>> > >> On 2023/07/24 20:44:14 German Eichberger via
>>>>>> dev
>>>>>> > > wrote:
>>>>>> > > > >
>>>>>> > > > >>>>>>>>>>>
>>>>>> > > >
>>>>>> > > > >>>>>>>>>>> > >> All,
>>>>>> > > > >
>>>>>> > > > >>>>>>>>>>>
>>>>>> > > >
>>>>>> > > > >>>>>>>>>>> > >> We had a brief discussion in [2] about the
>>>>>> Uber article
>>>>>> > > [1]
>>>>>> > > > where they talk about having integrated repair into Cassandra
>>>>>> and how
>>>>>> > > great
>>>>>> > > > that is. I expressed my disappointment that they didn't work
>>>>>> with the
>>>>>> > > > community on that (Uber, if you are listening time to make
>>>>>> amends 🙂)
>>>>>> > > and it
>>>>>> > > > turns out Joey already had the idea and wrote the code [3] - so
>>>>>> I wanted
>>>>>> > > to
>>>>>> > > > start a discussion to gauge interest and maybe how to revive
>>>>>> that
>>>>>> > > effort.
>>>>>> > > > >
>>>>>> > > > >>>>>>>>>>>
>>>>>> > > >
>>>>>> > > > >>>>>>>>>>> > >> Thanks,
>>>>>> > > > >
>>>>>> > > > >>>>>>>>>>>
>>>>>> > > >
>>>>>> > > > >>>>>>>>>>> > >> German
>>>>>> > > > >
>>>>>> > > > >>>>>>>>>>>
>>>>>> > > >
>>>>>> > > > >>>>>>>>>>> > >> [1] <
>>>>>> > > https://www.uber.com/blog/how-uber-optimized-cassandra-
>>>>>> > > > operations-at-scale/>
>>>>>> > > > >
>>>>>> > > > >>>>>>>>>>>
>>>>>> > > >
>>>>>> > > > >>>>>>>>>>> > >> [2] <https://the-
>>>>>> > > > asf.slack.com/archives/CK23JSY2K/p1690225062383619>
>>>>>> > > > >
>>>>>> > > > >>>>>>>>>>>
>>>>>> > > >
>>>>>> > > > >>>>>>>>>>> > >> [3] <
>>>>>> > > https://issues.apache.org/jira/browse/CASSANDRA-14346>
>>>>>> > > > >
>>>>>> > > > >>>>>>>>>>>
>>>>>> > > >
>>>>>> > > > >>>>>>>>>>> >
>>>>>> > > > >
>>>>>> > > > >>>>>>>>>
>>>>>> > > >
>>>>>> > > > >>>>>>>>>
>>>>>> > > > >
>>>>>> > > > >>>>>>
>>>>>> > > >
>>>>>> > > > >>>>>>
>>>>>> > > > >
>>>>>> > > >
>>>>>> > > >
>>>>>> > >
>>>>>> >
>>>>>>
>>>>>
Re: [Discuss] Repair inside C*
Posted by Jaydeep Chovatia <ch...@gmail.com>.
Thanks a lot, Alexander, for the review! Please find my response below:
> making these replicas process 3 concurrent repairs while others could be
left uninvolved in any repair at all...Taking a range centric approach
(we're not repairing nodes, we're repairing the token ranges) allows to
spread the load evenly without overlap in the replica sets.
The scheduler repairs, by default, the primary ranges for all the nodes
going through the repair. Since it uses the primary ranges, all the nodes
repairing parallelly would not overlap in any form for the primary ranges.
However, the replica set for the nodes going through repair may or may not
overlap, but it totally depends on the cluster size and parallelism used.
If a cluster is small, there is a possibility, but if it is large, the
possibility reduces. Even if we go with a range-centric approach and if we
repair N token ranges in parallel, there is no guarantee that their replica
sets won't overlap for smaller clusters.
> I'm more worried even with incremental repair here, because you might end
up with some conflicts around sstables which would be in the pending repair
pool but would be needed by a competing repair job.
This can happen regardless of whether we go by "node-centric" vs.
"range-centric" if we run multiple parallel repair sessions. The reason is
that SSTables for all the nodes going through repair may not be physically
isolated 1:1 as per the token ranges being repaired. We just had a detailed
discussion about the SSTable overlap for incremental repair (IR) last week
in Slack (#cassandra-repair-scheduling-cep37), and the general consensus
was that there is no better way to address it than just to retry a few
times. The only solution is to reduce the repair parallelism to one. node
at a time.
The ideal and reliable way to repair IR is to calculate the token ranges
based on the unrepaired data size and also apply the upper cap on the data
size being repaired. The good news is that Andy Tolbert already extended
the CEP-37 MVP for this, and he is working on making it perfect by adding
necessary tests, etc., so it can be landed on top of this MVP. tl;dr Andy T
and Chris L are already on top of this and soon it will be available on top
of CEP-37 MVP.
>I don't know if in the latest versions such sstables would be totally
ignored or if the competing repair job would fail.
The competing IR session would be aborted, and the scheduler would retry a
few times.
>Continuous repair might create a lot of overhead for full repairs which
often don't require more than 1 run per week.
This is supported with the MVP, we can set "min_repair_interval: 7d" (the
default is 24h) and the nodes will repair only once every 7 days.
>It also will not allow running a mix of scheduled full/incremental repairs
The MVP implementation allows running full and incremental repairs (and
Preview repair code changes are done and it is coming soon) independently
and in parallel. One can set the above config for each repair type with
their preferred schedule.
>Here, nodes will be processed sequentially and each node will process the
keyspaces sequentially, tying the repair cycle of all keyspaces together.
The keyspaces and tables on each node will be randomly shuffled to avoid
multiple nodes working on the same table/keyspaces.
>There are many cases where one might have differentiated gc_grace_seconds
settings to optimize reclaiming tombstones when applicable. That requires
having some fine control over the repair cycle for a given keyspace/set of
tables.
As I mentioned, there is already a way to schedule a frequency of repair
cycle, but the frequency is currently a global config on a node; hence
applicable to all the tables on a node. However, the MVP design is flexible
enough to be easily extended to add the schedule as a new CQL table-level
property, which will then honor the table-level schedule as opposed to a
global schedule. There was another suggestion from @masokol (from
Ecchronos) to maybe assign a repair priority on a table level to prioritize
one table over the other, and that can also solve this problem, which is
also feasible on top of the MVP. I have already created a ticket to add
this as an enhancement https://issues.apache.org/jira/browse/CASSANDRA-20013
>I think the 3 hours timeout might be quite large and probably means a lot
of data is being repaired for each split. That usually involves some level
of overstreaming
This timeout applies to unstuck stuck repair sessions due to some bug in
the repair code path. e.g.
https://issues.apache.org/jira/browse/CASSANDRA-14674
If a repair session finishes gracefully, then this timeout is not
applicable. Anyway, I do not have any strong opinion on the value. I am
open to lowering it to *1h* or something.
Jaydeep
On Mon, Oct 28, 2024 at 4:45 AM Alexander DEJANOVSKI <ad...@gmail.com>
wrote:
> Hi Jaydeep,
>
> I've taken a look at the proposed design and have a few comments/questions.
> As one of the maintainers of Reaper, I'm looking this through the lens of
> how Reaper does things.
>
>
> *The approach taken in the CEP-37 design is "node-centric" vs a "range
> centric" approach (which is the one Reaper takes).*I'm worried that this
> will not allow spreading the repair load evenly across the cluster, since
> nodes are the concurrency unit. You could allow running repair on 3 nodes
> concurrently for example, but these 3 nodes could all involve the same
> replicas, making these replicas process 3 concurrent repairs while others
> could be left uninvolved in any repair at all.
> Taking a range centric approach (we're not repairing nodes, we're
> repairing the token ranges) allows to spread the load evenly without
> overlap in the replica sets.
> I'm more worried even with incremental repair here, because you might end
> up with some conflicts around sstables which would be in the pending repair
> pool but would be needed by a competing repair job.
> I don't know if in the latest versions such sstables would be totally
> ignored or if the competing repair job would fail.
>
> *Each repair command will repair all keyspaces (with the ability to fully
> exclude some tables) and **I haven't seen a notion of schedule which
> seems to suggest repairs are running continuously (unless I missed
> something?).*
> There are many cases where one might have differentiated gc_grace_seconds
> settings to optimize reclaiming tombstones when applicable. That requires
> having some fine control over the repair cycle for a given keyspace/set of
> tables.
> Here, nodes will be processed sequentially and each node will process the
> keyspaces sequentially, tying the repair cycle of all keyspaces together.
> If one of the ranges for a specific keyspace cannot be repaired within the
> 3 hours timeout, it could block all the other keyspaces repairs.
> Continuous repair might create a lot of overhead for full repairs which
> often don't require more than 1 run per week.
> It also will not allow running a mix of scheduled full/incremental repairs
> (I'm unsure if that is still a recommendation, but it was still recommended
> not so long ago)
>
> *The timeout base duration is large*
> I think the 3 hours timeout might be quite large and probably means a lot
> of data is being repaired for each split. That usually involves some level
> of overstreaming. I don't have numbers to support this, it's more about my
> own experience on sizing splits in production with Reaper to reduce the
> impact as much as possible on cluster performance.
> We use 30 minutes as default in Reaper with subsequent attempts growing
> the timeout dynamically for challenging splits.
>
> Finally thanks for picking this up, I'm eager to see Reaper not being
> needed anymore and having the database manage its own repairs!
>
>
> Le mar. 22 oct. 2024 à 21:10, Benedict <be...@apache.org> a écrit :
>
>> I realise it’s out of scope, but to counterbalance all of the
>> pro-decomposition messages I wanted to chime in with a strong -1. But we
>> can debate that in a suitable context later.
>>
>> On 22 Oct 2024, at 16:36, Jordan West <jw...@apache.org> wrote:
>>
>>
>> Agreed with the sentiment that decomposition is a good target but out of
>> scope here. I’m personally excited to see an in-tree repair scheduler and
>> am supportive of the approach shared here.
>>
>> Jordan
>>
>> On Tue, Oct 22, 2024 at 08:12 Dinesh Joshi <dj...@apache.org> wrote:
>>
>>> Decomposing Cassandra may be architecturally desirable but that is not
>>> the goal of this CEP. This CEP brings value to operators today so it should
>>> be considered on that merit. We definitely need to have a separate
>>> conversation on Cassandra's architectural direction.
>>>
>>> On Tue, Oct 22, 2024 at 7:51 AM Joseph Lynch <jo...@gmail.com>
>>> wrote:
>>>
>>>> Definitely like this in C* itself. We only changed our proposal to
>>>> putting repair scheduling in the sidecar before because trunk was frozen
>>>> for the foreseeable future at that time. With trunk unfrozen and
>>>> development on the main process going at a fast pace I think it makes way
>>>> more sense to integrate natively as table properties as this CEP proposes.
>>>> Completely agree the scheduling overhead should be minimal.
>>>>
>>>> Moving the actual repair operation (comparing data and streaming
>>>> mismatches) along with compaction operations to a separate process long
>>>> term makes a lot of sense but imo only once we both have a release of
>>>> sidecar and a contract figured out between them on communication. I'm
>>>> watching CEP-38 there as I think CQL and virtual tables are looking much
>>>> stronger than when we wrote CEP-1 and chose HTTP but that's for that
>>>> discussion and not this one.
>>>>
>>>> -Joey
>>>>
>>>> On Mon, Oct 21, 2024 at 3:25 PM Francisco Guerrero <fr...@apache.org>
>>>> wrote:
>>>>
>>>>> Like others have said, I was expecting the scheduling portion of
>>>>> repair is
>>>>> negligible. I was mostly curious if you had something handy that you
>>>>> can
>>>>> quickly share.
>>>>>
>>>>> On 2024/10/21 18:59:41 Jaydeep Chovatia wrote:
>>>>> > >Jaydeep, do you have any metrics on your clusters comparing them
>>>>> before
>>>>> > and after introducing repair scheduling into the Cassandra process?
>>>>> >
>>>>> > Yes, I had made some comparisons when I started rolling this feature
>>>>> out to
>>>>> > our production five years ago :) Here are the details:
>>>>> > *The Scheduling*
>>>>> > The scheduling itself is exceptionally lightweight, as only one
>>>>> additional
>>>>> > thread monitors the repair activity, updating the status to a system
>>>>> table
>>>>> > once every few minutes or so. So, it does not appear anywhere in the
>>>>> CPU
>>>>> > charts, etc. Unfortunately, I do not have those graphs now, but I
>>>>> can do a
>>>>> > quick comparison if it helps!
>>>>> >
>>>>> > *The Repair Itself*
>>>>> > As we all know, the Cassandra repair algorithm is a heavy-weight
>>>>> process
>>>>> > due to Merkle tree/streaming, etc., no matter how we schedule it.
>>>>> But it is
>>>>> > an orthogonal topic and folks are already discussing creating a new
>>>>> CEP.
>>>>> >
>>>>> > Jaydeep
>>>>> >
>>>>> >
>>>>> > On Mon, Oct 21, 2024 at 10:02 AM Francisco Guerrero <
>>>>> frankgh@apache.org>
>>>>> > wrote:
>>>>> >
>>>>> > > Jaydeep, do you have any metrics on your clusters comparing them
>>>>> before
>>>>> > > and after introducing repair scheduling into the Cassandra process?
>>>>> > >
>>>>> > > On 2024/10/21 16:57:57 "J. D. Jordan" wrote:
>>>>> > > > Sounds good. Just wanted to bring it up. I agree that the
>>>>> scheduling bit
>>>>> > > is
>>>>> > > > pretty light weight and the ideal would be to bring the whole of
>>>>> the
>>>>> > > repair
>>>>> > > > external, which is a much bigger can of worms to open.
>>>>> > > >
>>>>> > > >
>>>>> > > >
>>>>> > > > -Jeremiah
>>>>> > > >
>>>>> > > >
>>>>> > > >
>>>>> > > > > On Oct 21, 2024, at 11:21 AM, Chris Lohfink <
>>>>> clohfink85@gmail.com>
>>>>> > > wrote:
>>>>> > > > >
>>>>> > > > >
>>>>> > > >
>>>>> > > > >
>>>>> > > > >
>>>>> > > > > > I actually think we should be looking at how we can move
>>>>> things out
>>>>> > > of the
>>>>> > > > > database process.
>>>>> > > > >
>>>>> > > > >
>>>>> > > > >
>>>>> > > > >
>>>>> > > > >
>>>>> > > > > While worth pursuing, I think we would need a different CEP
>>>>> just to
>>>>> > > figure
>>>>> > > > > out how to do that. Not only is there a lot of infrastructure
>>>>> > > difficulty in
>>>>> > > > > running multi process, the inter app communication needs to be
>>>>> figured
>>>>> > > out
>>>>> > > > > better then JMX. Even the sidecar we dont have a solid story
>>>>> on how to
>>>>> > > > > ensure both are running or anything yet. It's up to each app
>>>>> owner to
>>>>> > > figure
>>>>> > > > > it out. Once we have a good thing in place I think we can
>>>>> start moving
>>>>> > > > > compactions, repairs, etc out of the database. Even then it's
>>>>> the
>>>>> > > _repairs_
>>>>> > > > > that is expensive, not the scheduling.
>>>>> > > > >
>>>>> > > > >
>>>>> > > > >
>>>>> > > > >
>>>>> > > > > On Mon, Oct 21, 2024 at 9:45 AM Jeremiah Jordan
>>>>> > > > > <[jeremiah.jordan@gmail.com](mailto:jeremiah.jordan@gmail.com
>>>>> )>
>>>>> > > wrote:
>>>>> > > > >
>>>>> > > > >
>>>>> > > >
>>>>> > > > >> I love the idea of a repair service being there by default
>>>>> for an
>>>>> > > install
>>>>> > > > of C*. My main concern here is that it is putting more services
>>>>> into
>>>>> > > the main
>>>>> > > > database process. I actually think we should be looking at how
>>>>> we can
>>>>> > > move
>>>>> > > > things out of the database process. The C* process being a giant
>>>>> > > monolith has
>>>>> > > > always been a pain point. Is there anyway it makes sense for
>>>>> this to be
>>>>> > > an
>>>>> > > > external process rather than a new thread pool inside the C*
>>>>> process?
>>>>> > > >
>>>>> > > > >>
>>>>> > > >
>>>>> > > > >>
>>>>> > > > >
>>>>> > > > >>
>>>>> > > >
>>>>> > > > >> -Jeremiah Jordan
>>>>> > > >
>>>>> > > > >>
>>>>> > > >
>>>>> > > > >>
>>>>> > > > >
>>>>> > > > >>
>>>>> > > >
>>>>> > > > >> On Oct 18, 2024 at 2:58:15 PM, Mick Semb Wever
>>>>> > > > <[mck@apache.org](mailto:mck@apache.org)> wrote:
>>>>> > > > >
>>>>> > > > >>
>>>>> > > >
>>>>> > > > >>>
>>>>> > > > >
>>>>> > > > >>>
>>>>> > > >
>>>>> > > > >>> This is looking strong, thanks Jaydeep.
>>>>> > > >
>>>>> > > > >>>
>>>>> > > >
>>>>> > > > >>>
>>>>> > > > >
>>>>> > > > >>>
>>>>> > > >
>>>>> > > > >>> I would suggest folk take a look at the design doc and the
>>>>> PR in the
>>>>> > > CEP.
>>>>> > > > A lot is there (that I have completely missed).
>>>>> > > >
>>>>> > > > >>>
>>>>> > > >
>>>>> > > > >>>
>>>>> > > > >
>>>>> > > > >>>
>>>>> > > >
>>>>> > > > >>> I would especially ask all authors of prior art (Reaper, DSE
>>>>> > > nodesync,
>>>>> > > > ecchronos) to take a final review of the proposal
>>>>> > > > >
>>>>> > > > >>>
>>>>> > > >
>>>>> > > > >>>
>>>>> > > > >
>>>>> > > > >>>
>>>>> > > >
>>>>> > > > >>> Jaydeep, can we ask for a two week window while we reach out
>>>>> to these
>>>>> > > > people ? There's a lot of prior art in this space, and it feels
>>>>> like
>>>>> > > we're in
>>>>> > > > a good place now where it's clear this has legs and we can use
>>>>> that to
>>>>> > > bring
>>>>> > > > folk in and make sure there's no remaining blindspots.
>>>>> > > >
>>>>> > > > >>>
>>>>> > > >
>>>>> > > > >>>
>>>>> > > > >
>>>>> > > > >>>
>>>>> > > >
>>>>> > > > >>>
>>>>> > > > >
>>>>> > > > >>>
>>>>> > > >
>>>>> > > > >>> On Fri, 18 Oct 2024 at 01:40, Jaydeep Chovatia
>>>>> > > > <[chovatia.jaydeep@gmail.com](mailto:chovatia.jaydeep@gmail.com
>>>>> )>
>>>>> > > wrote:
>>>>> > > > >
>>>>> > > > >>>
>>>>> > > >
>>>>> > > > >>>> Sorry, there is a typo in the CEP-37 link; here is the
>>>>> correct
>>>>> > > > [link](
>>>>> > >
>>>>> https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-37+Apache+Cassandra+Unified+Repair+Solution
>>>>> > > )
>>>>> > > >
>>>>> > > > >>>>
>>>>> > > >
>>>>> > > > >>>>
>>>>> > > > >
>>>>> > > > >>>>
>>>>> > > >
>>>>> > > > >>>>
>>>>> > > > >
>>>>> > > > >>>>
>>>>> > > >
>>>>> > > > >>>> On Thu, Oct 17, 2024 at 4:36 PM Jaydeep Chovatia
>>>>> > > > <[chovatia.jaydeep@gmail.com](mailto:chovatia.jaydeep@gmail.com
>>>>> )>
>>>>> > > wrote:
>>>>> > > > >
>>>>> > > > >>>>
>>>>> > > >
>>>>> > > > >>>>> First, thank you for your patience while we strengthened
>>>>> the
>>>>> > > CEP-37.
>>>>> > > >
>>>>> > > > >>>>>
>>>>> > > >
>>>>> > > > >>>>>
>>>>> > > > >
>>>>> > > > >>>>>
>>>>> > > >
>>>>> > > > >>>>> Over the last eight months, Chris Lohfink, Andy Tolbert,
>>>>> Josh
>>>>> > > McKenzie,
>>>>> > > > Dinesh Joshi, Kristijonas Zalys, and I have done tons of work
>>>>> (online
>>>>> > > > discussions/a dedicated Slack channel
>>>>> > > #cassandra-repair-scheduling-cep37) to
>>>>> > > > come up with the best possible design that not only significantly
>>>>> > > simplifies
>>>>> > > > repair operations but also includes the most common features that
>>>>> > > everyone
>>>>> > > > will benefit from running at Scale.
>>>>> > > >
>>>>> > > > >>>>>
>>>>> > > >
>>>>> > > > >>>>> For example,
>>>>> > > >
>>>>> > > > >>>>>
>>>>> > > >
>>>>> > > > >>>>> * Apache Cassandra must be capable of running multiple
>>>>> repair
>>>>> > > types,
>>>>> > > > such as Full, Incremental, Paxos, and Preview - so the framework
>>>>> should
>>>>> > > be
>>>>> > > > easily extendable with no additional overhead from the
>>>>> operator’s point
>>>>> > > of
>>>>> > > > view.
>>>>> > > >
>>>>> > > > >>>>>
>>>>> > > >
>>>>> > > > >>>>> * An easy way to extend the token-split calculation
>>>>> algorithm
>>>>> > > with a
>>>>> > > > default implementation should exist.
>>>>> > > >
>>>>> > > > >>>>>
>>>>> > > >
>>>>> > > > >>>>> * Running incremental repair reliably at Scale is pretty
>>>>> > > challenging,
>>>>> > > > so we need to place safeguards, such as migration/rollback w/o
>>>>> restart
>>>>> > > and
>>>>> > > > stopping incremental repair automatically if the disk is about
>>>>> to get
>>>>> > > full.
>>>>> > > >
>>>>> > > > >>>>>
>>>>> > > >
>>>>> > > > >>>>>
>>>>> > > >
>>>>> > > > >>>>>
>>>>> > > >
>>>>> > > > >>>>> We are glad to inform you that CEP-37 (i.e., Repair inside
>>>>> > > Cassandra) is
>>>>> > > > now officially ready for review after multiple rounds of design,
>>>>> > > testing, code
>>>>> > > > reviews, documentation reviews, and, more importantly,
>>>>> validation that
>>>>> > > it runs
>>>>> > > > at Scale!
>>>>> > > >
>>>>> > > > >>>>>
>>>>> > > >
>>>>> > > > >>>>>
>>>>> > > > >
>>>>> > > > >>>>>
>>>>> > > >
>>>>> > > > >>>>> Some facts about CEP-37.
>>>>> > > >
>>>>> > > > >>>>>
>>>>> > > >
>>>>> > > > >>>>> * Multiple members have verified all aspects of CEP-37
>>>>> numerous
>>>>> > > times.
>>>>> > > >
>>>>> > > > >>>>>
>>>>> > > >
>>>>> > > > >>>>> * The design proposed in CEP-37 has been thoroughly
>>>>> tried and
>>>>> > > tested
>>>>> > > > on an immense scale (hundreds of unique Cassandra clusters, tens
>>>>> of
>>>>> > > thousands
>>>>> > > > of Cassandra nodes, with tens of millions of QPS) on top of 4.1
>>>>> > > open-source
>>>>> > > > for more than five years; please see more details[
>>>>> > > > here](
>>>>> > >
>>>>> https://www.uber.com/en-US/blog/how-uber-optimized-cassandra-operations-
>>>>> > > > at-scale/).
>>>>> > > >
>>>>> > > > >>>>>
>>>>> > > >
>>>>> > > > >>>>> * The following
>>>>> > > > [presentation](
>>>>> > >
>>>>> https://docs.google.com/presentation/d/1Zilww9c7LihHULk_ckErI2s4XbObxjWknKqRtbvHyZc/edit#slide=id.g30a4fd4fcf7_0_13
>>>>> > > )
>>>>> > > > highlights the rigorous applied to CEP-37, which was given
>>>>> during last
>>>>> > > week’s
>>>>> > > > Apache Cassandra Bay Area [Meetup](
>>>>> > > https://www.meetup.com/apache-cassandra-
>>>>> > > > bay-area/events/303469006/),
>>>>> > > >
>>>>> > > > >>>>>
>>>>> > > >
>>>>> > > > >>>>>
>>>>> > > >
>>>>> > > >
>>>>> > > > >
>>>>> > > > >>>>>
>>>>> > > >
>>>>> > > > >>>>> Since things are massively overhauled, we believe it is
>>>>> almost
>>>>> > > ready for
>>>>> > > > a final pass pre-VOTE. We would like you to please review the
>>>>> > > > [CEP-37](
>>>>> > >
>>>>> https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-37+Apache+Cassandra+Unified+Repair+Solution\
>>>>> <https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-37+Apache+Cassandra+Unified+Repair+Solution%5C>
>>>>> > > <
>>>>> https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-37+Apache+Cassandra+Unified+Repair+Solution%5C
>>>>> >
>>>>> > > ))
>>>>> > > > and the associated detailed design
>>>>> > > > [doc](https://docs.google.com/document/d/1CJWxjEi-
>>>>> > > > mBABPMZ3VWJ9w5KavWfJETAGxfUpsViPcPo/edit#heading=h.r112r46toau0).
>>>>> > > >
>>>>> > > > >>>>>
>>>>> > > >
>>>>> > > > >>>>>
>>>>> > > > >
>>>>> > > > >>>>>
>>>>> > > >
>>>>> > > > >>>>> Thank you everyone!
>>>>> > > >
>>>>> > > > >>>>>
>>>>> > > >
>>>>> > > > >>>>> Chris, Andy, Josh, Dinesh, Kristijonas, and Jaydeep
>>>>> > > >
>>>>> > > > >>>>>
>>>>> > > >
>>>>> > > > >>>>>
>>>>> > > > >
>>>>> > > > >
>>>>> > > > >>>>>
>>>>> > > >
>>>>> > > > >>>>>
>>>>> > > > >
>>>>> > > > >>>>>
>>>>> > > >
>>>>> > > > >>>>> On Thu, Sep 19, 2024 at 11:26 AM Josh McKenzie
>>>>> > > > <[jmckenzie@apache.org](mailto:jmckenzie@apache.org)> wrote:
>>>>> > > > >
>>>>> > > > >>>>>
>>>>> > > >
>>>>> > > > >>>>>> __
>>>>> > > >
>>>>> > > > >>>>>>
>>>>> > > >
>>>>> > > > >>>>>> Not quite; finishing touches on the CEP and design doc
>>>>> are in
>>>>> > > flight
>>>>> > > > (as of last / this week).
>>>>> > > > >
>>>>> > > > >>>>>>
>>>>> > > >
>>>>> > > > >>>>>>
>>>>> > > > >
>>>>> > > > >>>>>>
>>>>> > > >
>>>>> > > > >>>>>> Soon(tm).
>>>>> > > >
>>>>> > > > >>>>>>
>>>>> > > >
>>>>> > > > >>>>>>
>>>>> > > > >
>>>>> > > > >>>>>>
>>>>> > > >
>>>>> > > > >>>>>> On Thu, Sep 19, 2024, at 2:07 PM, Patrick McFadin wrote:
>>>>> > > > >
>>>>> > > > >>>>>>
>>>>> > > >
>>>>> > > > >>>>>>> Is this CEP ready for a VOTE thread?
>>>>> > > > <
>>>>> > >
>>>>> https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-37+%28DRAFT%29+Apache+Cassandra+Unified+Repair+Solution
>>>>> >
>>>>> > >
>>>>> > > > >
>>>>> > > > >>>>>>>
>>>>> > > >
>>>>> > > > >>>>>>>
>>>>> > > > >
>>>>> > > > >>>>>>>
>>>>> > > >
>>>>> > > > >>>>>>> On Sun, Feb 25, 2024 at 12:25 PM Jaydeep Chovatia
>>>>> > > > <[chovatia.jaydeep@gmail.com](mailto:chovatia.jaydeep@gmail.com
>>>>> )>
>>>>> > > wrote:
>>>>> > > > >
>>>>> > > > >>>>>>>
>>>>> > > >
>>>>> > > > >>>>>>>> Thanks, Josh. I've just updated the
>>>>> > > > [CEP](
>>>>> > >
>>>>> https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-37+%28DRAFT%29+Apache+Cassandra+Official+Repair+Solution
>>>>> > > )
>>>>> > > > and included all the solutions you mentioned below.
>>>>> > > > >
>>>>> > > > >>>>>>>>
>>>>> > > >
>>>>> > > > >>>>>>>>
>>>>> > > > >
>>>>> > > > >>>>>>>>
>>>>> > > >
>>>>> > > > >>>>>>>> Jaydeep
>>>>> > > > >
>>>>> > > > >>>>>>>>
>>>>> > > >
>>>>> > > > >>>>>>>>
>>>>> > > > >
>>>>> > > > >>>>>>>>
>>>>> > > >
>>>>> > > > >>>>>>>> On Thu, Feb 22, 2024 at 9:33 AM Josh McKenzie
>>>>> > > > <[jmckenzie@apache.org](mailto:jmckenzie@apache.org)> wrote:
>>>>> > > > >
>>>>> > > > >>>>>>>>
>>>>> > > >
>>>>> > > > >>>>>>>>> __
>>>>> > > > >
>>>>> > > > >>>>>>>>>
>>>>> > > >
>>>>> > > > >>>>>>>>> Very late response from me here (basically necro'ing
>>>>> this
>>>>> > > thread).
>>>>> > > > >
>>>>> > > > >>>>>>>>>
>>>>> > > >
>>>>> > > > >>>>>>>>>
>>>>> > > > >
>>>>> > > > >>>>>>>>>
>>>>> > > >
>>>>> > > > >>>>>>>>> I think it'd be useful to get this condensed into a
>>>>> CEP that
>>>>> > > we can
>>>>> > > > then discuss in that format. It's clearly something we all agree
>>>>> we need
>>>>> > > and
>>>>> > > > having an implementation that works, even if it's not in your
>>>>> preferred
>>>>> > > > execution domain, is vastly better than nothing IMO.
>>>>> > > > >
>>>>> > > > >>>>>>>>>
>>>>> > > >
>>>>> > > > >>>>>>>>>
>>>>> > > > >
>>>>> > > > >>>>>>>>>
>>>>> > > >
>>>>> > > > >>>>>>>>> I don't have cycles (nor background ;) ) to do that,
>>>>> but it
>>>>> > > sounds
>>>>> > > > like you do Jaydeep given the implementation you have on a
>>>>> private fork +
>>>>> > > > design.
>>>>> > > > >
>>>>> > > > >>>>>>>>>
>>>>> > > >
>>>>> > > > >>>>>>>>>
>>>>> > > > >
>>>>> > > > >>>>>>>>>
>>>>> > > >
>>>>> > > > >>>>>>>>> A non-exhaustive list of things that might be useful
>>>>> > > incorporating
>>>>> > > > into or referencing from a CEP:
>>>>> > > > >
>>>>> > > > >>>>>>>>>
>>>>> > > >
>>>>> > > > >>>>>>>>> Slack thread: <https://the-
>>>>> > > > asf.slack.com/archives/CK23JSY2K/p1690225062383619>
>>>>> > > > >
>>>>> > > > >>>>>>>>>
>>>>> > > >
>>>>> > > > >>>>>>>>> Joey's old C* ticket:
>>>>> > > > <https://issues.apache.org/jira/browse/CASSANDRA-14346>
>>>>> > > > >
>>>>> > > > >>>>>>>>>
>>>>> > > >
>>>>> > > > >>>>>>>>> Even older automatic repair scheduling:
>>>>> > > > <https://issues.apache.org/jira/browse/CASSANDRA-10070>
>>>>> > > > >
>>>>> > > > >>>>>>>>>
>>>>> > > >
>>>>> > > > >>>>>>>>> Your design gdoc: <
>>>>> > > https://docs.google.com/document/d/1CJWxjEi-
>>>>> > > > mBABPMZ3VWJ9w5KavWfJETAGxfUpsViPcPo/edit#heading=h.r112r46toau0>
>>>>> > > > >
>>>>> > > > >>>>>>>>>
>>>>> > > >
>>>>> > > > >>>>>>>>> PR with automated repair:
>>>>> > > > <
>>>>> > >
>>>>> https://github.com/jaydeepkumar1984/cassandra/commit/ef6456d652c0d07cf29d88dfea03b73704814c2c
>>>>> >
>>>>> > >
>>>>> > > > >
>>>>> > > > >>>>>>>>>
>>>>> > > >
>>>>> > > > >>>>>>>>>
>>>>> > > > >
>>>>> > > > >>>>>>>>>
>>>>> > > >
>>>>> > > > >>>>>>>>> My intuition is that we're all basically in agreement
>>>>> that
>>>>> > > this is
>>>>> > > > something the DB needs, we're all willing to bikeshed for our
>>>>> personal
>>>>> > > > preference on where it lives and how it's implemented, and at
>>>>> the end of
>>>>> > > the
>>>>> > > > day, code talks. I don't think anyone's said they'll die on the
>>>>> hill of
>>>>> > > > implementation details, so that feels like CEP time to me.
>>>>> > > > >
>>>>> > > > >>>>>>>>>
>>>>> > > >
>>>>> > > > >>>>>>>>>
>>>>> > > > >
>>>>> > > > >>>>>>>>>
>>>>> > > >
>>>>> > > > >>>>>>>>> If you were willing and able to get a CEP together for
>>>>> > > automated
>>>>> > > > repair based on the above material, given you've done the work
>>>>> and have
>>>>> > > the
>>>>> > > > proof points it's working at scale, I think this would be a
>>>>> _huge
>>>>> > > > contribution_ to the community.
>>>>> > > > >
>>>>> > > > >>>>>>>>>
>>>>> > > >
>>>>> > > > >>>>>>>>>
>>>>> > > > >
>>>>> > > > >>>>>>>>>
>>>>> > > >
>>>>> > > > >>>>>>>>> On Thu, Aug 24, 2023, at 7:26 PM, Jaydeep Chovatia
>>>>> wrote:
>>>>> > > > >
>>>>> > > > >>>>>>>>>
>>>>> > > >
>>>>> > > > >>>>>>>>>> Is anyone going to file an official CEP for this?
>>>>> > > > >
>>>>> > > > >>>>>>>>>>
>>>>> > > >
>>>>> > > > >>>>>>>>>> As mentioned in this email thread, here is one of the
>>>>> > > solution's
>>>>> > > > [design doc](https://docs.google.com/document/d/1CJWxjEi-
>>>>> > > > mBABPMZ3VWJ9w5KavWfJETAGxfUpsViPcPo/edit#heading=h.r112r46toau0)
>>>>> and
>>>>> > > source
>>>>> > > > code on a private Apache Cassandra patch. Could you go through
>>>>> it and
>>>>> > > let me
>>>>> > > > know what you think?
>>>>> > > > >
>>>>> > > > >>>>>>>>>>
>>>>> > > >
>>>>> > > > >>>>>>>>>>
>>>>> > > > >
>>>>> > > > >>>>>>>>>>
>>>>> > > >
>>>>> > > > >>>>>>>>>> Jaydeep
>>>>> > > > >
>>>>> > > > >>>>>>>>>>
>>>>> > > >
>>>>> > > > >>>>>>>>>>
>>>>> > > > >
>>>>> > > > >>>>>>>>>>
>>>>> > > >
>>>>> > > > >>>>>>>>>> On Wed, Aug 2, 2023 at 3:54 PM Jon Haddad
>>>>> > > > <[rustyrazorblade@apache.org](mailto:rustyrazorblade@apache.org
>>>>> )>
>>>>> > > wrote:
>>>>> > > > >
>>>>> > > > >>>>>>>>>>
>>>>> > > >
>>>>> > > > >>>>>>>>>>> > That said I would happily support an effort to
>>>>> bring repair
>>>>> > > > scheduling to the sidecar immediately. This has nothing blocking
>>>>> it, and
>>>>> > > would
>>>>> > > > potentially enable the sidecar to provide an official repair
>>>>> scheduling
>>>>> > > > solution that is compatible with current or even previous
>>>>> versions of the
>>>>> > > > database.
>>>>> > > > >
>>>>> > > > >>>>>>>>>>>
>>>>> > > >
>>>>> > > > >>>>>>>>>>>
>>>>> > > > >
>>>>> > > > >>>>>>>>>>>
>>>>> > > >
>>>>> > > > >>>>>>>>>>> This is something I hadn't thought much about, and
>>>>> is a
>>>>> > > pretty
>>>>> > > > good argument for using the sidecar initially. There's a lot of
>>>>> > > deployments
>>>>> > > > out there and having an official repair option would be a big
>>>>> win.
>>>>> > > > >
>>>>> > > > >>>>>>>>>>>
>>>>> > > >
>>>>> > > > >>>>>>>>>>>
>>>>> > > > >
>>>>> > > > >>>>>>>>>>>
>>>>> > > >
>>>>> > > > >>>>>>>>>>>
>>>>> > > > >
>>>>> > > > >>>>>>>>>>>
>>>>> > > >
>>>>> > > > >>>>>>>>>>> On 2023/07/26 23:20:07 "C. Scott Andreas" wrote:
>>>>> > > > >
>>>>> > > > >>>>>>>>>>>
>>>>> > > >
>>>>> > > > >>>>>>>>>>> > I agree that it would be ideal for Cassandra to
>>>>> have a
>>>>> > > repair
>>>>> > > > scheduler in-DB.
>>>>> > > > >
>>>>> > > > >>>>>>>>>>>
>>>>> > > >
>>>>> > > > >>>>>>>>>>> >
>>>>> > > > >
>>>>> > > > >>>>>>>>>>>
>>>>> > > >
>>>>> > > > >>>>>>>>>>> > That said I would happily support an effort to
>>>>> bring repair
>>>>> > > > scheduling to the sidecar immediately. This has nothing blocking
>>>>> it, and
>>>>> > > would
>>>>> > > > potentially enable the sidecar to provide an official repair
>>>>> scheduling
>>>>> > > > solution that is compatible with current or even previous
>>>>> versions of the
>>>>> > > > database.
>>>>> > > > >
>>>>> > > > >>>>>>>>>>>
>>>>> > > >
>>>>> > > > >>>>>>>>>>> >
>>>>> > > > >
>>>>> > > > >>>>>>>>>>>
>>>>> > > >
>>>>> > > > >>>>>>>>>>> > Once TCM has landed, we’ll have much stronger
>>>>> primitives
>>>>> > > for
>>>>> > > > repair orchestration in the database itself. But I don’t think
>>>>> that
>>>>> > > should
>>>>> > > > block progress on a repair scheduling solution in the sidecar,
>>>>> and there
>>>>> > > is
>>>>> > > > nothing that would prevent someone from continuing to use a
>>>>> sidecar-based
>>>>> > > > solution in perpetuity if they preferred.
>>>>> > > > >
>>>>> > > > >>>>>>>>>>>
>>>>> > > >
>>>>> > > > >>>>>>>>>>> >
>>>>> > > > >
>>>>> > > > >>>>>>>>>>>
>>>>> > > >
>>>>> > > > >>>>>>>>>>> > \- Scott
>>>>> > > > >
>>>>> > > > >>>>>>>>>>>
>>>>> > > >
>>>>> > > > >>>>>>>>>>> >
>>>>> > > > >
>>>>> > > > >>>>>>>>>>>
>>>>> > > >
>>>>> > > > >>>>>>>>>>> > > On Jul 26, 2023, at 3:25 PM, Jon Haddad
>>>>> > > > <[rustyrazorblade@apache.org](mailto:rustyrazorblade@apache.org
>>>>> )>
>>>>> > > wrote:
>>>>> > > > >
>>>>> > > > >>>>>>>>>>>
>>>>> > > >
>>>>> > > > >>>>>>>>>>> > >
>>>>> > > > >
>>>>> > > > >>>>>>>>>>>
>>>>> > > >
>>>>> > > > >>>>>>>>>>> > > I'm 100% in favor of repair being part of the
>>>>> core DB,
>>>>> > > not
>>>>> > > > the sidecar. The current (and past) state of things where
>>>>> running the DB
>>>>> > > > correctly *requires* running a separate process (either community
>>>>> > > maintained
>>>>> > > > or official C* sidecar) is incredibly painful for folks. The
>>>>> idea that
>>>>> > > your
>>>>> > > > data integrity needs to be opt-in has never made sense to me
>>>>> from the
>>>>> > > > perspective of either the product or the end user.
>>>>> > > > >
>>>>> > > > >>>>>>>>>>>
>>>>> > > >
>>>>> > > > >>>>>>>>>>> > >
>>>>> > > > >
>>>>> > > > >>>>>>>>>>>
>>>>> > > >
>>>>> > > > >>>>>>>>>>> > > I've worked with way too many teams that have
>>>>> either
>>>>> > > > configured this incorrectly or not at all.
>>>>> > > > >
>>>>> > > > >>>>>>>>>>>
>>>>> > > >
>>>>> > > > >>>>>>>>>>> > >
>>>>> > > > >
>>>>> > > > >>>>>>>>>>>
>>>>> > > >
>>>>> > > > >>>>>>>>>>> > > Ideally Cassandra would ship with repair built
>>>>> in and on
>>>>> > > by
>>>>> > > > default. Power users can disable if they want to continue to
>>>>> maintain
>>>>> > > their
>>>>> > > > own repair tooling for some reason.
>>>>> > > > >
>>>>> > > > >>>>>>>>>>>
>>>>> > > >
>>>>> > > > >>>>>>>>>>> > >
>>>>> > > > >
>>>>> > > > >>>>>>>>>>>
>>>>> > > >
>>>>> > > > >>>>>>>>>>> > > Jon
>>>>> > > > >
>>>>> > > > >>>>>>>>>>>
>>>>> > > >
>>>>> > > > >>>>>>>>>>> > >
>>>>> > > > >
>>>>> > > > >>>>>>>>>>>
>>>>> > > >
>>>>> > > > >>>>>>>>>>> > >> On 2023/07/24 20:44:14 German Eichberger via dev
>>>>> > > wrote:
>>>>> > > > >
>>>>> > > > >>>>>>>>>>>
>>>>> > > >
>>>>> > > > >>>>>>>>>>> > >> All,
>>>>> > > > >
>>>>> > > > >>>>>>>>>>>
>>>>> > > >
>>>>> > > > >>>>>>>>>>> > >> We had a brief discussion in [2] about the Uber
>>>>> article
>>>>> > > [1]
>>>>> > > > where they talk about having integrated repair into Cassandra
>>>>> and how
>>>>> > > great
>>>>> > > > that is. I expressed my disappointment that they didn't work
>>>>> with the
>>>>> > > > community on that (Uber, if you are listening time to make
>>>>> amends 🙂)
>>>>> > > and it
>>>>> > > > turns out Joey already had the idea and wrote the code [3] - so
>>>>> I wanted
>>>>> > > to
>>>>> > > > start a discussion to gauge interest and maybe how to revive that
>>>>> > > effort.
>>>>> > > > >
>>>>> > > > >>>>>>>>>>>
>>>>> > > >
>>>>> > > > >>>>>>>>>>> > >> Thanks,
>>>>> > > > >
>>>>> > > > >>>>>>>>>>>
>>>>> > > >
>>>>> > > > >>>>>>>>>>> > >> German
>>>>> > > > >
>>>>> > > > >>>>>>>>>>>
>>>>> > > >
>>>>> > > > >>>>>>>>>>> > >> [1] <
>>>>> > > https://www.uber.com/blog/how-uber-optimized-cassandra-
>>>>> > > > operations-at-scale/>
>>>>> > > > >
>>>>> > > > >>>>>>>>>>>
>>>>> > > >
>>>>> > > > >>>>>>>>>>> > >> [2] <https://the-
>>>>> > > > asf.slack.com/archives/CK23JSY2K/p1690225062383619>
>>>>> > > > >
>>>>> > > > >>>>>>>>>>>
>>>>> > > >
>>>>> > > > >>>>>>>>>>> > >> [3] <
>>>>> > > https://issues.apache.org/jira/browse/CASSANDRA-14346>
>>>>> > > > >
>>>>> > > > >>>>>>>>>>>
>>>>> > > >
>>>>> > > > >>>>>>>>>>> >
>>>>> > > > >
>>>>> > > > >>>>>>>>>
>>>>> > > >
>>>>> > > > >>>>>>>>>
>>>>> > > > >
>>>>> > > > >>>>>>
>>>>> > > >
>>>>> > > > >>>>>>
>>>>> > > > >
>>>>> > > >
>>>>> > > >
>>>>> > >
>>>>> >
>>>>>
>>>>
Re: [Discuss] Repair inside C*
Posted by Alexander DEJANOVSKI <ad...@gmail.com>.
Hi Jaydeep,
I've taken a look at the proposed design and have a few comments/questions.
As one of the maintainers of Reaper, I'm looking this through the lens of
how Reaper does things.
*The approach taken in the CEP-37 design is "node-centric" vs a "range
centric" approach (which is the one Reaper takes).*I'm worried that this
will not allow spreading the repair load evenly across the cluster, since
nodes are the concurrency unit. You could allow running repair on 3 nodes
concurrently for example, but these 3 nodes could all involve the same
replicas, making these replicas process 3 concurrent repairs while others
could be left uninvolved in any repair at all.
Taking a range centric approach (we're not repairing nodes, we're repairing
the token ranges) allows to spread the load evenly without overlap in the
replica sets.
I'm more worried even with incremental repair here, because you might end
up with some conflicts around sstables which would be in the pending repair
pool but would be needed by a competing repair job.
I don't know if in the latest versions such sstables would be totally
ignored or if the competing repair job would fail.
*Each repair command will repair all keyspaces (with the ability to fully
exclude some tables) and **I haven't seen a notion of schedule which seems
to suggest repairs are running continuously (unless I missed something?).*
There are many cases where one might have differentiated gc_grace_seconds
settings to optimize reclaiming tombstones when applicable. That requires
having some fine control over the repair cycle for a given keyspace/set of
tables.
Here, nodes will be processed sequentially and each node will process the
keyspaces sequentially, tying the repair cycle of all keyspaces together.
If one of the ranges for a specific keyspace cannot be repaired within the
3 hours timeout, it could block all the other keyspaces repairs.
Continuous repair might create a lot of overhead for full repairs which
often don't require more than 1 run per week.
It also will not allow running a mix of scheduled full/incremental repairs
(I'm unsure if that is still a recommendation, but it was still recommended
not so long ago)
*The timeout base duration is large*
I think the 3 hours timeout might be quite large and probably means a lot
of data is being repaired for each split. That usually involves some level
of overstreaming. I don't have numbers to support this, it's more about my
own experience on sizing splits in production with Reaper to reduce the
impact as much as possible on cluster performance.
We use 30 minutes as default in Reaper with subsequent attempts growing the
timeout dynamically for challenging splits.
Finally thanks for picking this up, I'm eager to see Reaper not being
needed anymore and having the database manage its own repairs!
Le mar. 22 oct. 2024 à 21:10, Benedict <be...@apache.org> a écrit :
> I realise it’s out of scope, but to counterbalance all of the
> pro-decomposition messages I wanted to chime in with a strong -1. But we
> can debate that in a suitable context later.
>
> On 22 Oct 2024, at 16:36, Jordan West <jw...@apache.org> wrote:
>
>
> Agreed with the sentiment that decomposition is a good target but out of
> scope here. I’m personally excited to see an in-tree repair scheduler and
> am supportive of the approach shared here.
>
> Jordan
>
> On Tue, Oct 22, 2024 at 08:12 Dinesh Joshi <dj...@apache.org> wrote:
>
>> Decomposing Cassandra may be architecturally desirable but that is not
>> the goal of this CEP. This CEP brings value to operators today so it should
>> be considered on that merit. We definitely need to have a separate
>> conversation on Cassandra's architectural direction.
>>
>> On Tue, Oct 22, 2024 at 7:51 AM Joseph Lynch <jo...@gmail.com>
>> wrote:
>>
>>> Definitely like this in C* itself. We only changed our proposal to
>>> putting repair scheduling in the sidecar before because trunk was frozen
>>> for the foreseeable future at that time. With trunk unfrozen and
>>> development on the main process going at a fast pace I think it makes way
>>> more sense to integrate natively as table properties as this CEP proposes.
>>> Completely agree the scheduling overhead should be minimal.
>>>
>>> Moving the actual repair operation (comparing data and streaming
>>> mismatches) along with compaction operations to a separate process long
>>> term makes a lot of sense but imo only once we both have a release of
>>> sidecar and a contract figured out between them on communication. I'm
>>> watching CEP-38 there as I think CQL and virtual tables are looking much
>>> stronger than when we wrote CEP-1 and chose HTTP but that's for that
>>> discussion and not this one.
>>>
>>> -Joey
>>>
>>> On Mon, Oct 21, 2024 at 3:25 PM Francisco Guerrero <fr...@apache.org>
>>> wrote:
>>>
>>>> Like others have said, I was expecting the scheduling portion of repair
>>>> is
>>>> negligible. I was mostly curious if you had something handy that you can
>>>> quickly share.
>>>>
>>>> On 2024/10/21 18:59:41 Jaydeep Chovatia wrote:
>>>> > >Jaydeep, do you have any metrics on your clusters comparing them
>>>> before
>>>> > and after introducing repair scheduling into the Cassandra process?
>>>> >
>>>> > Yes, I had made some comparisons when I started rolling this feature
>>>> out to
>>>> > our production five years ago :) Here are the details:
>>>> > *The Scheduling*
>>>> > The scheduling itself is exceptionally lightweight, as only one
>>>> additional
>>>> > thread monitors the repair activity, updating the status to a system
>>>> table
>>>> > once every few minutes or so. So, it does not appear anywhere in the
>>>> CPU
>>>> > charts, etc. Unfortunately, I do not have those graphs now, but I can
>>>> do a
>>>> > quick comparison if it helps!
>>>> >
>>>> > *The Repair Itself*
>>>> > As we all know, the Cassandra repair algorithm is a heavy-weight
>>>> process
>>>> > due to Merkle tree/streaming, etc., no matter how we schedule it. But
>>>> it is
>>>> > an orthogonal topic and folks are already discussing creating a new
>>>> CEP.
>>>> >
>>>> > Jaydeep
>>>> >
>>>> >
>>>> > On Mon, Oct 21, 2024 at 10:02 AM Francisco Guerrero <
>>>> frankgh@apache.org>
>>>> > wrote:
>>>> >
>>>> > > Jaydeep, do you have any metrics on your clusters comparing them
>>>> before
>>>> > > and after introducing repair scheduling into the Cassandra process?
>>>> > >
>>>> > > On 2024/10/21 16:57:57 "J. D. Jordan" wrote:
>>>> > > > Sounds good. Just wanted to bring it up. I agree that the
>>>> scheduling bit
>>>> > > is
>>>> > > > pretty light weight and the ideal would be to bring the whole of
>>>> the
>>>> > > repair
>>>> > > > external, which is a much bigger can of worms to open.
>>>> > > >
>>>> > > >
>>>> > > >
>>>> > > > -Jeremiah
>>>> > > >
>>>> > > >
>>>> > > >
>>>> > > > > On Oct 21, 2024, at 11:21 AM, Chris Lohfink <
>>>> clohfink85@gmail.com>
>>>> > > wrote:
>>>> > > > >
>>>> > > > >
>>>> > > >
>>>> > > > >
>>>> > > > >
>>>> > > > > > I actually think we should be looking at how we can move
>>>> things out
>>>> > > of the
>>>> > > > > database process.
>>>> > > > >
>>>> > > > >
>>>> > > > >
>>>> > > > >
>>>> > > > >
>>>> > > > > While worth pursuing, I think we would need a different CEP
>>>> just to
>>>> > > figure
>>>> > > > > out how to do that. Not only is there a lot of infrastructure
>>>> > > difficulty in
>>>> > > > > running multi process, the inter app communication needs to be
>>>> figured
>>>> > > out
>>>> > > > > better then JMX. Even the sidecar we dont have a solid story on
>>>> how to
>>>> > > > > ensure both are running or anything yet. It's up to each app
>>>> owner to
>>>> > > figure
>>>> > > > > it out. Once we have a good thing in place I think we can start
>>>> moving
>>>> > > > > compactions, repairs, etc out of the database. Even then it's
>>>> the
>>>> > > _repairs_
>>>> > > > > that is expensive, not the scheduling.
>>>> > > > >
>>>> > > > >
>>>> > > > >
>>>> > > > >
>>>> > > > > On Mon, Oct 21, 2024 at 9:45 AM Jeremiah Jordan
>>>> > > > > <[jeremiah.jordan@gmail.com](mailto:jeremiah.jordan@gmail.com)>
>>>> > > wrote:
>>>> > > > >
>>>> > > > >
>>>> > > >
>>>> > > > >> I love the idea of a repair service being there by default for
>>>> an
>>>> > > install
>>>> > > > of C*. My main concern here is that it is putting more services
>>>> into
>>>> > > the main
>>>> > > > database process. I actually think we should be looking at how
>>>> we can
>>>> > > move
>>>> > > > things out of the database process. The C* process being a giant
>>>> > > monolith has
>>>> > > > always been a pain point. Is there anyway it makes sense for
>>>> this to be
>>>> > > an
>>>> > > > external process rather than a new thread pool inside the C*
>>>> process?
>>>> > > >
>>>> > > > >>
>>>> > > >
>>>> > > > >>
>>>> > > > >
>>>> > > > >>
>>>> > > >
>>>> > > > >> -Jeremiah Jordan
>>>> > > >
>>>> > > > >>
>>>> > > >
>>>> > > > >>
>>>> > > > >
>>>> > > > >>
>>>> > > >
>>>> > > > >> On Oct 18, 2024 at 2:58:15 PM, Mick Semb Wever
>>>> > > > <[mck@apache.org](mailto:mck@apache.org)> wrote:
>>>> > > > >
>>>> > > > >>
>>>> > > >
>>>> > > > >>>
>>>> > > > >
>>>> > > > >>>
>>>> > > >
>>>> > > > >>> This is looking strong, thanks Jaydeep.
>>>> > > >
>>>> > > > >>>
>>>> > > >
>>>> > > > >>>
>>>> > > > >
>>>> > > > >>>
>>>> > > >
>>>> > > > >>> I would suggest folk take a look at the design doc and the PR
>>>> in the
>>>> > > CEP.
>>>> > > > A lot is there (that I have completely missed).
>>>> > > >
>>>> > > > >>>
>>>> > > >
>>>> > > > >>>
>>>> > > > >
>>>> > > > >>>
>>>> > > >
>>>> > > > >>> I would especially ask all authors of prior art (Reaper, DSE
>>>> > > nodesync,
>>>> > > > ecchronos) to take a final review of the proposal
>>>> > > > >
>>>> > > > >>>
>>>> > > >
>>>> > > > >>>
>>>> > > > >
>>>> > > > >>>
>>>> > > >
>>>> > > > >>> Jaydeep, can we ask for a two week window while we reach out
>>>> to these
>>>> > > > people ? There's a lot of prior art in this space, and it feels
>>>> like
>>>> > > we're in
>>>> > > > a good place now where it's clear this has legs and we can use
>>>> that to
>>>> > > bring
>>>> > > > folk in and make sure there's no remaining blindspots.
>>>> > > >
>>>> > > > >>>
>>>> > > >
>>>> > > > >>>
>>>> > > > >
>>>> > > > >>>
>>>> > > >
>>>> > > > >>>
>>>> > > > >
>>>> > > > >>>
>>>> > > >
>>>> > > > >>> On Fri, 18 Oct 2024 at 01:40, Jaydeep Chovatia
>>>> > > > <[chovatia.jaydeep@gmail.com](mailto:chovatia.jaydeep@gmail.com)>
>>>> > > wrote:
>>>> > > > >
>>>> > > > >>>
>>>> > > >
>>>> > > > >>>> Sorry, there is a typo in the CEP-37 link; here is the
>>>> correct
>>>> > > > [link](
>>>> > >
>>>> https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-37+Apache+Cassandra+Unified+Repair+Solution
>>>> > > )
>>>> > > >
>>>> > > > >>>>
>>>> > > >
>>>> > > > >>>>
>>>> > > > >
>>>> > > > >>>>
>>>> > > >
>>>> > > > >>>>
>>>> > > > >
>>>> > > > >>>>
>>>> > > >
>>>> > > > >>>> On Thu, Oct 17, 2024 at 4:36 PM Jaydeep Chovatia
>>>> > > > <[chovatia.jaydeep@gmail.com](mailto:chovatia.jaydeep@gmail.com)>
>>>> > > wrote:
>>>> > > > >
>>>> > > > >>>>
>>>> > > >
>>>> > > > >>>>> First, thank you for your patience while we strengthened the
>>>> > > CEP-37.
>>>> > > >
>>>> > > > >>>>>
>>>> > > >
>>>> > > > >>>>>
>>>> > > > >
>>>> > > > >>>>>
>>>> > > >
>>>> > > > >>>>> Over the last eight months, Chris Lohfink, Andy Tolbert,
>>>> Josh
>>>> > > McKenzie,
>>>> > > > Dinesh Joshi, Kristijonas Zalys, and I have done tons of work
>>>> (online
>>>> > > > discussions/a dedicated Slack channel
>>>> > > #cassandra-repair-scheduling-cep37) to
>>>> > > > come up with the best possible design that not only significantly
>>>> > > simplifies
>>>> > > > repair operations but also includes the most common features that
>>>> > > everyone
>>>> > > > will benefit from running at Scale.
>>>> > > >
>>>> > > > >>>>>
>>>> > > >
>>>> > > > >>>>> For example,
>>>> > > >
>>>> > > > >>>>>
>>>> > > >
>>>> > > > >>>>> * Apache Cassandra must be capable of running multiple
>>>> repair
>>>> > > types,
>>>> > > > such as Full, Incremental, Paxos, and Preview - so the framework
>>>> should
>>>> > > be
>>>> > > > easily extendable with no additional overhead from the operator’s
>>>> point
>>>> > > of
>>>> > > > view.
>>>> > > >
>>>> > > > >>>>>
>>>> > > >
>>>> > > > >>>>> * An easy way to extend the token-split calculation
>>>> algorithm
>>>> > > with a
>>>> > > > default implementation should exist.
>>>> > > >
>>>> > > > >>>>>
>>>> > > >
>>>> > > > >>>>> * Running incremental repair reliably at Scale is pretty
>>>> > > challenging,
>>>> > > > so we need to place safeguards, such as migration/rollback w/o
>>>> restart
>>>> > > and
>>>> > > > stopping incremental repair automatically if the disk is about to
>>>> get
>>>> > > full.
>>>> > > >
>>>> > > > >>>>>
>>>> > > >
>>>> > > > >>>>>
>>>> > > >
>>>> > > > >>>>>
>>>> > > >
>>>> > > > >>>>> We are glad to inform you that CEP-37 (i.e., Repair inside
>>>> > > Cassandra) is
>>>> > > > now officially ready for review after multiple rounds of design,
>>>> > > testing, code
>>>> > > > reviews, documentation reviews, and, more importantly, validation
>>>> that
>>>> > > it runs
>>>> > > > at Scale!
>>>> > > >
>>>> > > > >>>>>
>>>> > > >
>>>> > > > >>>>>
>>>> > > > >
>>>> > > > >>>>>
>>>> > > >
>>>> > > > >>>>> Some facts about CEP-37.
>>>> > > >
>>>> > > > >>>>>
>>>> > > >
>>>> > > > >>>>> * Multiple members have verified all aspects of CEP-37
>>>> numerous
>>>> > > times.
>>>> > > >
>>>> > > > >>>>>
>>>> > > >
>>>> > > > >>>>> * The design proposed in CEP-37 has been thoroughly tried
>>>> and
>>>> > > tested
>>>> > > > on an immense scale (hundreds of unique Cassandra clusters, tens
>>>> of
>>>> > > thousands
>>>> > > > of Cassandra nodes, with tens of millions of QPS) on top of 4.1
>>>> > > open-source
>>>> > > > for more than five years; please see more details[
>>>> > > > here](
>>>> > >
>>>> https://www.uber.com/en-US/blog/how-uber-optimized-cassandra-operations-
>>>> > > > at-scale/).
>>>> > > >
>>>> > > > >>>>>
>>>> > > >
>>>> > > > >>>>> * The following
>>>> > > > [presentation](
>>>> > >
>>>> https://docs.google.com/presentation/d/1Zilww9c7LihHULk_ckErI2s4XbObxjWknKqRtbvHyZc/edit#slide=id.g30a4fd4fcf7_0_13
>>>> > > )
>>>> > > > highlights the rigorous applied to CEP-37, which was given during
>>>> last
>>>> > > week’s
>>>> > > > Apache Cassandra Bay Area [Meetup](
>>>> > > https://www.meetup.com/apache-cassandra-
>>>> > > > bay-area/events/303469006/),
>>>> > > >
>>>> > > > >>>>>
>>>> > > >
>>>> > > > >>>>>
>>>> > > >
>>>> > > >
>>>> > > > >
>>>> > > > >>>>>
>>>> > > >
>>>> > > > >>>>> Since things are massively overhauled, we believe it is
>>>> almost
>>>> > > ready for
>>>> > > > a final pass pre-VOTE. We would like you to please review the
>>>> > > > [CEP-37](
>>>> > >
>>>> https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-37+Apache+Cassandra+Unified+Repair+Solution\
>>>> <https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-37+Apache+Cassandra+Unified+Repair+Solution%5C>
>>>> > > <
>>>> https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-37+Apache+Cassandra+Unified+Repair+Solution%5C
>>>> >
>>>> > > ))
>>>> > > > and the associated detailed design
>>>> > > > [doc](https://docs.google.com/document/d/1CJWxjEi-
>>>> > > > mBABPMZ3VWJ9w5KavWfJETAGxfUpsViPcPo/edit#heading=h.r112r46toau0).
>>>> > > >
>>>> > > > >>>>>
>>>> > > >
>>>> > > > >>>>>
>>>> > > > >
>>>> > > > >>>>>
>>>> > > >
>>>> > > > >>>>> Thank you everyone!
>>>> > > >
>>>> > > > >>>>>
>>>> > > >
>>>> > > > >>>>> Chris, Andy, Josh, Dinesh, Kristijonas, and Jaydeep
>>>> > > >
>>>> > > > >>>>>
>>>> > > >
>>>> > > > >>>>>
>>>> > > > >
>>>> > > > >
>>>> > > > >>>>>
>>>> > > >
>>>> > > > >>>>>
>>>> > > > >
>>>> > > > >>>>>
>>>> > > >
>>>> > > > >>>>> On Thu, Sep 19, 2024 at 11:26 AM Josh McKenzie
>>>> > > > <[jmckenzie@apache.org](mailto:jmckenzie@apache.org)> wrote:
>>>> > > > >
>>>> > > > >>>>>
>>>> > > >
>>>> > > > >>>>>> __
>>>> > > >
>>>> > > > >>>>>>
>>>> > > >
>>>> > > > >>>>>> Not quite; finishing touches on the CEP and design doc are
>>>> in
>>>> > > flight
>>>> > > > (as of last / this week).
>>>> > > > >
>>>> > > > >>>>>>
>>>> > > >
>>>> > > > >>>>>>
>>>> > > > >
>>>> > > > >>>>>>
>>>> > > >
>>>> > > > >>>>>> Soon(tm).
>>>> > > >
>>>> > > > >>>>>>
>>>> > > >
>>>> > > > >>>>>>
>>>> > > > >
>>>> > > > >>>>>>
>>>> > > >
>>>> > > > >>>>>> On Thu, Sep 19, 2024, at 2:07 PM, Patrick McFadin wrote:
>>>> > > > >
>>>> > > > >>>>>>
>>>> > > >
>>>> > > > >>>>>>> Is this CEP ready for a VOTE thread?
>>>> > > > <
>>>> > >
>>>> https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-37+%28DRAFT%29+Apache+Cassandra+Unified+Repair+Solution
>>>> >
>>>> > >
>>>> > > > >
>>>> > > > >>>>>>>
>>>> > > >
>>>> > > > >>>>>>>
>>>> > > > >
>>>> > > > >>>>>>>
>>>> > > >
>>>> > > > >>>>>>> On Sun, Feb 25, 2024 at 12:25 PM Jaydeep Chovatia
>>>> > > > <[chovatia.jaydeep@gmail.com](mailto:chovatia.jaydeep@gmail.com)>
>>>> > > wrote:
>>>> > > > >
>>>> > > > >>>>>>>
>>>> > > >
>>>> > > > >>>>>>>> Thanks, Josh. I've just updated the
>>>> > > > [CEP](
>>>> > >
>>>> https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-37+%28DRAFT%29+Apache+Cassandra+Official+Repair+Solution
>>>> > > )
>>>> > > > and included all the solutions you mentioned below.
>>>> > > > >
>>>> > > > >>>>>>>>
>>>> > > >
>>>> > > > >>>>>>>>
>>>> > > > >
>>>> > > > >>>>>>>>
>>>> > > >
>>>> > > > >>>>>>>> Jaydeep
>>>> > > > >
>>>> > > > >>>>>>>>
>>>> > > >
>>>> > > > >>>>>>>>
>>>> > > > >
>>>> > > > >>>>>>>>
>>>> > > >
>>>> > > > >>>>>>>> On Thu, Feb 22, 2024 at 9:33 AM Josh McKenzie
>>>> > > > <[jmckenzie@apache.org](mailto:jmckenzie@apache.org)> wrote:
>>>> > > > >
>>>> > > > >>>>>>>>
>>>> > > >
>>>> > > > >>>>>>>>> __
>>>> > > > >
>>>> > > > >>>>>>>>>
>>>> > > >
>>>> > > > >>>>>>>>> Very late response from me here (basically necro'ing
>>>> this
>>>> > > thread).
>>>> > > > >
>>>> > > > >>>>>>>>>
>>>> > > >
>>>> > > > >>>>>>>>>
>>>> > > > >
>>>> > > > >>>>>>>>>
>>>> > > >
>>>> > > > >>>>>>>>> I think it'd be useful to get this condensed into a CEP
>>>> that
>>>> > > we can
>>>> > > > then discuss in that format. It's clearly something we all agree
>>>> we need
>>>> > > and
>>>> > > > having an implementation that works, even if it's not in your
>>>> preferred
>>>> > > > execution domain, is vastly better than nothing IMO.
>>>> > > > >
>>>> > > > >>>>>>>>>
>>>> > > >
>>>> > > > >>>>>>>>>
>>>> > > > >
>>>> > > > >>>>>>>>>
>>>> > > >
>>>> > > > >>>>>>>>> I don't have cycles (nor background ;) ) to do that,
>>>> but it
>>>> > > sounds
>>>> > > > like you do Jaydeep given the implementation you have on a
>>>> private fork +
>>>> > > > design.
>>>> > > > >
>>>> > > > >>>>>>>>>
>>>> > > >
>>>> > > > >>>>>>>>>
>>>> > > > >
>>>> > > > >>>>>>>>>
>>>> > > >
>>>> > > > >>>>>>>>> A non-exhaustive list of things that might be useful
>>>> > > incorporating
>>>> > > > into or referencing from a CEP:
>>>> > > > >
>>>> > > > >>>>>>>>>
>>>> > > >
>>>> > > > >>>>>>>>> Slack thread: <https://the-
>>>> > > > asf.slack.com/archives/CK23JSY2K/p1690225062383619>
>>>> > > > >
>>>> > > > >>>>>>>>>
>>>> > > >
>>>> > > > >>>>>>>>> Joey's old C* ticket:
>>>> > > > <https://issues.apache.org/jira/browse/CASSANDRA-14346>
>>>> > > > >
>>>> > > > >>>>>>>>>
>>>> > > >
>>>> > > > >>>>>>>>> Even older automatic repair scheduling:
>>>> > > > <https://issues.apache.org/jira/browse/CASSANDRA-10070>
>>>> > > > >
>>>> > > > >>>>>>>>>
>>>> > > >
>>>> > > > >>>>>>>>> Your design gdoc: <
>>>> > > https://docs.google.com/document/d/1CJWxjEi-
>>>> > > > mBABPMZ3VWJ9w5KavWfJETAGxfUpsViPcPo/edit#heading=h.r112r46toau0>
>>>> > > > >
>>>> > > > >>>>>>>>>
>>>> > > >
>>>> > > > >>>>>>>>> PR with automated repair:
>>>> > > > <
>>>> > >
>>>> https://github.com/jaydeepkumar1984/cassandra/commit/ef6456d652c0d07cf29d88dfea03b73704814c2c
>>>> >
>>>> > >
>>>> > > > >
>>>> > > > >>>>>>>>>
>>>> > > >
>>>> > > > >>>>>>>>>
>>>> > > > >
>>>> > > > >>>>>>>>>
>>>> > > >
>>>> > > > >>>>>>>>> My intuition is that we're all basically in agreement
>>>> that
>>>> > > this is
>>>> > > > something the DB needs, we're all willing to bikeshed for our
>>>> personal
>>>> > > > preference on where it lives and how it's implemented, and at the
>>>> end of
>>>> > > the
>>>> > > > day, code talks. I don't think anyone's said they'll die on the
>>>> hill of
>>>> > > > implementation details, so that feels like CEP time to me.
>>>> > > > >
>>>> > > > >>>>>>>>>
>>>> > > >
>>>> > > > >>>>>>>>>
>>>> > > > >
>>>> > > > >>>>>>>>>
>>>> > > >
>>>> > > > >>>>>>>>> If you were willing and able to get a CEP together for
>>>> > > automated
>>>> > > > repair based on the above material, given you've done the work
>>>> and have
>>>> > > the
>>>> > > > proof points it's working at scale, I think this would be a _huge
>>>> > > > contribution_ to the community.
>>>> > > > >
>>>> > > > >>>>>>>>>
>>>> > > >
>>>> > > > >>>>>>>>>
>>>> > > > >
>>>> > > > >>>>>>>>>
>>>> > > >
>>>> > > > >>>>>>>>> On Thu, Aug 24, 2023, at 7:26 PM, Jaydeep Chovatia
>>>> wrote:
>>>> > > > >
>>>> > > > >>>>>>>>>
>>>> > > >
>>>> > > > >>>>>>>>>> Is anyone going to file an official CEP for this?
>>>> > > > >
>>>> > > > >>>>>>>>>>
>>>> > > >
>>>> > > > >>>>>>>>>> As mentioned in this email thread, here is one of the
>>>> > > solution's
>>>> > > > [design doc](https://docs.google.com/document/d/1CJWxjEi-
>>>> > > > mBABPMZ3VWJ9w5KavWfJETAGxfUpsViPcPo/edit#heading=h.r112r46toau0)
>>>> and
>>>> > > source
>>>> > > > code on a private Apache Cassandra patch. Could you go through it
>>>> and
>>>> > > let me
>>>> > > > know what you think?
>>>> > > > >
>>>> > > > >>>>>>>>>>
>>>> > > >
>>>> > > > >>>>>>>>>>
>>>> > > > >
>>>> > > > >>>>>>>>>>
>>>> > > >
>>>> > > > >>>>>>>>>> Jaydeep
>>>> > > > >
>>>> > > > >>>>>>>>>>
>>>> > > >
>>>> > > > >>>>>>>>>>
>>>> > > > >
>>>> > > > >>>>>>>>>>
>>>> > > >
>>>> > > > >>>>>>>>>> On Wed, Aug 2, 2023 at 3:54 PM Jon Haddad
>>>> > > > <[rustyrazorblade@apache.org](mailto:rustyrazorblade@apache.org)>
>>>> > > wrote:
>>>> > > > >
>>>> > > > >>>>>>>>>>
>>>> > > >
>>>> > > > >>>>>>>>>>> > That said I would happily support an effort to
>>>> bring repair
>>>> > > > scheduling to the sidecar immediately. This has nothing blocking
>>>> it, and
>>>> > > would
>>>> > > > potentially enable the sidecar to provide an official repair
>>>> scheduling
>>>> > > > solution that is compatible with current or even previous
>>>> versions of the
>>>> > > > database.
>>>> > > > >
>>>> > > > >>>>>>>>>>>
>>>> > > >
>>>> > > > >>>>>>>>>>>
>>>> > > > >
>>>> > > > >>>>>>>>>>>
>>>> > > >
>>>> > > > >>>>>>>>>>> This is something I hadn't thought much about, and is
>>>> a
>>>> > > pretty
>>>> > > > good argument for using the sidecar initially. There's a lot of
>>>> > > deployments
>>>> > > > out there and having an official repair option would be a big win.
>>>> > > > >
>>>> > > > >>>>>>>>>>>
>>>> > > >
>>>> > > > >>>>>>>>>>>
>>>> > > > >
>>>> > > > >>>>>>>>>>>
>>>> > > >
>>>> > > > >>>>>>>>>>>
>>>> > > > >
>>>> > > > >>>>>>>>>>>
>>>> > > >
>>>> > > > >>>>>>>>>>> On 2023/07/26 23:20:07 "C. Scott Andreas" wrote:
>>>> > > > >
>>>> > > > >>>>>>>>>>>
>>>> > > >
>>>> > > > >>>>>>>>>>> > I agree that it would be ideal for Cassandra to
>>>> have a
>>>> > > repair
>>>> > > > scheduler in-DB.
>>>> > > > >
>>>> > > > >>>>>>>>>>>
>>>> > > >
>>>> > > > >>>>>>>>>>> >
>>>> > > > >
>>>> > > > >>>>>>>>>>>
>>>> > > >
>>>> > > > >>>>>>>>>>> > That said I would happily support an effort to
>>>> bring repair
>>>> > > > scheduling to the sidecar immediately. This has nothing blocking
>>>> it, and
>>>> > > would
>>>> > > > potentially enable the sidecar to provide an official repair
>>>> scheduling
>>>> > > > solution that is compatible with current or even previous
>>>> versions of the
>>>> > > > database.
>>>> > > > >
>>>> > > > >>>>>>>>>>>
>>>> > > >
>>>> > > > >>>>>>>>>>> >
>>>> > > > >
>>>> > > > >>>>>>>>>>>
>>>> > > >
>>>> > > > >>>>>>>>>>> > Once TCM has landed, we’ll have much stronger
>>>> primitives
>>>> > > for
>>>> > > > repair orchestration in the database itself. But I don’t think
>>>> that
>>>> > > should
>>>> > > > block progress on a repair scheduling solution in the sidecar,
>>>> and there
>>>> > > is
>>>> > > > nothing that would prevent someone from continuing to use a
>>>> sidecar-based
>>>> > > > solution in perpetuity if they preferred.
>>>> > > > >
>>>> > > > >>>>>>>>>>>
>>>> > > >
>>>> > > > >>>>>>>>>>> >
>>>> > > > >
>>>> > > > >>>>>>>>>>>
>>>> > > >
>>>> > > > >>>>>>>>>>> > \- Scott
>>>> > > > >
>>>> > > > >>>>>>>>>>>
>>>> > > >
>>>> > > > >>>>>>>>>>> >
>>>> > > > >
>>>> > > > >>>>>>>>>>>
>>>> > > >
>>>> > > > >>>>>>>>>>> > > On Jul 26, 2023, at 3:25 PM, Jon Haddad
>>>> > > > <[rustyrazorblade@apache.org](mailto:rustyrazorblade@apache.org)>
>>>> > > wrote:
>>>> > > > >
>>>> > > > >>>>>>>>>>>
>>>> > > >
>>>> > > > >>>>>>>>>>> > >
>>>> > > > >
>>>> > > > >>>>>>>>>>>
>>>> > > >
>>>> > > > >>>>>>>>>>> > > I'm 100% in favor of repair being part of the
>>>> core DB,
>>>> > > not
>>>> > > > the sidecar. The current (and past) state of things where
>>>> running the DB
>>>> > > > correctly *requires* running a separate process (either community
>>>> > > maintained
>>>> > > > or official C* sidecar) is incredibly painful for folks. The
>>>> idea that
>>>> > > your
>>>> > > > data integrity needs to be opt-in has never made sense to me from
>>>> the
>>>> > > > perspective of either the product or the end user.
>>>> > > > >
>>>> > > > >>>>>>>>>>>
>>>> > > >
>>>> > > > >>>>>>>>>>> > >
>>>> > > > >
>>>> > > > >>>>>>>>>>>
>>>> > > >
>>>> > > > >>>>>>>>>>> > > I've worked with way too many teams that have
>>>> either
>>>> > > > configured this incorrectly or not at all.
>>>> > > > >
>>>> > > > >>>>>>>>>>>
>>>> > > >
>>>> > > > >>>>>>>>>>> > >
>>>> > > > >
>>>> > > > >>>>>>>>>>>
>>>> > > >
>>>> > > > >>>>>>>>>>> > > Ideally Cassandra would ship with repair built in
>>>> and on
>>>> > > by
>>>> > > > default. Power users can disable if they want to continue to
>>>> maintain
>>>> > > their
>>>> > > > own repair tooling for some reason.
>>>> > > > >
>>>> > > > >>>>>>>>>>>
>>>> > > >
>>>> > > > >>>>>>>>>>> > >
>>>> > > > >
>>>> > > > >>>>>>>>>>>
>>>> > > >
>>>> > > > >>>>>>>>>>> > > Jon
>>>> > > > >
>>>> > > > >>>>>>>>>>>
>>>> > > >
>>>> > > > >>>>>>>>>>> > >
>>>> > > > >
>>>> > > > >>>>>>>>>>>
>>>> > > >
>>>> > > > >>>>>>>>>>> > >> On 2023/07/24 20:44:14 German Eichberger via dev
>>>> > > wrote:
>>>> > > > >
>>>> > > > >>>>>>>>>>>
>>>> > > >
>>>> > > > >>>>>>>>>>> > >> All,
>>>> > > > >
>>>> > > > >>>>>>>>>>>
>>>> > > >
>>>> > > > >>>>>>>>>>> > >> We had a brief discussion in [2] about the Uber
>>>> article
>>>> > > [1]
>>>> > > > where they talk about having integrated repair into Cassandra and
>>>> how
>>>> > > great
>>>> > > > that is. I expressed my disappointment that they didn't work with
>>>> the
>>>> > > > community on that (Uber, if you are listening time to make amends
>>>> 🙂)
>>>> > > and it
>>>> > > > turns out Joey already had the idea and wrote the code [3] - so I
>>>> wanted
>>>> > > to
>>>> > > > start a discussion to gauge interest and maybe how to revive that
>>>> > > effort.
>>>> > > > >
>>>> > > > >>>>>>>>>>>
>>>> > > >
>>>> > > > >>>>>>>>>>> > >> Thanks,
>>>> > > > >
>>>> > > > >>>>>>>>>>>
>>>> > > >
>>>> > > > >>>>>>>>>>> > >> German
>>>> > > > >
>>>> > > > >>>>>>>>>>>
>>>> > > >
>>>> > > > >>>>>>>>>>> > >> [1] <
>>>> > > https://www.uber.com/blog/how-uber-optimized-cassandra-
>>>> > > > operations-at-scale/>
>>>> > > > >
>>>> > > > >>>>>>>>>>>
>>>> > > >
>>>> > > > >>>>>>>>>>> > >> [2] <https://the-
>>>> > > > asf.slack.com/archives/CK23JSY2K/p1690225062383619>
>>>> > > > >
>>>> > > > >>>>>>>>>>>
>>>> > > >
>>>> > > > >>>>>>>>>>> > >> [3] <
>>>> > > https://issues.apache.org/jira/browse/CASSANDRA-14346>
>>>> > > > >
>>>> > > > >>>>>>>>>>>
>>>> > > >
>>>> > > > >>>>>>>>>>> >
>>>> > > > >
>>>> > > > >>>>>>>>>
>>>> > > >
>>>> > > > >>>>>>>>>
>>>> > > > >
>>>> > > > >>>>>>
>>>> > > >
>>>> > > > >>>>>>
>>>> > > > >
>>>> > > >
>>>> > > >
>>>> > >
>>>> >
>>>>
>>>
Re: [Discuss] Repair inside C*
Posted by Benedict <be...@apache.org>.
I realise it’s out of scope, but to counterbalance all of the pro-
decomposition messages I wanted to chime in with a strong -1. But we can
debate that in a suitable context later.
> On 22 Oct 2024, at 16:36, Jordan West <jw...@apache.org> wrote:
>
>
>
>
> Agreed with the sentiment that decomposition is a good target but out of
> scope here. I’m personally excited to see an in-tree repair scheduler and am
> supportive of the approach shared here.
>
>
>
>
> Jordan
>
>
>
>
> On Tue, Oct 22, 2024 at 08:12 Dinesh Joshi
> <[djoshi@apache.org](mailto:djoshi@apache.org)> wrote:
>
>
>> Decomposing Cassandra may be architecturally desirable but that is not the
goal of this CEP. This CEP brings value to operators today so it should be
considered on that merit. We definitely need to have a separate conversation
on Cassandra's architectural direction.
>>
>>
>
>>
>> On Tue, Oct 22, 2024 at 7:51 AM Joseph Lynch
<[joe.e.lynch@gmail.com](mailto:joe.e.lynch@gmail.com)> wrote:
>
>>
>>> Definitely like this in C* itself. We only changed our proposal to putting
repair scheduling in the sidecar before because trunk was frozen for the
foreseeable future at that time. With trunk unfrozen and development on the
main process going at a fast pace I think it makes way more sense to integrate
natively as table properties as this CEP proposes. Completely agree the
scheduling overhead should be minimal.
>>>
>>>
>
>>>
>>> Moving the actual repair operation (comparing data and streaming
mismatches) along with compaction operations to a separate process long term
makes a lot of sense but imo only once we both have a release of sidecar and a
contract figured out between them on communication. I'm watching CEP-38 there
as I think CQL and virtual tables are looking much stronger than when we wrote
CEP-1 and chose HTTP but that's for that discussion and not this one.
>>>
>>>
>
>>>
>>> -Joey
>>>
>>>
>
>>>
>>> On Mon, Oct 21, 2024 at 3:25 PM Francisco Guerrero
<[frankgh@apache.org](mailto:frankgh@apache.org)> wrote:
>
>>>
>>>> Like others have said, I was expecting the scheduling portion of repair
is
> negligible. I was mostly curious if you had something handy that you can
> quickly share.
>
> On 2024/10/21 18:59:41 Jaydeep Chovatia wrote:
> > >Jaydeep, do you have any metrics on your clusters comparing them before
> > and after introducing repair scheduling into the Cassandra process?
> >
> > Yes, I had made some comparisons when I started rolling this feature out
> to
> > our production five years ago :) Here are the details:
> > *The Scheduling*
> > The scheduling itself is exceptionally lightweight, as only one
> additional
> > thread monitors the repair activity, updating the status to a system
> table
> > once every few minutes or so. So, it does not appear anywhere in the CPU
> > charts, etc. Unfortunately, I do not have those graphs now, but I can do
> a
> > quick comparison if it helps!
> >
> > *The Repair Itself*
> > As we all know, the Cassandra repair algorithm is a heavy-weight process
> > due to Merkle tree/streaming, etc., no matter how we schedule it. But it
> is
> > an orthogonal topic and folks are already discussing creating a new CEP.
> >
> > Jaydeep
> >
> >
> > On Mon, Oct 21, 2024 at 10:02 AM Francisco Guerrero
> <[frankgh@apache.org](mailto:frankgh@apache.org)>
> > wrote:
> >
> > > Jaydeep, do you have any metrics on your clusters comparing them before
> > > and after introducing repair scheduling into the Cassandra process?
> > >
> > > On 2024/10/21 16:57:57 "J. D. Jordan" wrote:
> > > > Sounds good. Just wanted to bring it up. I agree that the scheduling
> bit
> > > is
> > > > pretty light weight and the ideal would be to bring the whole of the
> > > repair
> > > > external, which is a much bigger can of worms to open.
> > > >
> > > >
> > > >
> > > > -Jeremiah
> > > >
> > > >
> > > >
> > > > > On Oct 21, 2024, at 11:21 AM, Chris Lohfink
> <[clohfink85@gmail.com](mailto:clohfink85@gmail.com)>
> > > wrote:
> > > > >
> > > > >
> > > >
> > > > >
> > > > >
> > > > > > I actually think we should be looking at how we can move things
> out
> > > of the
> > > > > database process.
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > While worth pursuing, I think we would need a different CEP just to
> > > figure
> > > > > out how to do that. Not only is there a lot of infrastructure
> > > difficulty in
> > > > > running multi process, the inter app communication needs to be
> figured
> > > out
> > > > > better then JMX. Even the sidecar we dont have a solid story on how
> to
> > > > > ensure both are running or anything yet. It's up to each app owner
> to
> > > figure
> > > > > it out. Once we have a good thing in place I think we can start
> moving
> > > > > compactions, repairs, etc out of the database. Even then it's the
> > > _repairs_
> > > > > that is expensive, not the scheduling.
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > On Mon, Oct 21, 2024 at 9:45 AM Jeremiah Jordan
> > > > >
> <[[jeremiah.jordan@gmail.com](mailto:jeremiah.jordan@gmail.com)](mailto:[jeremiah.jordan@gmail.com](mailto:jeremiah.jordan@gmail.com))>
> > > wrote:
> > > > >
> > > > >
> > > >
> > > > >> I love the idea of a repair service being there by default for an
> > > install
> > > > of C*. My main concern here is that it is putting more services into
> > > the main
> > > > database process. I actually think we should be looking at how we
> can
> > > move
> > > > things out of the database process. The C* process being a giant
> > > monolith has
> > > > always been a pain point. Is there anyway it makes sense for this to
> be
> > > an
> > > > external process rather than a new thread pool inside the C* process?
> > > >
> > > > >>
> > > >
> > > > >>
> > > > >
> > > > >>
> > > >
> > > > >> -Jeremiah Jordan
> > > >
> > > > >>
> > > >
> > > > >>
> > > > >
> > > > >>
> > > >
> > > > >> On Oct 18, 2024 at 2:58:15 PM, Mick Semb Wever
> > > >
> <[[mck@apache.org](mailto:mck@apache.org)](mailto:[mck@apache.org](mailto:mck@apache.org))>
> wrote:
> > > > >
> > > > >>
> > > >
> > > > >>>
> > > > >
> > > > >>>
> > > >
> > > > >>> This is looking strong, thanks Jaydeep.
> > > >
> > > > >>>
> > > >
> > > > >>>
> > > > >
> > > > >>>
> > > >
> > > > >>> I would suggest folk take a look at the design doc and the PR in
> the
> > > CEP.
> > > > A lot is there (that I have completely missed).
> > > >
> > > > >>>
> > > >
> > > > >>>
> > > > >
> > > > >>>
> > > >
> > > > >>> I would especially ask all authors of prior art (Reaper, DSE
> > > nodesync,
> > > > ecchronos) to take a final review of the proposal
> > > > >
> > > > >>>
> > > >
> > > > >>>
> > > > >
> > > > >>>
> > > >
> > > > >>> Jaydeep, can we ask for a two week window while we reach out to
> these
> > > > people ? There's a lot of prior art in this space, and it feels like
> > > we're in
> > > > a good place now where it's clear this has legs and we can use that
> to
> > > bring
> > > > folk in and make sure there's no remaining blindspots.
> > > >
> > > > >>>
> > > >
> > > > >>>
> > > > >
> > > > >>>
> > > >
> > > > >>>
> > > > >
> > > > >>>
> > > >
> > > > >>> On Fri, 18 Oct 2024 at 01:40, Jaydeep Chovatia
> > > >
> <[[chovatia.jaydeep@gmail.com](mailto:chovatia.jaydeep@gmail.com)](mailto:[chovatia.jaydeep@gmail.com](mailto:chovatia.jaydeep@gmail.com))>
> > > wrote:
> > > > >
> > > > >>>
> > > >
> > > > >>>> Sorry, there is a typo in the CEP-37 link; here is the correct
> > > > [link](
> > >
> <https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-37+Apache+Cassandra+Unified+Repair+Solution>
> > > )
> > > >
> > > > >>>>
> > > >
> > > > >>>>
> > > > >
> > > > >>>>
> > > >
> > > > >>>>
> > > > >
> > > > >>>>
> > > >
> > > > >>>> On Thu, Oct 17, 2024 at 4:36 PM Jaydeep Chovatia
> > > >
> <[[chovatia.jaydeep@gmail.com](mailto:chovatia.jaydeep@gmail.com)](mailto:[chovatia.jaydeep@gmail.com](mailto:chovatia.jaydeep@gmail.com))>
> > > wrote:
> > > > >
> > > > >>>>
> > > >
> > > > >>>>> First, thank you for your patience while we strengthened the
> > > CEP-37.
> > > >
> > > > >>>>>
> > > >
> > > > >>>>>
> > > > >
> > > > >>>>>
> > > >
> > > > >>>>> Over the last eight months, Chris Lohfink, Andy Tolbert, Josh
> > > McKenzie,
> > > > Dinesh Joshi, Kristijonas Zalys, and I have done tons of work (online
> > > > discussions/a dedicated Slack channel
> > > #cassandra-repair-scheduling-cep37) to
> > > > come up with the best possible design that not only significantly
> > > simplifies
> > > > repair operations but also includes the most common features that
> > > everyone
> > > > will benefit from running at Scale.
> > > >
> > > > >>>>>
> > > >
> > > > >>>>> For example,
> > > >
> > > > >>>>>
> > > >
> > > > >>>>> * Apache Cassandra must be capable of running multiple repair
> > > types,
> > > > such as Full, Incremental, Paxos, and Preview - so the framework
> should
> > > be
> > > > easily extendable with no additional overhead from the operator’s
> point
> > > of
> > > > view.
> > > >
> > > > >>>>>
> > > >
> > > > >>>>> * An easy way to extend the token-split calculation algorithm
> > > with a
> > > > default implementation should exist.
> > > >
> > > > >>>>>
> > > >
> > > > >>>>> * Running incremental repair reliably at Scale is pretty
> > > challenging,
> > > > so we need to place safeguards, such as migration/rollback w/o
> restart
> > > and
> > > > stopping incremental repair automatically if the disk is about to get
> > > full.
> > > >
> > > > >>>>>
> > > >
> > > > >>>>>
> > > >
> > > > >>>>>
> > > >
> > > > >>>>> We are glad to inform you that CEP-37 (i.e., Repair inside
> > > Cassandra) is
> > > > now officially ready for review after multiple rounds of design,
> > > testing, code
> > > > reviews, documentation reviews, and, more importantly, validation
> that
> > > it runs
> > > > at Scale!
> > > >
> > > > >>>>>
> > > >
> > > > >>>>>
> > > > >
> > > > >>>>>
> > > >
> > > > >>>>> Some facts about CEP-37.
> > > >
> > > > >>>>>
> > > >
> > > > >>>>> * Multiple members have verified all aspects of CEP-37
> numerous
> > > times.
> > > >
> > > > >>>>>
> > > >
> > > > >>>>> * The design proposed in CEP-37 has been thoroughly tried and
> > > tested
> > > > on an immense scale (hundreds of unique Cassandra clusters, tens of
> > > thousands
> > > > of Cassandra nodes, with tens of millions of QPS) on top of 4.1
> > > open-source
> > > > for more than five years; please see more details[
> > > > here](
> > > <https://www.uber.com/en-US/blog/how-uber-optimized-cassandra-
> operations->
> > > > at-scale/).
> > > >
> > > > >>>>>
> > > >
> > > > >>>>> * The following
> > > > [presentation](
> > >
> <https://docs.google.com/presentation/d/1Zilww9c7LihHULk_ckErI2s4XbObxjWknKqRtbvHyZc/edit#slide=id.g30a4fd4fcf7_0_13>
> > > )
> > > > highlights the rigorous applied to CEP-37, which was given during
> last
> > > week’s
> > > > Apache Cassandra Bay Area [Meetup](
> > > <https://www.meetup.com/apache-cassandra->
> > > > bay-area/events/303469006/),
> > > >
> > > > >>>>>
> > > >
> > > > >>>>>
> > > >
> > > >
> > > > >
> > > > >>>>>
> > > >
> > > > >>>>> Since things are massively overhauled, we believe it is almost
> > > ready for
> > > > a final pass pre-VOTE. We would like you to please review the
> > > > [CEP-37](
> > >
> [https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-37+Apache+Cassandra+Unified+Repair+Solution\](https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-37+Apache+Cassandra+Unified+Repair+Solution%5C)
> > >
> <<https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-37+Apache+Cassandra+Unified+Repair+Solution%5C>>
> > > ))
> > > > and the associated detailed design
> > > > [doc](<https://docs.google.com/document/d/1CJWxjEi->
> > > > mBABPMZ3VWJ9w5KavWfJETAGxfUpsViPcPo/edit#heading=h.r112r46toau0).
> > > >
> > > > >>>>>
> > > >
> > > > >>>>>
> > > > >
> > > > >>>>>
> > > >
> > > > >>>>> Thank you everyone!
> > > >
> > > > >>>>>
> > > >
> > > > >>>>> Chris, Andy, Josh, Dinesh, Kristijonas, and Jaydeep
> > > >
> > > > >>>>>
> > > >
> > > > >>>>>
> > > > >
> > > > >
> > > > >>>>>
> > > >
> > > > >>>>>
> > > > >
> > > > >>>>>
> > > >
> > > > >>>>> On Thu, Sep 19, 2024 at 11:26 AM Josh McKenzie
> > > >
> <[[jmckenzie@apache.org](mailto:jmckenzie@apache.org)](mailto:[jmckenzie@apache.org](mailto:jmckenzie@apache.org))>
> wrote:
> > > > >
> > > > >>>>>
> > > >
> > > > >>>>>> __
> > > >
> > > > >>>>>>
> > > >
> > > > >>>>>> Not quite; finishing touches on the CEP and design doc are in
> > > flight
> > > > (as of last / this week).
> > > > >
> > > > >>>>>>
> > > >
> > > > >>>>>>
> > > > >
> > > > >>>>>>
> > > >
> > > > >>>>>> Soon(tm).
> > > >
> > > > >>>>>>
> > > >
> > > > >>>>>>
> > > > >
> > > > >>>>>>
> > > >
> > > > >>>>>> On Thu, Sep 19, 2024, at 2:07 PM, Patrick McFadin wrote:
> > > > >
> > > > >>>>>>
> > > >
> > > > >>>>>>> Is this CEP ready for a VOTE thread?
> > > > <
> > >
> <https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-37+%28DRAFT%29+Apache+Cassandra+Unified+Repair+Solution>>
> > >
> > > > >
> > > > >>>>>>>
> > > >
> > > > >>>>>>>
> > > > >
> > > > >>>>>>>
> > > >
> > > > >>>>>>> On Sun, Feb 25, 2024 at 12:25 PM Jaydeep Chovatia
> > > >
> <[[chovatia.jaydeep@gmail.com](mailto:chovatia.jaydeep@gmail.com)](mailto:[chovatia.jaydeep@gmail.com](mailto:chovatia.jaydeep@gmail.com))>
> > > wrote:
> > > > >
> > > > >>>>>>>
> > > >
> > > > >>>>>>>> Thanks, Josh. I've just updated the
> > > > [CEP](
> > >
> <https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-37+%28DRAFT%29+Apache+Cassandra+Official+Repair+Solution>
> > > )
> > > > and included all the solutions you mentioned below.
> > > > >
> > > > >>>>>>>>
> > > >
> > > > >>>>>>>>
> > > > >
> > > > >>>>>>>>
> > > >
> > > > >>>>>>>> Jaydeep
> > > > >
> > > > >>>>>>>>
> > > >
> > > > >>>>>>>>
> > > > >
> > > > >>>>>>>>
> > > >
> > > > >>>>>>>> On Thu, Feb 22, 2024 at 9:33 AM Josh McKenzie
> > > >
> <[[jmckenzie@apache.org](mailto:jmckenzie@apache.org)](mailto:[jmckenzie@apache.org](mailto:jmckenzie@apache.org))>
> wrote:
> > > > >
> > > > >>>>>>>>
> > > >
> > > > >>>>>>>>> __
> > > > >
> > > > >>>>>>>>>
> > > >
> > > > >>>>>>>>> Very late response from me here (basically necro'ing this
> > > thread).
> > > > >
> > > > >>>>>>>>>
> > > >
> > > > >>>>>>>>>
> > > > >
> > > > >>>>>>>>>
> > > >
> > > > >>>>>>>>> I think it'd be useful to get this condensed into a CEP
> that
> > > we can
> > > > then discuss in that format. It's clearly something we all agree we
> need
> > > and
> > > > having an implementation that works, even if it's not in your
> preferred
> > > > execution domain, is vastly better than nothing IMO.
> > > > >
> > > > >>>>>>>>>
> > > >
> > > > >>>>>>>>>
> > > > >
> > > > >>>>>>>>>
> > > >
> > > > >>>>>>>>> I don't have cycles (nor background ;) ) to do that, but it
> > > sounds
> > > > like you do Jaydeep given the implementation you have on a private
> fork +
> > > > design.
> > > > >
> > > > >>>>>>>>>
> > > >
> > > > >>>>>>>>>
> > > > >
> > > > >>>>>>>>>
> > > >
> > > > >>>>>>>>> A non-exhaustive list of things that might be useful
> > > incorporating
> > > > into or referencing from a CEP:
> > > > >
> > > > >>>>>>>>>
> > > >
> > > > >>>>>>>>> Slack thread: <<https://the->
> > > >
> [asf.slack.com/archives/CK23JSY2K/p1690225062383619](http://asf.slack.com/archives/CK23JSY2K/p1690225062383619)>
> > > > >
> > > > >>>>>>>>>
> > > >
> > > > >>>>>>>>> Joey's old C* ticket:
> > > > <<https://issues.apache.org/jira/browse/CASSANDRA-14346>>
> > > > >
> > > > >>>>>>>>>
> > > >
> > > > >>>>>>>>> Even older automatic repair scheduling:
> > > > <<https://issues.apache.org/jira/browse/CASSANDRA-10070>>
> > > > >
> > > > >>>>>>>>>
> > > >
> > > > >>>>>>>>> Your design gdoc: <
> > > <https://docs.google.com/document/d/1CJWxjEi->
> > > > mBABPMZ3VWJ9w5KavWfJETAGxfUpsViPcPo/edit#heading=h.r112r46toau0>
> > > > >
> > > > >>>>>>>>>
> > > >
> > > > >>>>>>>>> PR with automated repair:
> > > > <
> > >
> <https://github.com/jaydeepkumar1984/cassandra/commit/ef6456d652c0d07cf29d88dfea03b73704814c2c>>
> > >
> > > > >
> > > > >>>>>>>>>
> > > >
> > > > >>>>>>>>>
> > > > >
> > > > >>>>>>>>>
> > > >
> > > > >>>>>>>>> My intuition is that we're all basically in agreement that
> > > this is
> > > > something the DB needs, we're all willing to bikeshed for our
> personal
> > > > preference on where it lives and how it's implemented, and at the end
> of
> > > the
> > > > day, code talks. I don't think anyone's said they'll die on the hill
> of
> > > > implementation details, so that feels like CEP time to me.
> > > > >
> > > > >>>>>>>>>
> > > >
> > > > >>>>>>>>>
> > > > >
> > > > >>>>>>>>>
> > > >
> > > > >>>>>>>>> If you were willing and able to get a CEP together for
> > > automated
> > > > repair based on the above material, given you've done the work and
> have
> > > the
> > > > proof points it's working at scale, I think this would be a _huge
> > > > contribution_ to the community.
> > > > >
> > > > >>>>>>>>>
> > > >
> > > > >>>>>>>>>
> > > > >
> > > > >>>>>>>>>
> > > >
> > > > >>>>>>>>> On Thu, Aug 24, 2023, at 7:26 PM, Jaydeep Chovatia wrote:
> > > > >
> > > > >>>>>>>>>
> > > >
> > > > >>>>>>>>>> Is anyone going to file an official CEP for this?
> > > > >
> > > > >>>>>>>>>>
> > > >
> > > > >>>>>>>>>> As mentioned in this email thread, here is one of the
> > > solution's
> > > > [design doc](<https://docs.google.com/document/d/1CJWxjEi->
> > > > mBABPMZ3VWJ9w5KavWfJETAGxfUpsViPcPo/edit#heading=h.r112r46toau0) and
> > > source
> > > > code on a private Apache Cassandra patch. Could you go through it and
> > > let me
> > > > know what you think?
> > > > >
> > > > >>>>>>>>>>
> > > >
> > > > >>>>>>>>>>
> > > > >
> > > > >>>>>>>>>>
> > > >
> > > > >>>>>>>>>> Jaydeep
> > > > >
> > > > >>>>>>>>>>
> > > >
> > > > >>>>>>>>>>
> > > > >
> > > > >>>>>>>>>>
> > > >
> > > > >>>>>>>>>> On Wed, Aug 2, 2023 at 3:54 PM Jon Haddad
> > > >
> <[[rustyrazorblade@apache.org](mailto:rustyrazorblade@apache.org)](mailto:[rustyrazorblade@apache.org](mailto:rustyrazorblade@apache.org))>
> > > wrote:
> > > > >
> > > > >>>>>>>>>>
> > > >
> > > > >>>>>>>>>>> > That said I would happily support an effort to bring
> repair
> > > > scheduling to the sidecar immediately. This has nothing blocking it,
> and
> > > would
> > > > potentially enable the sidecar to provide an official repair
> scheduling
> > > > solution that is compatible with current or even previous versions of
> the
> > > > database.
> > > > >
> > > > >>>>>>>>>>>
> > > >
> > > > >>>>>>>>>>>
> > > > >
> > > > >>>>>>>>>>>
> > > >
> > > > >>>>>>>>>>> This is something I hadn't thought much about, and is a
> > > pretty
> > > > good argument for using the sidecar initially. There's a lot of
> > > deployments
> > > > out there and having an official repair option would be a big win.
> > > > >
> > > > >>>>>>>>>>>
> > > >
> > > > >>>>>>>>>>>
> > > > >
> > > > >>>>>>>>>>>
> > > >
> > > > >>>>>>>>>>>
> > > > >
> > > > >>>>>>>>>>>
> > > >
> > > > >>>>>>>>>>> On 2023/07/26 23:20:07 "C. Scott Andreas" wrote:
> > > > >
> > > > >>>>>>>>>>>
> > > >
> > > > >>>>>>>>>>> > I agree that it would be ideal for Cassandra to have a
> > > repair
> > > > scheduler in-DB.
> > > > >
> > > > >>>>>>>>>>>
> > > >
> > > > >>>>>>>>>>> >
> > > > >
> > > > >>>>>>>>>>>
> > > >
> > > > >>>>>>>>>>> > That said I would happily support an effort to bring
> repair
> > > > scheduling to the sidecar immediately. This has nothing blocking it,
> and
> > > would
> > > > potentially enable the sidecar to provide an official repair
> scheduling
> > > > solution that is compatible with current or even previous versions of
> the
> > > > database.
> > > > >
> > > > >>>>>>>>>>>
> > > >
> > > > >>>>>>>>>>> >
> > > > >
> > > > >>>>>>>>>>>
> > > >
> > > > >>>>>>>>>>> > Once TCM has landed, we’ll have much stronger
> primitives
> > > for
> > > > repair orchestration in the database itself. But I don’t think that
> > > should
> > > > block progress on a repair scheduling solution in the sidecar, and
> there
> > > is
> > > > nothing that would prevent someone from continuing to use a sidecar-
> based
> > > > solution in perpetuity if they preferred.
> > > > >
> > > > >>>>>>>>>>>
> > > >
> > > > >>>>>>>>>>> >
> > > > >
> > > > >>>>>>>>>>>
> > > >
> > > > >>>>>>>>>>> > \\- Scott
> > > > >
> > > > >>>>>>>>>>>
> > > >
> > > > >>>>>>>>>>> >
> > > > >
> > > > >>>>>>>>>>>
> > > >
> > > > >>>>>>>>>>> > > On Jul 26, 2023, at 3:25 PM, Jon Haddad
> > > >
> <[[rustyrazorblade@apache.org](mailto:rustyrazorblade@apache.org)](mailto:[rustyrazorblade@apache.org](mailto:rustyrazorblade@apache.org))>
> > > wrote:
> > > > >
> > > > >>>>>>>>>>>
> > > >
> > > > >>>>>>>>>>> > >
> > > > >
> > > > >>>>>>>>>>>
> > > >
> > > > >>>>>>>>>>> > > I'm 100% in favor of repair being part of the core
> DB,
> > > not
> > > > the sidecar. The current (and past) state of things where running
> the DB
> > > > correctly *requires* running a separate process (either community
> > > maintained
> > > > or official C* sidecar) is incredibly painful for folks. The idea
> that
> > > your
> > > > data integrity needs to be opt-in has never made sense to me from the
> > > > perspective of either the product or the end user.
> > > > >
> > > > >>>>>>>>>>>
> > > >
> > > > >>>>>>>>>>> > >
> > > > >
> > > > >>>>>>>>>>>
> > > >
> > > > >>>>>>>>>>> > > I've worked with way too many teams that have either
> > > > configured this incorrectly or not at all.
> > > > >
> > > > >>>>>>>>>>>
> > > >
> > > > >>>>>>>>>>> > >
> > > > >
> > > > >>>>>>>>>>>
> > > >
> > > > >>>>>>>>>>> > > Ideally Cassandra would ship with repair built in and
> on
> > > by
> > > > default. Power users can disable if they want to continue to
> maintain
> > > their
> > > > own repair tooling for some reason.
> > > > >
> > > > >>>>>>>>>>>
> > > >
> > > > >>>>>>>>>>> > >
> > > > >
> > > > >>>>>>>>>>>
> > > >
> > > > >>>>>>>>>>> > > Jon
> > > > >
> > > > >>>>>>>>>>>
> > > >
> > > > >>>>>>>>>>> > >
> > > > >
> > > > >>>>>>>>>>>
> > > >
> > > > >>>>>>>>>>> > >> On 2023/07/24 20:44:14 German Eichberger via dev
> > > wrote:
> > > > >
> > > > >>>>>>>>>>>
> > > >
> > > > >>>>>>>>>>> > >> All,
> > > > >
> > > > >>>>>>>>>>>
> > > >
> > > > >>>>>>>>>>> > >> We had a brief discussion in [2] about the Uber
> article
> > > [1]
> > > > where they talk about having integrated repair into Cassandra and how
> > > great
> > > > that is. I expressed my disappointment that they didn't work with the
> > > > community on that (Uber, if you are listening time to make amends 🙂)
> > > and it
> > > > turns out Joey already had the idea and wrote the code [3] - so I
> wanted
> > > to
> > > > start a discussion to gauge interest and maybe how to revive that
> > > effort.
> > > > >
> > > > >>>>>>>>>>>
> > > >
> > > > >>>>>>>>>>> > >> Thanks,
> > > > >
> > > > >>>>>>>>>>>
> > > >
> > > > >>>>>>>>>>> > >> German
> > > > >
> > > > >>>>>>>>>>>
> > > >
> > > > >>>>>>>>>>> > >> [1] <
> > > <https://www.uber.com/blog/how-uber-optimized-cassandra->
> > > > operations-at-scale/>
> > > > >
> > > > >>>>>>>>>>>
> > > >
> > > > >>>>>>>>>>> > >> [2] <<https://the->
> > > >
> [asf.slack.com/archives/CK23JSY2K/p1690225062383619](http://asf.slack.com/archives/CK23JSY2K/p1690225062383619)>
> > > > >
> > > > >>>>>>>>>>>
> > > >
> > > > >>>>>>>>>>> > >> [3] <
> > > <https://issues.apache.org/jira/browse/CASSANDRA-14346>>
> > > > >
> > > > >>>>>>>>>>>
> > > >
> > > > >>>>>>>>>>> >
> > > > >
> > > > >>>>>>>>>
> > > >
> > > > >>>>>>>>>
> > > > >
> > > > >>>>>>
> > > >
> > > > >>>>>>
> > > > >
> > > >
> > > >
> > >
> >
>
Re: [Discuss] Repair inside C*
Posted by Jordan West <jw...@apache.org>.
Agreed with the sentiment that decomposition is a good target but out of
scope here. I’m personally excited to see an in-tree repair scheduler and
am supportive of the approach shared here.
Jordan
On Tue, Oct 22, 2024 at 08:12 Dinesh Joshi <dj...@apache.org> wrote:
> Decomposing Cassandra may be architecturally desirable but that is not the
> goal of this CEP. This CEP brings value to operators today so it should be
> considered on that merit. We definitely need to have a separate
> conversation on Cassandra's architectural direction.
>
> On Tue, Oct 22, 2024 at 7:51 AM Joseph Lynch <jo...@gmail.com>
> wrote:
>
>> Definitely like this in C* itself. We only changed our proposal to
>> putting repair scheduling in the sidecar before because trunk was frozen
>> for the foreseeable future at that time. With trunk unfrozen and
>> development on the main process going at a fast pace I think it makes way
>> more sense to integrate natively as table properties as this CEP proposes.
>> Completely agree the scheduling overhead should be minimal.
>>
>> Moving the actual repair operation (comparing data and streaming
>> mismatches) along with compaction operations to a separate process long
>> term makes a lot of sense but imo only once we both have a release of
>> sidecar and a contract figured out between them on communication. I'm
>> watching CEP-38 there as I think CQL and virtual tables are looking much
>> stronger than when we wrote CEP-1 and chose HTTP but that's for that
>> discussion and not this one.
>>
>> -Joey
>>
>> On Mon, Oct 21, 2024 at 3:25 PM Francisco Guerrero <fr...@apache.org>
>> wrote:
>>
>>> Like others have said, I was expecting the scheduling portion of repair
>>> is
>>> negligible. I was mostly curious if you had something handy that you can
>>> quickly share.
>>>
>>> On 2024/10/21 18:59:41 Jaydeep Chovatia wrote:
>>> > >Jaydeep, do you have any metrics on your clusters comparing them
>>> before
>>> > and after introducing repair scheduling into the Cassandra process?
>>> >
>>> > Yes, I had made some comparisons when I started rolling this feature
>>> out to
>>> > our production five years ago :) Here are the details:
>>> > *The Scheduling*
>>> > The scheduling itself is exceptionally lightweight, as only one
>>> additional
>>> > thread monitors the repair activity, updating the status to a system
>>> table
>>> > once every few minutes or so. So, it does not appear anywhere in the
>>> CPU
>>> > charts, etc. Unfortunately, I do not have those graphs now, but I can
>>> do a
>>> > quick comparison if it helps!
>>> >
>>> > *The Repair Itself*
>>> > As we all know, the Cassandra repair algorithm is a heavy-weight
>>> process
>>> > due to Merkle tree/streaming, etc., no matter how we schedule it. But
>>> it is
>>> > an orthogonal topic and folks are already discussing creating a new
>>> CEP.
>>> >
>>> > Jaydeep
>>> >
>>> >
>>> > On Mon, Oct 21, 2024 at 10:02 AM Francisco Guerrero <
>>> frankgh@apache.org>
>>> > wrote:
>>> >
>>> > > Jaydeep, do you have any metrics on your clusters comparing them
>>> before
>>> > > and after introducing repair scheduling into the Cassandra process?
>>> > >
>>> > > On 2024/10/21 16:57:57 "J. D. Jordan" wrote:
>>> > > > Sounds good. Just wanted to bring it up. I agree that the
>>> scheduling bit
>>> > > is
>>> > > > pretty light weight and the ideal would be to bring the whole of
>>> the
>>> > > repair
>>> > > > external, which is a much bigger can of worms to open.
>>> > > >
>>> > > >
>>> > > >
>>> > > > -Jeremiah
>>> > > >
>>> > > >
>>> > > >
>>> > > > > On Oct 21, 2024, at 11:21 AM, Chris Lohfink <
>>> clohfink85@gmail.com>
>>> > > wrote:
>>> > > > >
>>> > > > >
>>> > > >
>>> > > > >
>>> > > > >
>>> > > > > > I actually think we should be looking at how we can move
>>> things out
>>> > > of the
>>> > > > > database process.
>>> > > > >
>>> > > > >
>>> > > > >
>>> > > > >
>>> > > > >
>>> > > > > While worth pursuing, I think we would need a different CEP just
>>> to
>>> > > figure
>>> > > > > out how to do that. Not only is there a lot of infrastructure
>>> > > difficulty in
>>> > > > > running multi process, the inter app communication needs to be
>>> figured
>>> > > out
>>> > > > > better then JMX. Even the sidecar we dont have a solid story on
>>> how to
>>> > > > > ensure both are running or anything yet. It's up to each app
>>> owner to
>>> > > figure
>>> > > > > it out. Once we have a good thing in place I think we can start
>>> moving
>>> > > > > compactions, repairs, etc out of the database. Even then it's the
>>> > > _repairs_
>>> > > > > that is expensive, not the scheduling.
>>> > > > >
>>> > > > >
>>> > > > >
>>> > > > >
>>> > > > > On Mon, Oct 21, 2024 at 9:45 AM Jeremiah Jordan
>>> > > > > <[jeremiah.jordan@gmail.com](mailto:jeremiah.jordan@gmail.com)>
>>> > > wrote:
>>> > > > >
>>> > > > >
>>> > > >
>>> > > > >> I love the idea of a repair service being there by default for
>>> an
>>> > > install
>>> > > > of C*. My main concern here is that it is putting more services
>>> into
>>> > > the main
>>> > > > database process. I actually think we should be looking at how we
>>> can
>>> > > move
>>> > > > things out of the database process. The C* process being a giant
>>> > > monolith has
>>> > > > always been a pain point. Is there anyway it makes sense for this
>>> to be
>>> > > an
>>> > > > external process rather than a new thread pool inside the C*
>>> process?
>>> > > >
>>> > > > >>
>>> > > >
>>> > > > >>
>>> > > > >
>>> > > > >>
>>> > > >
>>> > > > >> -Jeremiah Jordan
>>> > > >
>>> > > > >>
>>> > > >
>>> > > > >>
>>> > > > >
>>> > > > >>
>>> > > >
>>> > > > >> On Oct 18, 2024 at 2:58:15 PM, Mick Semb Wever
>>> > > > <[mck@apache.org](mailto:mck@apache.org)> wrote:
>>> > > > >
>>> > > > >>
>>> > > >
>>> > > > >>>
>>> > > > >
>>> > > > >>>
>>> > > >
>>> > > > >>> This is looking strong, thanks Jaydeep.
>>> > > >
>>> > > > >>>
>>> > > >
>>> > > > >>>
>>> > > > >
>>> > > > >>>
>>> > > >
>>> > > > >>> I would suggest folk take a look at the design doc and the PR
>>> in the
>>> > > CEP.
>>> > > > A lot is there (that I have completely missed).
>>> > > >
>>> > > > >>>
>>> > > >
>>> > > > >>>
>>> > > > >
>>> > > > >>>
>>> > > >
>>> > > > >>> I would especially ask all authors of prior art (Reaper, DSE
>>> > > nodesync,
>>> > > > ecchronos) to take a final review of the proposal
>>> > > > >
>>> > > > >>>
>>> > > >
>>> > > > >>>
>>> > > > >
>>> > > > >>>
>>> > > >
>>> > > > >>> Jaydeep, can we ask for a two week window while we reach out
>>> to these
>>> > > > people ? There's a lot of prior art in this space, and it feels
>>> like
>>> > > we're in
>>> > > > a good place now where it's clear this has legs and we can use
>>> that to
>>> > > bring
>>> > > > folk in and make sure there's no remaining blindspots.
>>> > > >
>>> > > > >>>
>>> > > >
>>> > > > >>>
>>> > > > >
>>> > > > >>>
>>> > > >
>>> > > > >>>
>>> > > > >
>>> > > > >>>
>>> > > >
>>> > > > >>> On Fri, 18 Oct 2024 at 01:40, Jaydeep Chovatia
>>> > > > <[chovatia.jaydeep@gmail.com](mailto:chovatia.jaydeep@gmail.com)>
>>> > > wrote:
>>> > > > >
>>> > > > >>>
>>> > > >
>>> > > > >>>> Sorry, there is a typo in the CEP-37 link; here is the correct
>>> > > > [link](
>>> > >
>>> https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-37+Apache+Cassandra+Unified+Repair+Solution
>>> > > )
>>> > > >
>>> > > > >>>>
>>> > > >
>>> > > > >>>>
>>> > > > >
>>> > > > >>>>
>>> > > >
>>> > > > >>>>
>>> > > > >
>>> > > > >>>>
>>> > > >
>>> > > > >>>> On Thu, Oct 17, 2024 at 4:36 PM Jaydeep Chovatia
>>> > > > <[chovatia.jaydeep@gmail.com](mailto:chovatia.jaydeep@gmail.com)>
>>> > > wrote:
>>> > > > >
>>> > > > >>>>
>>> > > >
>>> > > > >>>>> First, thank you for your patience while we strengthened the
>>> > > CEP-37.
>>> > > >
>>> > > > >>>>>
>>> > > >
>>> > > > >>>>>
>>> > > > >
>>> > > > >>>>>
>>> > > >
>>> > > > >>>>> Over the last eight months, Chris Lohfink, Andy Tolbert, Josh
>>> > > McKenzie,
>>> > > > Dinesh Joshi, Kristijonas Zalys, and I have done tons of work
>>> (online
>>> > > > discussions/a dedicated Slack channel
>>> > > #cassandra-repair-scheduling-cep37) to
>>> > > > come up with the best possible design that not only significantly
>>> > > simplifies
>>> > > > repair operations but also includes the most common features that
>>> > > everyone
>>> > > > will benefit from running at Scale.
>>> > > >
>>> > > > >>>>>
>>> > > >
>>> > > > >>>>> For example,
>>> > > >
>>> > > > >>>>>
>>> > > >
>>> > > > >>>>> * Apache Cassandra must be capable of running multiple
>>> repair
>>> > > types,
>>> > > > such as Full, Incremental, Paxos, and Preview - so the framework
>>> should
>>> > > be
>>> > > > easily extendable with no additional overhead from the operator’s
>>> point
>>> > > of
>>> > > > view.
>>> > > >
>>> > > > >>>>>
>>> > > >
>>> > > > >>>>> * An easy way to extend the token-split calculation
>>> algorithm
>>> > > with a
>>> > > > default implementation should exist.
>>> > > >
>>> > > > >>>>>
>>> > > >
>>> > > > >>>>> * Running incremental repair reliably at Scale is pretty
>>> > > challenging,
>>> > > > so we need to place safeguards, such as migration/rollback w/o
>>> restart
>>> > > and
>>> > > > stopping incremental repair automatically if the disk is about to
>>> get
>>> > > full.
>>> > > >
>>> > > > >>>>>
>>> > > >
>>> > > > >>>>>
>>> > > >
>>> > > > >>>>>
>>> > > >
>>> > > > >>>>> We are glad to inform you that CEP-37 (i.e., Repair inside
>>> > > Cassandra) is
>>> > > > now officially ready for review after multiple rounds of design,
>>> > > testing, code
>>> > > > reviews, documentation reviews, and, more importantly, validation
>>> that
>>> > > it runs
>>> > > > at Scale!
>>> > > >
>>> > > > >>>>>
>>> > > >
>>> > > > >>>>>
>>> > > > >
>>> > > > >>>>>
>>> > > >
>>> > > > >>>>> Some facts about CEP-37.
>>> > > >
>>> > > > >>>>>
>>> > > >
>>> > > > >>>>> * Multiple members have verified all aspects of CEP-37
>>> numerous
>>> > > times.
>>> > > >
>>> > > > >>>>>
>>> > > >
>>> > > > >>>>> * The design proposed in CEP-37 has been thoroughly tried
>>> and
>>> > > tested
>>> > > > on an immense scale (hundreds of unique Cassandra clusters, tens of
>>> > > thousands
>>> > > > of Cassandra nodes, with tens of millions of QPS) on top of 4.1
>>> > > open-source
>>> > > > for more than five years; please see more details[
>>> > > > here](
>>> > >
>>> https://www.uber.com/en-US/blog/how-uber-optimized-cassandra-operations-
>>> > > > at-scale/).
>>> > > >
>>> > > > >>>>>
>>> > > >
>>> > > > >>>>> * The following
>>> > > > [presentation](
>>> > >
>>> https://docs.google.com/presentation/d/1Zilww9c7LihHULk_ckErI2s4XbObxjWknKqRtbvHyZc/edit#slide=id.g30a4fd4fcf7_0_13
>>> > > )
>>> > > > highlights the rigorous applied to CEP-37, which was given during
>>> last
>>> > > week’s
>>> > > > Apache Cassandra Bay Area [Meetup](
>>> > > https://www.meetup.com/apache-cassandra-
>>> > > > bay-area/events/303469006/),
>>> > > >
>>> > > > >>>>>
>>> > > >
>>> > > > >>>>>
>>> > > >
>>> > > >
>>> > > > >
>>> > > > >>>>>
>>> > > >
>>> > > > >>>>> Since things are massively overhauled, we believe it is
>>> almost
>>> > > ready for
>>> > > > a final pass pre-VOTE. We would like you to please review the
>>> > > > [CEP-37](
>>> > >
>>> https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-37+Apache+Cassandra+Unified+Repair+Solution\
>>> <https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-37+Apache+Cassandra+Unified+Repair+Solution%5C>
>>> > > <
>>> https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-37+Apache+Cassandra+Unified+Repair+Solution%5C
>>> >
>>> > > ))
>>> > > > and the associated detailed design
>>> > > > [doc](https://docs.google.com/document/d/1CJWxjEi-
>>> > > > mBABPMZ3VWJ9w5KavWfJETAGxfUpsViPcPo/edit#heading=h.r112r46toau0).
>>> > > >
>>> > > > >>>>>
>>> > > >
>>> > > > >>>>>
>>> > > > >
>>> > > > >>>>>
>>> > > >
>>> > > > >>>>> Thank you everyone!
>>> > > >
>>> > > > >>>>>
>>> > > >
>>> > > > >>>>> Chris, Andy, Josh, Dinesh, Kristijonas, and Jaydeep
>>> > > >
>>> > > > >>>>>
>>> > > >
>>> > > > >>>>>
>>> > > > >
>>> > > > >
>>> > > > >>>>>
>>> > > >
>>> > > > >>>>>
>>> > > > >
>>> > > > >>>>>
>>> > > >
>>> > > > >>>>> On Thu, Sep 19, 2024 at 11:26 AM Josh McKenzie
>>> > > > <[jmckenzie@apache.org](mailto:jmckenzie@apache.org)> wrote:
>>> > > > >
>>> > > > >>>>>
>>> > > >
>>> > > > >>>>>> __
>>> > > >
>>> > > > >>>>>>
>>> > > >
>>> > > > >>>>>> Not quite; finishing touches on the CEP and design doc are
>>> in
>>> > > flight
>>> > > > (as of last / this week).
>>> > > > >
>>> > > > >>>>>>
>>> > > >
>>> > > > >>>>>>
>>> > > > >
>>> > > > >>>>>>
>>> > > >
>>> > > > >>>>>> Soon(tm).
>>> > > >
>>> > > > >>>>>>
>>> > > >
>>> > > > >>>>>>
>>> > > > >
>>> > > > >>>>>>
>>> > > >
>>> > > > >>>>>> On Thu, Sep 19, 2024, at 2:07 PM, Patrick McFadin wrote:
>>> > > > >
>>> > > > >>>>>>
>>> > > >
>>> > > > >>>>>>> Is this CEP ready for a VOTE thread?
>>> > > > <
>>> > >
>>> https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-37+%28DRAFT%29+Apache+Cassandra+Unified+Repair+Solution
>>> >
>>> > >
>>> > > > >
>>> > > > >>>>>>>
>>> > > >
>>> > > > >>>>>>>
>>> > > > >
>>> > > > >>>>>>>
>>> > > >
>>> > > > >>>>>>> On Sun, Feb 25, 2024 at 12:25 PM Jaydeep Chovatia
>>> > > > <[chovatia.jaydeep@gmail.com](mailto:chovatia.jaydeep@gmail.com)>
>>> > > wrote:
>>> > > > >
>>> > > > >>>>>>>
>>> > > >
>>> > > > >>>>>>>> Thanks, Josh. I've just updated the
>>> > > > [CEP](
>>> > >
>>> https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-37+%28DRAFT%29+Apache+Cassandra+Official+Repair+Solution
>>> > > )
>>> > > > and included all the solutions you mentioned below.
>>> > > > >
>>> > > > >>>>>>>>
>>> > > >
>>> > > > >>>>>>>>
>>> > > > >
>>> > > > >>>>>>>>
>>> > > >
>>> > > > >>>>>>>> Jaydeep
>>> > > > >
>>> > > > >>>>>>>>
>>> > > >
>>> > > > >>>>>>>>
>>> > > > >
>>> > > > >>>>>>>>
>>> > > >
>>> > > > >>>>>>>> On Thu, Feb 22, 2024 at 9:33 AM Josh McKenzie
>>> > > > <[jmckenzie@apache.org](mailto:jmckenzie@apache.org)> wrote:
>>> > > > >
>>> > > > >>>>>>>>
>>> > > >
>>> > > > >>>>>>>>> __
>>> > > > >
>>> > > > >>>>>>>>>
>>> > > >
>>> > > > >>>>>>>>> Very late response from me here (basically necro'ing this
>>> > > thread).
>>> > > > >
>>> > > > >>>>>>>>>
>>> > > >
>>> > > > >>>>>>>>>
>>> > > > >
>>> > > > >>>>>>>>>
>>> > > >
>>> > > > >>>>>>>>> I think it'd be useful to get this condensed into a CEP
>>> that
>>> > > we can
>>> > > > then discuss in that format. It's clearly something we all agree
>>> we need
>>> > > and
>>> > > > having an implementation that works, even if it's not in your
>>> preferred
>>> > > > execution domain, is vastly better than nothing IMO.
>>> > > > >
>>> > > > >>>>>>>>>
>>> > > >
>>> > > > >>>>>>>>>
>>> > > > >
>>> > > > >>>>>>>>>
>>> > > >
>>> > > > >>>>>>>>> I don't have cycles (nor background ;) ) to do that, but
>>> it
>>> > > sounds
>>> > > > like you do Jaydeep given the implementation you have on a private
>>> fork +
>>> > > > design.
>>> > > > >
>>> > > > >>>>>>>>>
>>> > > >
>>> > > > >>>>>>>>>
>>> > > > >
>>> > > > >>>>>>>>>
>>> > > >
>>> > > > >>>>>>>>> A non-exhaustive list of things that might be useful
>>> > > incorporating
>>> > > > into or referencing from a CEP:
>>> > > > >
>>> > > > >>>>>>>>>
>>> > > >
>>> > > > >>>>>>>>> Slack thread: <https://the-
>>> > > > asf.slack.com/archives/CK23JSY2K/p1690225062383619>
>>> > > > >
>>> > > > >>>>>>>>>
>>> > > >
>>> > > > >>>>>>>>> Joey's old C* ticket:
>>> > > > <https://issues.apache.org/jira/browse/CASSANDRA-14346>
>>> > > > >
>>> > > > >>>>>>>>>
>>> > > >
>>> > > > >>>>>>>>> Even older automatic repair scheduling:
>>> > > > <https://issues.apache.org/jira/browse/CASSANDRA-10070>
>>> > > > >
>>> > > > >>>>>>>>>
>>> > > >
>>> > > > >>>>>>>>> Your design gdoc: <
>>> > > https://docs.google.com/document/d/1CJWxjEi-
>>> > > > mBABPMZ3VWJ9w5KavWfJETAGxfUpsViPcPo/edit#heading=h.r112r46toau0>
>>> > > > >
>>> > > > >>>>>>>>>
>>> > > >
>>> > > > >>>>>>>>> PR with automated repair:
>>> > > > <
>>> > >
>>> https://github.com/jaydeepkumar1984/cassandra/commit/ef6456d652c0d07cf29d88dfea03b73704814c2c
>>> >
>>> > >
>>> > > > >
>>> > > > >>>>>>>>>
>>> > > >
>>> > > > >>>>>>>>>
>>> > > > >
>>> > > > >>>>>>>>>
>>> > > >
>>> > > > >>>>>>>>> My intuition is that we're all basically in agreement
>>> that
>>> > > this is
>>> > > > something the DB needs, we're all willing to bikeshed for our
>>> personal
>>> > > > preference on where it lives and how it's implemented, and at the
>>> end of
>>> > > the
>>> > > > day, code talks. I don't think anyone's said they'll die on the
>>> hill of
>>> > > > implementation details, so that feels like CEP time to me.
>>> > > > >
>>> > > > >>>>>>>>>
>>> > > >
>>> > > > >>>>>>>>>
>>> > > > >
>>> > > > >>>>>>>>>
>>> > > >
>>> > > > >>>>>>>>> If you were willing and able to get a CEP together for
>>> > > automated
>>> > > > repair based on the above material, given you've done the work and
>>> have
>>> > > the
>>> > > > proof points it's working at scale, I think this would be a _huge
>>> > > > contribution_ to the community.
>>> > > > >
>>> > > > >>>>>>>>>
>>> > > >
>>> > > > >>>>>>>>>
>>> > > > >
>>> > > > >>>>>>>>>
>>> > > >
>>> > > > >>>>>>>>> On Thu, Aug 24, 2023, at 7:26 PM, Jaydeep Chovatia wrote:
>>> > > > >
>>> > > > >>>>>>>>>
>>> > > >
>>> > > > >>>>>>>>>> Is anyone going to file an official CEP for this?
>>> > > > >
>>> > > > >>>>>>>>>>
>>> > > >
>>> > > > >>>>>>>>>> As mentioned in this email thread, here is one of the
>>> > > solution's
>>> > > > [design doc](https://docs.google.com/document/d/1CJWxjEi-
>>> > > > mBABPMZ3VWJ9w5KavWfJETAGxfUpsViPcPo/edit#heading=h.r112r46toau0)
>>> and
>>> > > source
>>> > > > code on a private Apache Cassandra patch. Could you go through it
>>> and
>>> > > let me
>>> > > > know what you think?
>>> > > > >
>>> > > > >>>>>>>>>>
>>> > > >
>>> > > > >>>>>>>>>>
>>> > > > >
>>> > > > >>>>>>>>>>
>>> > > >
>>> > > > >>>>>>>>>> Jaydeep
>>> > > > >
>>> > > > >>>>>>>>>>
>>> > > >
>>> > > > >>>>>>>>>>
>>> > > > >
>>> > > > >>>>>>>>>>
>>> > > >
>>> > > > >>>>>>>>>> On Wed, Aug 2, 2023 at 3:54 PM Jon Haddad
>>> > > > <[rustyrazorblade@apache.org](mailto:rustyrazorblade@apache.org)>
>>> > > wrote:
>>> > > > >
>>> > > > >>>>>>>>>>
>>> > > >
>>> > > > >>>>>>>>>>> > That said I would happily support an effort to bring
>>> repair
>>> > > > scheduling to the sidecar immediately. This has nothing blocking
>>> it, and
>>> > > would
>>> > > > potentially enable the sidecar to provide an official repair
>>> scheduling
>>> > > > solution that is compatible with current or even previous versions
>>> of the
>>> > > > database.
>>> > > > >
>>> > > > >>>>>>>>>>>
>>> > > >
>>> > > > >>>>>>>>>>>
>>> > > > >
>>> > > > >>>>>>>>>>>
>>> > > >
>>> > > > >>>>>>>>>>> This is something I hadn't thought much about, and is a
>>> > > pretty
>>> > > > good argument for using the sidecar initially. There's a lot of
>>> > > deployments
>>> > > > out there and having an official repair option would be a big win.
>>> > > > >
>>> > > > >>>>>>>>>>>
>>> > > >
>>> > > > >>>>>>>>>>>
>>> > > > >
>>> > > > >>>>>>>>>>>
>>> > > >
>>> > > > >>>>>>>>>>>
>>> > > > >
>>> > > > >>>>>>>>>>>
>>> > > >
>>> > > > >>>>>>>>>>> On 2023/07/26 23:20:07 "C. Scott Andreas" wrote:
>>> > > > >
>>> > > > >>>>>>>>>>>
>>> > > >
>>> > > > >>>>>>>>>>> > I agree that it would be ideal for Cassandra to have
>>> a
>>> > > repair
>>> > > > scheduler in-DB.
>>> > > > >
>>> > > > >>>>>>>>>>>
>>> > > >
>>> > > > >>>>>>>>>>> >
>>> > > > >
>>> > > > >>>>>>>>>>>
>>> > > >
>>> > > > >>>>>>>>>>> > That said I would happily support an effort to bring
>>> repair
>>> > > > scheduling to the sidecar immediately. This has nothing blocking
>>> it, and
>>> > > would
>>> > > > potentially enable the sidecar to provide an official repair
>>> scheduling
>>> > > > solution that is compatible with current or even previous versions
>>> of the
>>> > > > database.
>>> > > > >
>>> > > > >>>>>>>>>>>
>>> > > >
>>> > > > >>>>>>>>>>> >
>>> > > > >
>>> > > > >>>>>>>>>>>
>>> > > >
>>> > > > >>>>>>>>>>> > Once TCM has landed, we’ll have much stronger
>>> primitives
>>> > > for
>>> > > > repair orchestration in the database itself. But I don’t think that
>>> > > should
>>> > > > block progress on a repair scheduling solution in the sidecar, and
>>> there
>>> > > is
>>> > > > nothing that would prevent someone from continuing to use a
>>> sidecar-based
>>> > > > solution in perpetuity if they preferred.
>>> > > > >
>>> > > > >>>>>>>>>>>
>>> > > >
>>> > > > >>>>>>>>>>> >
>>> > > > >
>>> > > > >>>>>>>>>>>
>>> > > >
>>> > > > >>>>>>>>>>> > \- Scott
>>> > > > >
>>> > > > >>>>>>>>>>>
>>> > > >
>>> > > > >>>>>>>>>>> >
>>> > > > >
>>> > > > >>>>>>>>>>>
>>> > > >
>>> > > > >>>>>>>>>>> > > On Jul 26, 2023, at 3:25 PM, Jon Haddad
>>> > > > <[rustyrazorblade@apache.org](mailto:rustyrazorblade@apache.org)>
>>> > > wrote:
>>> > > > >
>>> > > > >>>>>>>>>>>
>>> > > >
>>> > > > >>>>>>>>>>> > >
>>> > > > >
>>> > > > >>>>>>>>>>>
>>> > > >
>>> > > > >>>>>>>>>>> > > I'm 100% in favor of repair being part of the
>>> core DB,
>>> > > not
>>> > > > the sidecar. The current (and past) state of things where running
>>> the DB
>>> > > > correctly *requires* running a separate process (either community
>>> > > maintained
>>> > > > or official C* sidecar) is incredibly painful for folks. The idea
>>> that
>>> > > your
>>> > > > data integrity needs to be opt-in has never made sense to me from
>>> the
>>> > > > perspective of either the product or the end user.
>>> > > > >
>>> > > > >>>>>>>>>>>
>>> > > >
>>> > > > >>>>>>>>>>> > >
>>> > > > >
>>> > > > >>>>>>>>>>>
>>> > > >
>>> > > > >>>>>>>>>>> > > I've worked with way too many teams that have
>>> either
>>> > > > configured this incorrectly or not at all.
>>> > > > >
>>> > > > >>>>>>>>>>>
>>> > > >
>>> > > > >>>>>>>>>>> > >
>>> > > > >
>>> > > > >>>>>>>>>>>
>>> > > >
>>> > > > >>>>>>>>>>> > > Ideally Cassandra would ship with repair built in
>>> and on
>>> > > by
>>> > > > default. Power users can disable if they want to continue to
>>> maintain
>>> > > their
>>> > > > own repair tooling for some reason.
>>> > > > >
>>> > > > >>>>>>>>>>>
>>> > > >
>>> > > > >>>>>>>>>>> > >
>>> > > > >
>>> > > > >>>>>>>>>>>
>>> > > >
>>> > > > >>>>>>>>>>> > > Jon
>>> > > > >
>>> > > > >>>>>>>>>>>
>>> > > >
>>> > > > >>>>>>>>>>> > >
>>> > > > >
>>> > > > >>>>>>>>>>>
>>> > > >
>>> > > > >>>>>>>>>>> > >> On 2023/07/24 20:44:14 German Eichberger via dev
>>> > > wrote:
>>> > > > >
>>> > > > >>>>>>>>>>>
>>> > > >
>>> > > > >>>>>>>>>>> > >> All,
>>> > > > >
>>> > > > >>>>>>>>>>>
>>> > > >
>>> > > > >>>>>>>>>>> > >> We had a brief discussion in [2] about the Uber
>>> article
>>> > > [1]
>>> > > > where they talk about having integrated repair into Cassandra and
>>> how
>>> > > great
>>> > > > that is. I expressed my disappointment that they didn't work with
>>> the
>>> > > > community on that (Uber, if you are listening time to make amends
>>> 🙂)
>>> > > and it
>>> > > > turns out Joey already had the idea and wrote the code [3] - so I
>>> wanted
>>> > > to
>>> > > > start a discussion to gauge interest and maybe how to revive that
>>> > > effort.
>>> > > > >
>>> > > > >>>>>>>>>>>
>>> > > >
>>> > > > >>>>>>>>>>> > >> Thanks,
>>> > > > >
>>> > > > >>>>>>>>>>>
>>> > > >
>>> > > > >>>>>>>>>>> > >> German
>>> > > > >
>>> > > > >>>>>>>>>>>
>>> > > >
>>> > > > >>>>>>>>>>> > >> [1] <
>>> > > https://www.uber.com/blog/how-uber-optimized-cassandra-
>>> > > > operations-at-scale/>
>>> > > > >
>>> > > > >>>>>>>>>>>
>>> > > >
>>> > > > >>>>>>>>>>> > >> [2] <https://the-
>>> > > > asf.slack.com/archives/CK23JSY2K/p1690225062383619>
>>> > > > >
>>> > > > >>>>>>>>>>>
>>> > > >
>>> > > > >>>>>>>>>>> > >> [3] <
>>> > > https://issues.apache.org/jira/browse/CASSANDRA-14346>
>>> > > > >
>>> > > > >>>>>>>>>>>
>>> > > >
>>> > > > >>>>>>>>>>> >
>>> > > > >
>>> > > > >>>>>>>>>
>>> > > >
>>> > > > >>>>>>>>>
>>> > > > >
>>> > > > >>>>>>
>>> > > >
>>> > > > >>>>>>
>>> > > > >
>>> > > >
>>> > > >
>>> > >
>>> >
>>>
>>
Re: [Discuss] Repair inside C*
Posted by Dinesh Joshi <dj...@apache.org>.
Decomposing Cassandra may be architecturally desirable but that is not the
goal of this CEP. This CEP brings value to operators today so it should be
considered on that merit. We definitely need to have a separate
conversation on Cassandra's architectural direction.
On Tue, Oct 22, 2024 at 7:51 AM Joseph Lynch <jo...@gmail.com> wrote:
> Definitely like this in C* itself. We only changed our proposal to putting
> repair scheduling in the sidecar before because trunk was frozen for the
> foreseeable future at that time. With trunk unfrozen and development on the
> main process going at a fast pace I think it makes way more sense to
> integrate natively as table properties as this CEP proposes. Completely
> agree the scheduling overhead should be minimal.
>
> Moving the actual repair operation (comparing data and streaming
> mismatches) along with compaction operations to a separate process long
> term makes a lot of sense but imo only once we both have a release of
> sidecar and a contract figured out between them on communication. I'm
> watching CEP-38 there as I think CQL and virtual tables are looking much
> stronger than when we wrote CEP-1 and chose HTTP but that's for that
> discussion and not this one.
>
> -Joey
>
> On Mon, Oct 21, 2024 at 3:25 PM Francisco Guerrero <fr...@apache.org>
> wrote:
>
>> Like others have said, I was expecting the scheduling portion of repair is
>> negligible. I was mostly curious if you had something handy that you can
>> quickly share.
>>
>> On 2024/10/21 18:59:41 Jaydeep Chovatia wrote:
>> > >Jaydeep, do you have any metrics on your clusters comparing them before
>> > and after introducing repair scheduling into the Cassandra process?
>> >
>> > Yes, I had made some comparisons when I started rolling this feature
>> out to
>> > our production five years ago :) Here are the details:
>> > *The Scheduling*
>> > The scheduling itself is exceptionally lightweight, as only one
>> additional
>> > thread monitors the repair activity, updating the status to a system
>> table
>> > once every few minutes or so. So, it does not appear anywhere in the CPU
>> > charts, etc. Unfortunately, I do not have those graphs now, but I can
>> do a
>> > quick comparison if it helps!
>> >
>> > *The Repair Itself*
>> > As we all know, the Cassandra repair algorithm is a heavy-weight process
>> > due to Merkle tree/streaming, etc., no matter how we schedule it. But
>> it is
>> > an orthogonal topic and folks are already discussing creating a new CEP.
>> >
>> > Jaydeep
>> >
>> >
>> > On Mon, Oct 21, 2024 at 10:02 AM Francisco Guerrero <frankgh@apache.org
>> >
>> > wrote:
>> >
>> > > Jaydeep, do you have any metrics on your clusters comparing them
>> before
>> > > and after introducing repair scheduling into the Cassandra process?
>> > >
>> > > On 2024/10/21 16:57:57 "J. D. Jordan" wrote:
>> > > > Sounds good. Just wanted to bring it up. I agree that the
>> scheduling bit
>> > > is
>> > > > pretty light weight and the ideal would be to bring the whole of the
>> > > repair
>> > > > external, which is a much bigger can of worms to open.
>> > > >
>> > > >
>> > > >
>> > > > -Jeremiah
>> > > >
>> > > >
>> > > >
>> > > > > On Oct 21, 2024, at 11:21 AM, Chris Lohfink <clohfink85@gmail.com
>> >
>> > > wrote:
>> > > > >
>> > > > >
>> > > >
>> > > > >
>> > > > >
>> > > > > > I actually think we should be looking at how we can move things
>> out
>> > > of the
>> > > > > database process.
>> > > > >
>> > > > >
>> > > > >
>> > > > >
>> > > > >
>> > > > > While worth pursuing, I think we would need a different CEP just
>> to
>> > > figure
>> > > > > out how to do that. Not only is there a lot of infrastructure
>> > > difficulty in
>> > > > > running multi process, the inter app communication needs to be
>> figured
>> > > out
>> > > > > better then JMX. Even the sidecar we dont have a solid story on
>> how to
>> > > > > ensure both are running or anything yet. It's up to each app
>> owner to
>> > > figure
>> > > > > it out. Once we have a good thing in place I think we can start
>> moving
>> > > > > compactions, repairs, etc out of the database. Even then it's the
>> > > _repairs_
>> > > > > that is expensive, not the scheduling.
>> > > > >
>> > > > >
>> > > > >
>> > > > >
>> > > > > On Mon, Oct 21, 2024 at 9:45 AM Jeremiah Jordan
>> > > > > <[jeremiah.jordan@gmail.com](mailto:jeremiah.jordan@gmail.com)>
>> > > wrote:
>> > > > >
>> > > > >
>> > > >
>> > > > >> I love the idea of a repair service being there by default for an
>> > > install
>> > > > of C*. My main concern here is that it is putting more services
>> into
>> > > the main
>> > > > database process. I actually think we should be looking at how we
>> can
>> > > move
>> > > > things out of the database process. The C* process being a giant
>> > > monolith has
>> > > > always been a pain point. Is there anyway it makes sense for this
>> to be
>> > > an
>> > > > external process rather than a new thread pool inside the C*
>> process?
>> > > >
>> > > > >>
>> > > >
>> > > > >>
>> > > > >
>> > > > >>
>> > > >
>> > > > >> -Jeremiah Jordan
>> > > >
>> > > > >>
>> > > >
>> > > > >>
>> > > > >
>> > > > >>
>> > > >
>> > > > >> On Oct 18, 2024 at 2:58:15 PM, Mick Semb Wever
>> > > > <[mck@apache.org](mailto:mck@apache.org)> wrote:
>> > > > >
>> > > > >>
>> > > >
>> > > > >>>
>> > > > >
>> > > > >>>
>> > > >
>> > > > >>> This is looking strong, thanks Jaydeep.
>> > > >
>> > > > >>>
>> > > >
>> > > > >>>
>> > > > >
>> > > > >>>
>> > > >
>> > > > >>> I would suggest folk take a look at the design doc and the PR
>> in the
>> > > CEP.
>> > > > A lot is there (that I have completely missed).
>> > > >
>> > > > >>>
>> > > >
>> > > > >>>
>> > > > >
>> > > > >>>
>> > > >
>> > > > >>> I would especially ask all authors of prior art (Reaper, DSE
>> > > nodesync,
>> > > > ecchronos) to take a final review of the proposal
>> > > > >
>> > > > >>>
>> > > >
>> > > > >>>
>> > > > >
>> > > > >>>
>> > > >
>> > > > >>> Jaydeep, can we ask for a two week window while we reach out to
>> these
>> > > > people ? There's a lot of prior art in this space, and it feels
>> like
>> > > we're in
>> > > > a good place now where it's clear this has legs and we can use that
>> to
>> > > bring
>> > > > folk in and make sure there's no remaining blindspots.
>> > > >
>> > > > >>>
>> > > >
>> > > > >>>
>> > > > >
>> > > > >>>
>> > > >
>> > > > >>>
>> > > > >
>> > > > >>>
>> > > >
>> > > > >>> On Fri, 18 Oct 2024 at 01:40, Jaydeep Chovatia
>> > > > <[chovatia.jaydeep@gmail.com](mailto:chovatia.jaydeep@gmail.com)>
>> > > wrote:
>> > > > >
>> > > > >>>
>> > > >
>> > > > >>>> Sorry, there is a typo in the CEP-37 link; here is the correct
>> > > > [link](
>> > >
>> https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-37+Apache+Cassandra+Unified+Repair+Solution
>> > > )
>> > > >
>> > > > >>>>
>> > > >
>> > > > >>>>
>> > > > >
>> > > > >>>>
>> > > >
>> > > > >>>>
>> > > > >
>> > > > >>>>
>> > > >
>> > > > >>>> On Thu, Oct 17, 2024 at 4:36 PM Jaydeep Chovatia
>> > > > <[chovatia.jaydeep@gmail.com](mailto:chovatia.jaydeep@gmail.com)>
>> > > wrote:
>> > > > >
>> > > > >>>>
>> > > >
>> > > > >>>>> First, thank you for your patience while we strengthened the
>> > > CEP-37.
>> > > >
>> > > > >>>>>
>> > > >
>> > > > >>>>>
>> > > > >
>> > > > >>>>>
>> > > >
>> > > > >>>>> Over the last eight months, Chris Lohfink, Andy Tolbert, Josh
>> > > McKenzie,
>> > > > Dinesh Joshi, Kristijonas Zalys, and I have done tons of work
>> (online
>> > > > discussions/a dedicated Slack channel
>> > > #cassandra-repair-scheduling-cep37) to
>> > > > come up with the best possible design that not only significantly
>> > > simplifies
>> > > > repair operations but also includes the most common features that
>> > > everyone
>> > > > will benefit from running at Scale.
>> > > >
>> > > > >>>>>
>> > > >
>> > > > >>>>> For example,
>> > > >
>> > > > >>>>>
>> > > >
>> > > > >>>>> * Apache Cassandra must be capable of running multiple
>> repair
>> > > types,
>> > > > such as Full, Incremental, Paxos, and Preview - so the framework
>> should
>> > > be
>> > > > easily extendable with no additional overhead from the operator’s
>> point
>> > > of
>> > > > view.
>> > > >
>> > > > >>>>>
>> > > >
>> > > > >>>>> * An easy way to extend the token-split calculation
>> algorithm
>> > > with a
>> > > > default implementation should exist.
>> > > >
>> > > > >>>>>
>> > > >
>> > > > >>>>> * Running incremental repair reliably at Scale is pretty
>> > > challenging,
>> > > > so we need to place safeguards, such as migration/rollback w/o
>> restart
>> > > and
>> > > > stopping incremental repair automatically if the disk is about to
>> get
>> > > full.
>> > > >
>> > > > >>>>>
>> > > >
>> > > > >>>>>
>> > > >
>> > > > >>>>>
>> > > >
>> > > > >>>>> We are glad to inform you that CEP-37 (i.e., Repair inside
>> > > Cassandra) is
>> > > > now officially ready for review after multiple rounds of design,
>> > > testing, code
>> > > > reviews, documentation reviews, and, more importantly, validation
>> that
>> > > it runs
>> > > > at Scale!
>> > > >
>> > > > >>>>>
>> > > >
>> > > > >>>>>
>> > > > >
>> > > > >>>>>
>> > > >
>> > > > >>>>> Some facts about CEP-37.
>> > > >
>> > > > >>>>>
>> > > >
>> > > > >>>>> * Multiple members have verified all aspects of CEP-37
>> numerous
>> > > times.
>> > > >
>> > > > >>>>>
>> > > >
>> > > > >>>>> * The design proposed in CEP-37 has been thoroughly tried
>> and
>> > > tested
>> > > > on an immense scale (hundreds of unique Cassandra clusters, tens of
>> > > thousands
>> > > > of Cassandra nodes, with tens of millions of QPS) on top of 4.1
>> > > open-source
>> > > > for more than five years; please see more details[
>> > > > here](
>> > >
>> https://www.uber.com/en-US/blog/how-uber-optimized-cassandra-operations-
>> > > > at-scale/).
>> > > >
>> > > > >>>>>
>> > > >
>> > > > >>>>> * The following
>> > > > [presentation](
>> > >
>> https://docs.google.com/presentation/d/1Zilww9c7LihHULk_ckErI2s4XbObxjWknKqRtbvHyZc/edit#slide=id.g30a4fd4fcf7_0_13
>> > > )
>> > > > highlights the rigorous applied to CEP-37, which was given during
>> last
>> > > week’s
>> > > > Apache Cassandra Bay Area [Meetup](
>> > > https://www.meetup.com/apache-cassandra-
>> > > > bay-area/events/303469006/),
>> > > >
>> > > > >>>>>
>> > > >
>> > > > >>>>>
>> > > >
>> > > >
>> > > > >
>> > > > >>>>>
>> > > >
>> > > > >>>>> Since things are massively overhauled, we believe it is almost
>> > > ready for
>> > > > a final pass pre-VOTE. We would like you to please review the
>> > > > [CEP-37](
>> > >
>> https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-37+Apache+Cassandra+Unified+Repair+Solution\
>> <https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-37+Apache+Cassandra+Unified+Repair+Solution%5C>
>> > > <
>> https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-37+Apache+Cassandra+Unified+Repair+Solution%5C
>> >
>> > > ))
>> > > > and the associated detailed design
>> > > > [doc](https://docs.google.com/document/d/1CJWxjEi-
>> > > > mBABPMZ3VWJ9w5KavWfJETAGxfUpsViPcPo/edit#heading=h.r112r46toau0).
>> > > >
>> > > > >>>>>
>> > > >
>> > > > >>>>>
>> > > > >
>> > > > >>>>>
>> > > >
>> > > > >>>>> Thank you everyone!
>> > > >
>> > > > >>>>>
>> > > >
>> > > > >>>>> Chris, Andy, Josh, Dinesh, Kristijonas, and Jaydeep
>> > > >
>> > > > >>>>>
>> > > >
>> > > > >>>>>
>> > > > >
>> > > > >
>> > > > >>>>>
>> > > >
>> > > > >>>>>
>> > > > >
>> > > > >>>>>
>> > > >
>> > > > >>>>> On Thu, Sep 19, 2024 at 11:26 AM Josh McKenzie
>> > > > <[jmckenzie@apache.org](mailto:jmckenzie@apache.org)> wrote:
>> > > > >
>> > > > >>>>>
>> > > >
>> > > > >>>>>> __
>> > > >
>> > > > >>>>>>
>> > > >
>> > > > >>>>>> Not quite; finishing touches on the CEP and design doc are in
>> > > flight
>> > > > (as of last / this week).
>> > > > >
>> > > > >>>>>>
>> > > >
>> > > > >>>>>>
>> > > > >
>> > > > >>>>>>
>> > > >
>> > > > >>>>>> Soon(tm).
>> > > >
>> > > > >>>>>>
>> > > >
>> > > > >>>>>>
>> > > > >
>> > > > >>>>>>
>> > > >
>> > > > >>>>>> On Thu, Sep 19, 2024, at 2:07 PM, Patrick McFadin wrote:
>> > > > >
>> > > > >>>>>>
>> > > >
>> > > > >>>>>>> Is this CEP ready for a VOTE thread?
>> > > > <
>> > >
>> https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-37+%28DRAFT%29+Apache+Cassandra+Unified+Repair+Solution
>> >
>> > >
>> > > > >
>> > > > >>>>>>>
>> > > >
>> > > > >>>>>>>
>> > > > >
>> > > > >>>>>>>
>> > > >
>> > > > >>>>>>> On Sun, Feb 25, 2024 at 12:25 PM Jaydeep Chovatia
>> > > > <[chovatia.jaydeep@gmail.com](mailto:chovatia.jaydeep@gmail.com)>
>> > > wrote:
>> > > > >
>> > > > >>>>>>>
>> > > >
>> > > > >>>>>>>> Thanks, Josh. I've just updated the
>> > > > [CEP](
>> > >
>> https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-37+%28DRAFT%29+Apache+Cassandra+Official+Repair+Solution
>> > > )
>> > > > and included all the solutions you mentioned below.
>> > > > >
>> > > > >>>>>>>>
>> > > >
>> > > > >>>>>>>>
>> > > > >
>> > > > >>>>>>>>
>> > > >
>> > > > >>>>>>>> Jaydeep
>> > > > >
>> > > > >>>>>>>>
>> > > >
>> > > > >>>>>>>>
>> > > > >
>> > > > >>>>>>>>
>> > > >
>> > > > >>>>>>>> On Thu, Feb 22, 2024 at 9:33 AM Josh McKenzie
>> > > > <[jmckenzie@apache.org](mailto:jmckenzie@apache.org)> wrote:
>> > > > >
>> > > > >>>>>>>>
>> > > >
>> > > > >>>>>>>>> __
>> > > > >
>> > > > >>>>>>>>>
>> > > >
>> > > > >>>>>>>>> Very late response from me here (basically necro'ing this
>> > > thread).
>> > > > >
>> > > > >>>>>>>>>
>> > > >
>> > > > >>>>>>>>>
>> > > > >
>> > > > >>>>>>>>>
>> > > >
>> > > > >>>>>>>>> I think it'd be useful to get this condensed into a CEP
>> that
>> > > we can
>> > > > then discuss in that format. It's clearly something we all agree we
>> need
>> > > and
>> > > > having an implementation that works, even if it's not in your
>> preferred
>> > > > execution domain, is vastly better than nothing IMO.
>> > > > >
>> > > > >>>>>>>>>
>> > > >
>> > > > >>>>>>>>>
>> > > > >
>> > > > >>>>>>>>>
>> > > >
>> > > > >>>>>>>>> I don't have cycles (nor background ;) ) to do that, but
>> it
>> > > sounds
>> > > > like you do Jaydeep given the implementation you have on a private
>> fork +
>> > > > design.
>> > > > >
>> > > > >>>>>>>>>
>> > > >
>> > > > >>>>>>>>>
>> > > > >
>> > > > >>>>>>>>>
>> > > >
>> > > > >>>>>>>>> A non-exhaustive list of things that might be useful
>> > > incorporating
>> > > > into or referencing from a CEP:
>> > > > >
>> > > > >>>>>>>>>
>> > > >
>> > > > >>>>>>>>> Slack thread: <https://the-
>> > > > asf.slack.com/archives/CK23JSY2K/p1690225062383619>
>> > > > >
>> > > > >>>>>>>>>
>> > > >
>> > > > >>>>>>>>> Joey's old C* ticket:
>> > > > <https://issues.apache.org/jira/browse/CASSANDRA-14346>
>> > > > >
>> > > > >>>>>>>>>
>> > > >
>> > > > >>>>>>>>> Even older automatic repair scheduling:
>> > > > <https://issues.apache.org/jira/browse/CASSANDRA-10070>
>> > > > >
>> > > > >>>>>>>>>
>> > > >
>> > > > >>>>>>>>> Your design gdoc: <
>> > > https://docs.google.com/document/d/1CJWxjEi-
>> > > > mBABPMZ3VWJ9w5KavWfJETAGxfUpsViPcPo/edit#heading=h.r112r46toau0>
>> > > > >
>> > > > >>>>>>>>>
>> > > >
>> > > > >>>>>>>>> PR with automated repair:
>> > > > <
>> > >
>> https://github.com/jaydeepkumar1984/cassandra/commit/ef6456d652c0d07cf29d88dfea03b73704814c2c
>> >
>> > >
>> > > > >
>> > > > >>>>>>>>>
>> > > >
>> > > > >>>>>>>>>
>> > > > >
>> > > > >>>>>>>>>
>> > > >
>> > > > >>>>>>>>> My intuition is that we're all basically in agreement that
>> > > this is
>> > > > something the DB needs, we're all willing to bikeshed for our
>> personal
>> > > > preference on where it lives and how it's implemented, and at the
>> end of
>> > > the
>> > > > day, code talks. I don't think anyone's said they'll die on the
>> hill of
>> > > > implementation details, so that feels like CEP time to me.
>> > > > >
>> > > > >>>>>>>>>
>> > > >
>> > > > >>>>>>>>>
>> > > > >
>> > > > >>>>>>>>>
>> > > >
>> > > > >>>>>>>>> If you were willing and able to get a CEP together for
>> > > automated
>> > > > repair based on the above material, given you've done the work and
>> have
>> > > the
>> > > > proof points it's working at scale, I think this would be a _huge
>> > > > contribution_ to the community.
>> > > > >
>> > > > >>>>>>>>>
>> > > >
>> > > > >>>>>>>>>
>> > > > >
>> > > > >>>>>>>>>
>> > > >
>> > > > >>>>>>>>> On Thu, Aug 24, 2023, at 7:26 PM, Jaydeep Chovatia wrote:
>> > > > >
>> > > > >>>>>>>>>
>> > > >
>> > > > >>>>>>>>>> Is anyone going to file an official CEP for this?
>> > > > >
>> > > > >>>>>>>>>>
>> > > >
>> > > > >>>>>>>>>> As mentioned in this email thread, here is one of the
>> > > solution's
>> > > > [design doc](https://docs.google.com/document/d/1CJWxjEi-
>> > > > mBABPMZ3VWJ9w5KavWfJETAGxfUpsViPcPo/edit#heading=h.r112r46toau0) and
>> > > source
>> > > > code on a private Apache Cassandra patch. Could you go through it
>> and
>> > > let me
>> > > > know what you think?
>> > > > >
>> > > > >>>>>>>>>>
>> > > >
>> > > > >>>>>>>>>>
>> > > > >
>> > > > >>>>>>>>>>
>> > > >
>> > > > >>>>>>>>>> Jaydeep
>> > > > >
>> > > > >>>>>>>>>>
>> > > >
>> > > > >>>>>>>>>>
>> > > > >
>> > > > >>>>>>>>>>
>> > > >
>> > > > >>>>>>>>>> On Wed, Aug 2, 2023 at 3:54 PM Jon Haddad
>> > > > <[rustyrazorblade@apache.org](mailto:rustyrazorblade@apache.org)>
>> > > wrote:
>> > > > >
>> > > > >>>>>>>>>>
>> > > >
>> > > > >>>>>>>>>>> > That said I would happily support an effort to bring
>> repair
>> > > > scheduling to the sidecar immediately. This has nothing blocking
>> it, and
>> > > would
>> > > > potentially enable the sidecar to provide an official repair
>> scheduling
>> > > > solution that is compatible with current or even previous versions
>> of the
>> > > > database.
>> > > > >
>> > > > >>>>>>>>>>>
>> > > >
>> > > > >>>>>>>>>>>
>> > > > >
>> > > > >>>>>>>>>>>
>> > > >
>> > > > >>>>>>>>>>> This is something I hadn't thought much about, and is a
>> > > pretty
>> > > > good argument for using the sidecar initially. There's a lot of
>> > > deployments
>> > > > out there and having an official repair option would be a big win.
>> > > > >
>> > > > >>>>>>>>>>>
>> > > >
>> > > > >>>>>>>>>>>
>> > > > >
>> > > > >>>>>>>>>>>
>> > > >
>> > > > >>>>>>>>>>>
>> > > > >
>> > > > >>>>>>>>>>>
>> > > >
>> > > > >>>>>>>>>>> On 2023/07/26 23:20:07 "C. Scott Andreas" wrote:
>> > > > >
>> > > > >>>>>>>>>>>
>> > > >
>> > > > >>>>>>>>>>> > I agree that it would be ideal for Cassandra to have a
>> > > repair
>> > > > scheduler in-DB.
>> > > > >
>> > > > >>>>>>>>>>>
>> > > >
>> > > > >>>>>>>>>>> >
>> > > > >
>> > > > >>>>>>>>>>>
>> > > >
>> > > > >>>>>>>>>>> > That said I would happily support an effort to bring
>> repair
>> > > > scheduling to the sidecar immediately. This has nothing blocking
>> it, and
>> > > would
>> > > > potentially enable the sidecar to provide an official repair
>> scheduling
>> > > > solution that is compatible with current or even previous versions
>> of the
>> > > > database.
>> > > > >
>> > > > >>>>>>>>>>>
>> > > >
>> > > > >>>>>>>>>>> >
>> > > > >
>> > > > >>>>>>>>>>>
>> > > >
>> > > > >>>>>>>>>>> > Once TCM has landed, we’ll have much stronger
>> primitives
>> > > for
>> > > > repair orchestration in the database itself. But I don’t think that
>> > > should
>> > > > block progress on a repair scheduling solution in the sidecar, and
>> there
>> > > is
>> > > > nothing that would prevent someone from continuing to use a
>> sidecar-based
>> > > > solution in perpetuity if they preferred.
>> > > > >
>> > > > >>>>>>>>>>>
>> > > >
>> > > > >>>>>>>>>>> >
>> > > > >
>> > > > >>>>>>>>>>>
>> > > >
>> > > > >>>>>>>>>>> > \- Scott
>> > > > >
>> > > > >>>>>>>>>>>
>> > > >
>> > > > >>>>>>>>>>> >
>> > > > >
>> > > > >>>>>>>>>>>
>> > > >
>> > > > >>>>>>>>>>> > > On Jul 26, 2023, at 3:25 PM, Jon Haddad
>> > > > <[rustyrazorblade@apache.org](mailto:rustyrazorblade@apache.org)>
>> > > wrote:
>> > > > >
>> > > > >>>>>>>>>>>
>> > > >
>> > > > >>>>>>>>>>> > >
>> > > > >
>> > > > >>>>>>>>>>>
>> > > >
>> > > > >>>>>>>>>>> > > I'm 100% in favor of repair being part of the core
>> DB,
>> > > not
>> > > > the sidecar. The current (and past) state of things where running
>> the DB
>> > > > correctly *requires* running a separate process (either community
>> > > maintained
>> > > > or official C* sidecar) is incredibly painful for folks. The idea
>> that
>> > > your
>> > > > data integrity needs to be opt-in has never made sense to me from
>> the
>> > > > perspective of either the product or the end user.
>> > > > >
>> > > > >>>>>>>>>>>
>> > > >
>> > > > >>>>>>>>>>> > >
>> > > > >
>> > > > >>>>>>>>>>>
>> > > >
>> > > > >>>>>>>>>>> > > I've worked with way too many teams that have either
>> > > > configured this incorrectly or not at all.
>> > > > >
>> > > > >>>>>>>>>>>
>> > > >
>> > > > >>>>>>>>>>> > >
>> > > > >
>> > > > >>>>>>>>>>>
>> > > >
>> > > > >>>>>>>>>>> > > Ideally Cassandra would ship with repair built in
>> and on
>> > > by
>> > > > default. Power users can disable if they want to continue to
>> maintain
>> > > their
>> > > > own repair tooling for some reason.
>> > > > >
>> > > > >>>>>>>>>>>
>> > > >
>> > > > >>>>>>>>>>> > >
>> > > > >
>> > > > >>>>>>>>>>>
>> > > >
>> > > > >>>>>>>>>>> > > Jon
>> > > > >
>> > > > >>>>>>>>>>>
>> > > >
>> > > > >>>>>>>>>>> > >
>> > > > >
>> > > > >>>>>>>>>>>
>> > > >
>> > > > >>>>>>>>>>> > >> On 2023/07/24 20:44:14 German Eichberger via dev
>> > > wrote:
>> > > > >
>> > > > >>>>>>>>>>>
>> > > >
>> > > > >>>>>>>>>>> > >> All,
>> > > > >
>> > > > >>>>>>>>>>>
>> > > >
>> > > > >>>>>>>>>>> > >> We had a brief discussion in [2] about the Uber
>> article
>> > > [1]
>> > > > where they talk about having integrated repair into Cassandra and
>> how
>> > > great
>> > > > that is. I expressed my disappointment that they didn't work with
>> the
>> > > > community on that (Uber, if you are listening time to make amends
>> 🙂)
>> > > and it
>> > > > turns out Joey already had the idea and wrote the code [3] - so I
>> wanted
>> > > to
>> > > > start a discussion to gauge interest and maybe how to revive that
>> > > effort.
>> > > > >
>> > > > >>>>>>>>>>>
>> > > >
>> > > > >>>>>>>>>>> > >> Thanks,
>> > > > >
>> > > > >>>>>>>>>>>
>> > > >
>> > > > >>>>>>>>>>> > >> German
>> > > > >
>> > > > >>>>>>>>>>>
>> > > >
>> > > > >>>>>>>>>>> > >> [1] <
>> > > https://www.uber.com/blog/how-uber-optimized-cassandra-
>> > > > operations-at-scale/>
>> > > > >
>> > > > >>>>>>>>>>>
>> > > >
>> > > > >>>>>>>>>>> > >> [2] <https://the-
>> > > > asf.slack.com/archives/CK23JSY2K/p1690225062383619>
>> > > > >
>> > > > >>>>>>>>>>>
>> > > >
>> > > > >>>>>>>>>>> > >> [3] <
>> > > https://issues.apache.org/jira/browse/CASSANDRA-14346>
>> > > > >
>> > > > >>>>>>>>>>>
>> > > >
>> > > > >>>>>>>>>>> >
>> > > > >
>> > > > >>>>>>>>>
>> > > >
>> > > > >>>>>>>>>
>> > > > >
>> > > > >>>>>>
>> > > >
>> > > > >>>>>>
>> > > > >
>> > > >
>> > > >
>> > >
>> >
>>
>
Re: [Discuss] Repair inside C*
Posted by Joseph Lynch <jo...@gmail.com>.
Definitely like this in C* itself. We only changed our proposal to putting
repair scheduling in the sidecar before because trunk was frozen for the
foreseeable future at that time. With trunk unfrozen and development on the
main process going at a fast pace I think it makes way more sense to
integrate natively as table properties as this CEP proposes. Completely
agree the scheduling overhead should be minimal.
Moving the actual repair operation (comparing data and streaming
mismatches) along with compaction operations to a separate process long
term makes a lot of sense but imo only once we both have a release of
sidecar and a contract figured out between them on communication. I'm
watching CEP-38 there as I think CQL and virtual tables are looking much
stronger than when we wrote CEP-1 and chose HTTP but that's for that
discussion and not this one.
-Joey
On Mon, Oct 21, 2024 at 3:25 PM Francisco Guerrero <fr...@apache.org>
wrote:
> Like others have said, I was expecting the scheduling portion of repair is
> negligible. I was mostly curious if you had something handy that you can
> quickly share.
>
> On 2024/10/21 18:59:41 Jaydeep Chovatia wrote:
> > >Jaydeep, do you have any metrics on your clusters comparing them before
> > and after introducing repair scheduling into the Cassandra process?
> >
> > Yes, I had made some comparisons when I started rolling this feature out
> to
> > our production five years ago :) Here are the details:
> > *The Scheduling*
> > The scheduling itself is exceptionally lightweight, as only one
> additional
> > thread monitors the repair activity, updating the status to a system
> table
> > once every few minutes or so. So, it does not appear anywhere in the CPU
> > charts, etc. Unfortunately, I do not have those graphs now, but I can do
> a
> > quick comparison if it helps!
> >
> > *The Repair Itself*
> > As we all know, the Cassandra repair algorithm is a heavy-weight process
> > due to Merkle tree/streaming, etc., no matter how we schedule it. But it
> is
> > an orthogonal topic and folks are already discussing creating a new CEP.
> >
> > Jaydeep
> >
> >
> > On Mon, Oct 21, 2024 at 10:02 AM Francisco Guerrero <fr...@apache.org>
> > wrote:
> >
> > > Jaydeep, do you have any metrics on your clusters comparing them before
> > > and after introducing repair scheduling into the Cassandra process?
> > >
> > > On 2024/10/21 16:57:57 "J. D. Jordan" wrote:
> > > > Sounds good. Just wanted to bring it up. I agree that the scheduling
> bit
> > > is
> > > > pretty light weight and the ideal would be to bring the whole of the
> > > repair
> > > > external, which is a much bigger can of worms to open.
> > > >
> > > >
> > > >
> > > > -Jeremiah
> > > >
> > > >
> > > >
> > > > > On Oct 21, 2024, at 11:21 AM, Chris Lohfink <cl...@gmail.com>
> > > wrote:
> > > > >
> > > > >
> > > >
> > > > >
> > > > >
> > > > > > I actually think we should be looking at how we can move things
> out
> > > of the
> > > > > database process.
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > While worth pursuing, I think we would need a different CEP just to
> > > figure
> > > > > out how to do that. Not only is there a lot of infrastructure
> > > difficulty in
> > > > > running multi process, the inter app communication needs to be
> figured
> > > out
> > > > > better then JMX. Even the sidecar we dont have a solid story on
> how to
> > > > > ensure both are running or anything yet. It's up to each app owner
> to
> > > figure
> > > > > it out. Once we have a good thing in place I think we can start
> moving
> > > > > compactions, repairs, etc out of the database. Even then it's the
> > > _repairs_
> > > > > that is expensive, not the scheduling.
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > On Mon, Oct 21, 2024 at 9:45 AM Jeremiah Jordan
> > > > > <[jeremiah.jordan@gmail.com](mailto:jeremiah.jordan@gmail.com)>
> > > wrote:
> > > > >
> > > > >
> > > >
> > > > >> I love the idea of a repair service being there by default for an
> > > install
> > > > of C*. My main concern here is that it is putting more services into
> > > the main
> > > > database process. I actually think we should be looking at how we
> can
> > > move
> > > > things out of the database process. The C* process being a giant
> > > monolith has
> > > > always been a pain point. Is there anyway it makes sense for this
> to be
> > > an
> > > > external process rather than a new thread pool inside the C* process?
> > > >
> > > > >>
> > > >
> > > > >>
> > > > >
> > > > >>
> > > >
> > > > >> -Jeremiah Jordan
> > > >
> > > > >>
> > > >
> > > > >>
> > > > >
> > > > >>
> > > >
> > > > >> On Oct 18, 2024 at 2:58:15 PM, Mick Semb Wever
> > > > <[mck@apache.org](mailto:mck@apache.org)> wrote:
> > > > >
> > > > >>
> > > >
> > > > >>>
> > > > >
> > > > >>>
> > > >
> > > > >>> This is looking strong, thanks Jaydeep.
> > > >
> > > > >>>
> > > >
> > > > >>>
> > > > >
> > > > >>>
> > > >
> > > > >>> I would suggest folk take a look at the design doc and the PR in
> the
> > > CEP.
> > > > A lot is there (that I have completely missed).
> > > >
> > > > >>>
> > > >
> > > > >>>
> > > > >
> > > > >>>
> > > >
> > > > >>> I would especially ask all authors of prior art (Reaper, DSE
> > > nodesync,
> > > > ecchronos) to take a final review of the proposal
> > > > >
> > > > >>>
> > > >
> > > > >>>
> > > > >
> > > > >>>
> > > >
> > > > >>> Jaydeep, can we ask for a two week window while we reach out to
> these
> > > > people ? There's a lot of prior art in this space, and it feels like
> > > we're in
> > > > a good place now where it's clear this has legs and we can use that
> to
> > > bring
> > > > folk in and make sure there's no remaining blindspots.
> > > >
> > > > >>>
> > > >
> > > > >>>
> > > > >
> > > > >>>
> > > >
> > > > >>>
> > > > >
> > > > >>>
> > > >
> > > > >>> On Fri, 18 Oct 2024 at 01:40, Jaydeep Chovatia
> > > > <[chovatia.jaydeep@gmail.com](mailto:chovatia.jaydeep@gmail.com)>
> > > wrote:
> > > > >
> > > > >>>
> > > >
> > > > >>>> Sorry, there is a typo in the CEP-37 link; here is the correct
> > > > [link](
> > >
> https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-37+Apache+Cassandra+Unified+Repair+Solution
> > > )
> > > >
> > > > >>>>
> > > >
> > > > >>>>
> > > > >
> > > > >>>>
> > > >
> > > > >>>>
> > > > >
> > > > >>>>
> > > >
> > > > >>>> On Thu, Oct 17, 2024 at 4:36 PM Jaydeep Chovatia
> > > > <[chovatia.jaydeep@gmail.com](mailto:chovatia.jaydeep@gmail.com)>
> > > wrote:
> > > > >
> > > > >>>>
> > > >
> > > > >>>>> First, thank you for your patience while we strengthened the
> > > CEP-37.
> > > >
> > > > >>>>>
> > > >
> > > > >>>>>
> > > > >
> > > > >>>>>
> > > >
> > > > >>>>> Over the last eight months, Chris Lohfink, Andy Tolbert, Josh
> > > McKenzie,
> > > > Dinesh Joshi, Kristijonas Zalys, and I have done tons of work (online
> > > > discussions/a dedicated Slack channel
> > > #cassandra-repair-scheduling-cep37) to
> > > > come up with the best possible design that not only significantly
> > > simplifies
> > > > repair operations but also includes the most common features that
> > > everyone
> > > > will benefit from running at Scale.
> > > >
> > > > >>>>>
> > > >
> > > > >>>>> For example,
> > > >
> > > > >>>>>
> > > >
> > > > >>>>> * Apache Cassandra must be capable of running multiple repair
> > > types,
> > > > such as Full, Incremental, Paxos, and Preview - so the framework
> should
> > > be
> > > > easily extendable with no additional overhead from the operator’s
> point
> > > of
> > > > view.
> > > >
> > > > >>>>>
> > > >
> > > > >>>>> * An easy way to extend the token-split calculation algorithm
> > > with a
> > > > default implementation should exist.
> > > >
> > > > >>>>>
> > > >
> > > > >>>>> * Running incremental repair reliably at Scale is pretty
> > > challenging,
> > > > so we need to place safeguards, such as migration/rollback w/o
> restart
> > > and
> > > > stopping incremental repair automatically if the disk is about to get
> > > full.
> > > >
> > > > >>>>>
> > > >
> > > > >>>>>
> > > >
> > > > >>>>>
> > > >
> > > > >>>>> We are glad to inform you that CEP-37 (i.e., Repair inside
> > > Cassandra) is
> > > > now officially ready for review after multiple rounds of design,
> > > testing, code
> > > > reviews, documentation reviews, and, more importantly, validation
> that
> > > it runs
> > > > at Scale!
> > > >
> > > > >>>>>
> > > >
> > > > >>>>>
> > > > >
> > > > >>>>>
> > > >
> > > > >>>>> Some facts about CEP-37.
> > > >
> > > > >>>>>
> > > >
> > > > >>>>> * Multiple members have verified all aspects of CEP-37
> numerous
> > > times.
> > > >
> > > > >>>>>
> > > >
> > > > >>>>> * The design proposed in CEP-37 has been thoroughly tried and
> > > tested
> > > > on an immense scale (hundreds of unique Cassandra clusters, tens of
> > > thousands
> > > > of Cassandra nodes, with tens of millions of QPS) on top of 4.1
> > > open-source
> > > > for more than five years; please see more details[
> > > > here](
> > >
> https://www.uber.com/en-US/blog/how-uber-optimized-cassandra-operations-
> > > > at-scale/).
> > > >
> > > > >>>>>
> > > >
> > > > >>>>> * The following
> > > > [presentation](
> > >
> https://docs.google.com/presentation/d/1Zilww9c7LihHULk_ckErI2s4XbObxjWknKqRtbvHyZc/edit#slide=id.g30a4fd4fcf7_0_13
> > > )
> > > > highlights the rigorous applied to CEP-37, which was given during
> last
> > > week’s
> > > > Apache Cassandra Bay Area [Meetup](
> > > https://www.meetup.com/apache-cassandra-
> > > > bay-area/events/303469006/),
> > > >
> > > > >>>>>
> > > >
> > > > >>>>>
> > > >
> > > >
> > > > >
> > > > >>>>>
> > > >
> > > > >>>>> Since things are massively overhauled, we believe it is almost
> > > ready for
> > > > a final pass pre-VOTE. We would like you to please review the
> > > > [CEP-37](
> > >
> https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-37+Apache+Cassandra+Unified+Repair+Solution\
> <https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-37+Apache+Cassandra+Unified+Repair+Solution%5C>
> > > <
> https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-37+Apache+Cassandra+Unified+Repair+Solution%5C
> >
> > > ))
> > > > and the associated detailed design
> > > > [doc](https://docs.google.com/document/d/1CJWxjEi-
> > > > mBABPMZ3VWJ9w5KavWfJETAGxfUpsViPcPo/edit#heading=h.r112r46toau0).
> > > >
> > > > >>>>>
> > > >
> > > > >>>>>
> > > > >
> > > > >>>>>
> > > >
> > > > >>>>> Thank you everyone!
> > > >
> > > > >>>>>
> > > >
> > > > >>>>> Chris, Andy, Josh, Dinesh, Kristijonas, and Jaydeep
> > > >
> > > > >>>>>
> > > >
> > > > >>>>>
> > > > >
> > > > >
> > > > >>>>>
> > > >
> > > > >>>>>
> > > > >
> > > > >>>>>
> > > >
> > > > >>>>> On Thu, Sep 19, 2024 at 11:26 AM Josh McKenzie
> > > > <[jmckenzie@apache.org](mailto:jmckenzie@apache.org)> wrote:
> > > > >
> > > > >>>>>
> > > >
> > > > >>>>>> __
> > > >
> > > > >>>>>>
> > > >
> > > > >>>>>> Not quite; finishing touches on the CEP and design doc are in
> > > flight
> > > > (as of last / this week).
> > > > >
> > > > >>>>>>
> > > >
> > > > >>>>>>
> > > > >
> > > > >>>>>>
> > > >
> > > > >>>>>> Soon(tm).
> > > >
> > > > >>>>>>
> > > >
> > > > >>>>>>
> > > > >
> > > > >>>>>>
> > > >
> > > > >>>>>> On Thu, Sep 19, 2024, at 2:07 PM, Patrick McFadin wrote:
> > > > >
> > > > >>>>>>
> > > >
> > > > >>>>>>> Is this CEP ready for a VOTE thread?
> > > > <
> > >
> https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-37+%28DRAFT%29+Apache+Cassandra+Unified+Repair+Solution
> >
> > >
> > > > >
> > > > >>>>>>>
> > > >
> > > > >>>>>>>
> > > > >
> > > > >>>>>>>
> > > >
> > > > >>>>>>> On Sun, Feb 25, 2024 at 12:25 PM Jaydeep Chovatia
> > > > <[chovatia.jaydeep@gmail.com](mailto:chovatia.jaydeep@gmail.com)>
> > > wrote:
> > > > >
> > > > >>>>>>>
> > > >
> > > > >>>>>>>> Thanks, Josh. I've just updated the
> > > > [CEP](
> > >
> https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-37+%28DRAFT%29+Apache+Cassandra+Official+Repair+Solution
> > > )
> > > > and included all the solutions you mentioned below.
> > > > >
> > > > >>>>>>>>
> > > >
> > > > >>>>>>>>
> > > > >
> > > > >>>>>>>>
> > > >
> > > > >>>>>>>> Jaydeep
> > > > >
> > > > >>>>>>>>
> > > >
> > > > >>>>>>>>
> > > > >
> > > > >>>>>>>>
> > > >
> > > > >>>>>>>> On Thu, Feb 22, 2024 at 9:33 AM Josh McKenzie
> > > > <[jmckenzie@apache.org](mailto:jmckenzie@apache.org)> wrote:
> > > > >
> > > > >>>>>>>>
> > > >
> > > > >>>>>>>>> __
> > > > >
> > > > >>>>>>>>>
> > > >
> > > > >>>>>>>>> Very late response from me here (basically necro'ing this
> > > thread).
> > > > >
> > > > >>>>>>>>>
> > > >
> > > > >>>>>>>>>
> > > > >
> > > > >>>>>>>>>
> > > >
> > > > >>>>>>>>> I think it'd be useful to get this condensed into a CEP
> that
> > > we can
> > > > then discuss in that format. It's clearly something we all agree we
> need
> > > and
> > > > having an implementation that works, even if it's not in your
> preferred
> > > > execution domain, is vastly better than nothing IMO.
> > > > >
> > > > >>>>>>>>>
> > > >
> > > > >>>>>>>>>
> > > > >
> > > > >>>>>>>>>
> > > >
> > > > >>>>>>>>> I don't have cycles (nor background ;) ) to do that, but it
> > > sounds
> > > > like you do Jaydeep given the implementation you have on a private
> fork +
> > > > design.
> > > > >
> > > > >>>>>>>>>
> > > >
> > > > >>>>>>>>>
> > > > >
> > > > >>>>>>>>>
> > > >
> > > > >>>>>>>>> A non-exhaustive list of things that might be useful
> > > incorporating
> > > > into or referencing from a CEP:
> > > > >
> > > > >>>>>>>>>
> > > >
> > > > >>>>>>>>> Slack thread: <https://the-
> > > > asf.slack.com/archives/CK23JSY2K/p1690225062383619>
> > > > >
> > > > >>>>>>>>>
> > > >
> > > > >>>>>>>>> Joey's old C* ticket:
> > > > <https://issues.apache.org/jira/browse/CASSANDRA-14346>
> > > > >
> > > > >>>>>>>>>
> > > >
> > > > >>>>>>>>> Even older automatic repair scheduling:
> > > > <https://issues.apache.org/jira/browse/CASSANDRA-10070>
> > > > >
> > > > >>>>>>>>>
> > > >
> > > > >>>>>>>>> Your design gdoc: <
> > > https://docs.google.com/document/d/1CJWxjEi-
> > > > mBABPMZ3VWJ9w5KavWfJETAGxfUpsViPcPo/edit#heading=h.r112r46toau0>
> > > > >
> > > > >>>>>>>>>
> > > >
> > > > >>>>>>>>> PR with automated repair:
> > > > <
> > >
> https://github.com/jaydeepkumar1984/cassandra/commit/ef6456d652c0d07cf29d88dfea03b73704814c2c
> >
> > >
> > > > >
> > > > >>>>>>>>>
> > > >
> > > > >>>>>>>>>
> > > > >
> > > > >>>>>>>>>
> > > >
> > > > >>>>>>>>> My intuition is that we're all basically in agreement that
> > > this is
> > > > something the DB needs, we're all willing to bikeshed for our
> personal
> > > > preference on where it lives and how it's implemented, and at the
> end of
> > > the
> > > > day, code talks. I don't think anyone's said they'll die on the hill
> of
> > > > implementation details, so that feels like CEP time to me.
> > > > >
> > > > >>>>>>>>>
> > > >
> > > > >>>>>>>>>
> > > > >
> > > > >>>>>>>>>
> > > >
> > > > >>>>>>>>> If you were willing and able to get a CEP together for
> > > automated
> > > > repair based on the above material, given you've done the work and
> have
> > > the
> > > > proof points it's working at scale, I think this would be a _huge
> > > > contribution_ to the community.
> > > > >
> > > > >>>>>>>>>
> > > >
> > > > >>>>>>>>>
> > > > >
> > > > >>>>>>>>>
> > > >
> > > > >>>>>>>>> On Thu, Aug 24, 2023, at 7:26 PM, Jaydeep Chovatia wrote:
> > > > >
> > > > >>>>>>>>>
> > > >
> > > > >>>>>>>>>> Is anyone going to file an official CEP for this?
> > > > >
> > > > >>>>>>>>>>
> > > >
> > > > >>>>>>>>>> As mentioned in this email thread, here is one of the
> > > solution's
> > > > [design doc](https://docs.google.com/document/d/1CJWxjEi-
> > > > mBABPMZ3VWJ9w5KavWfJETAGxfUpsViPcPo/edit#heading=h.r112r46toau0) and
> > > source
> > > > code on a private Apache Cassandra patch. Could you go through it and
> > > let me
> > > > know what you think?
> > > > >
> > > > >>>>>>>>>>
> > > >
> > > > >>>>>>>>>>
> > > > >
> > > > >>>>>>>>>>
> > > >
> > > > >>>>>>>>>> Jaydeep
> > > > >
> > > > >>>>>>>>>>
> > > >
> > > > >>>>>>>>>>
> > > > >
> > > > >>>>>>>>>>
> > > >
> > > > >>>>>>>>>> On Wed, Aug 2, 2023 at 3:54 PM Jon Haddad
> > > > <[rustyrazorblade@apache.org](mailto:rustyrazorblade@apache.org)>
> > > wrote:
> > > > >
> > > > >>>>>>>>>>
> > > >
> > > > >>>>>>>>>>> > That said I would happily support an effort to bring
> repair
> > > > scheduling to the sidecar immediately. This has nothing blocking it,
> and
> > > would
> > > > potentially enable the sidecar to provide an official repair
> scheduling
> > > > solution that is compatible with current or even previous versions
> of the
> > > > database.
> > > > >
> > > > >>>>>>>>>>>
> > > >
> > > > >>>>>>>>>>>
> > > > >
> > > > >>>>>>>>>>>
> > > >
> > > > >>>>>>>>>>> This is something I hadn't thought much about, and is a
> > > pretty
> > > > good argument for using the sidecar initially. There's a lot of
> > > deployments
> > > > out there and having an official repair option would be a big win.
> > > > >
> > > > >>>>>>>>>>>
> > > >
> > > > >>>>>>>>>>>
> > > > >
> > > > >>>>>>>>>>>
> > > >
> > > > >>>>>>>>>>>
> > > > >
> > > > >>>>>>>>>>>
> > > >
> > > > >>>>>>>>>>> On 2023/07/26 23:20:07 "C. Scott Andreas" wrote:
> > > > >
> > > > >>>>>>>>>>>
> > > >
> > > > >>>>>>>>>>> > I agree that it would be ideal for Cassandra to have a
> > > repair
> > > > scheduler in-DB.
> > > > >
> > > > >>>>>>>>>>>
> > > >
> > > > >>>>>>>>>>> >
> > > > >
> > > > >>>>>>>>>>>
> > > >
> > > > >>>>>>>>>>> > That said I would happily support an effort to bring
> repair
> > > > scheduling to the sidecar immediately. This has nothing blocking it,
> and
> > > would
> > > > potentially enable the sidecar to provide an official repair
> scheduling
> > > > solution that is compatible with current or even previous versions
> of the
> > > > database.
> > > > >
> > > > >>>>>>>>>>>
> > > >
> > > > >>>>>>>>>>> >
> > > > >
> > > > >>>>>>>>>>>
> > > >
> > > > >>>>>>>>>>> > Once TCM has landed, we’ll have much stronger
> primitives
> > > for
> > > > repair orchestration in the database itself. But I don’t think that
> > > should
> > > > block progress on a repair scheduling solution in the sidecar, and
> there
> > > is
> > > > nothing that would prevent someone from continuing to use a
> sidecar-based
> > > > solution in perpetuity if they preferred.
> > > > >
> > > > >>>>>>>>>>>
> > > >
> > > > >>>>>>>>>>> >
> > > > >
> > > > >>>>>>>>>>>
> > > >
> > > > >>>>>>>>>>> > \- Scott
> > > > >
> > > > >>>>>>>>>>>
> > > >
> > > > >>>>>>>>>>> >
> > > > >
> > > > >>>>>>>>>>>
> > > >
> > > > >>>>>>>>>>> > > On Jul 26, 2023, at 3:25 PM, Jon Haddad
> > > > <[rustyrazorblade@apache.org](mailto:rustyrazorblade@apache.org)>
> > > wrote:
> > > > >
> > > > >>>>>>>>>>>
> > > >
> > > > >>>>>>>>>>> > >
> > > > >
> > > > >>>>>>>>>>>
> > > >
> > > > >>>>>>>>>>> > > I'm 100% in favor of repair being part of the core
> DB,
> > > not
> > > > the sidecar. The current (and past) state of things where running
> the DB
> > > > correctly *requires* running a separate process (either community
> > > maintained
> > > > or official C* sidecar) is incredibly painful for folks. The idea
> that
> > > your
> > > > data integrity needs to be opt-in has never made sense to me from the
> > > > perspective of either the product or the end user.
> > > > >
> > > > >>>>>>>>>>>
> > > >
> > > > >>>>>>>>>>> > >
> > > > >
> > > > >>>>>>>>>>>
> > > >
> > > > >>>>>>>>>>> > > I've worked with way too many teams that have either
> > > > configured this incorrectly or not at all.
> > > > >
> > > > >>>>>>>>>>>
> > > >
> > > > >>>>>>>>>>> > >
> > > > >
> > > > >>>>>>>>>>>
> > > >
> > > > >>>>>>>>>>> > > Ideally Cassandra would ship with repair built in
> and on
> > > by
> > > > default. Power users can disable if they want to continue to
> maintain
> > > their
> > > > own repair tooling for some reason.
> > > > >
> > > > >>>>>>>>>>>
> > > >
> > > > >>>>>>>>>>> > >
> > > > >
> > > > >>>>>>>>>>>
> > > >
> > > > >>>>>>>>>>> > > Jon
> > > > >
> > > > >>>>>>>>>>>
> > > >
> > > > >>>>>>>>>>> > >
> > > > >
> > > > >>>>>>>>>>>
> > > >
> > > > >>>>>>>>>>> > >> On 2023/07/24 20:44:14 German Eichberger via dev
> > > wrote:
> > > > >
> > > > >>>>>>>>>>>
> > > >
> > > > >>>>>>>>>>> > >> All,
> > > > >
> > > > >>>>>>>>>>>
> > > >
> > > > >>>>>>>>>>> > >> We had a brief discussion in [2] about the Uber
> article
> > > [1]
> > > > where they talk about having integrated repair into Cassandra and how
> > > great
> > > > that is. I expressed my disappointment that they didn't work with the
> > > > community on that (Uber, if you are listening time to make amends 🙂)
> > > and it
> > > > turns out Joey already had the idea and wrote the code [3] - so I
> wanted
> > > to
> > > > start a discussion to gauge interest and maybe how to revive that
> > > effort.
> > > > >
> > > > >>>>>>>>>>>
> > > >
> > > > >>>>>>>>>>> > >> Thanks,
> > > > >
> > > > >>>>>>>>>>>
> > > >
> > > > >>>>>>>>>>> > >> German
> > > > >
> > > > >>>>>>>>>>>
> > > >
> > > > >>>>>>>>>>> > >> [1] <
> > > https://www.uber.com/blog/how-uber-optimized-cassandra-
> > > > operations-at-scale/>
> > > > >
> > > > >>>>>>>>>>>
> > > >
> > > > >>>>>>>>>>> > >> [2] <https://the-
> > > > asf.slack.com/archives/CK23JSY2K/p1690225062383619>
> > > > >
> > > > >>>>>>>>>>>
> > > >
> > > > >>>>>>>>>>> > >> [3] <
> > > https://issues.apache.org/jira/browse/CASSANDRA-14346>
> > > > >
> > > > >>>>>>>>>>>
> > > >
> > > > >>>>>>>>>>> >
> > > > >
> > > > >>>>>>>>>
> > > >
> > > > >>>>>>>>>
> > > > >
> > > > >>>>>>
> > > >
> > > > >>>>>>
> > > > >
> > > >
> > > >
> > >
> >
>
Re: [Discuss] Repair inside C*
Posted by Francisco Guerrero <fr...@apache.org>.
Like others have said, I was expecting the scheduling portion of repair is
negligible. I was mostly curious if you had something handy that you can
quickly share.
On 2024/10/21 18:59:41 Jaydeep Chovatia wrote:
> >Jaydeep, do you have any metrics on your clusters comparing them before
> and after introducing repair scheduling into the Cassandra process?
>
> Yes, I had made some comparisons when I started rolling this feature out to
> our production five years ago :) Here are the details:
> *The Scheduling*
> The scheduling itself is exceptionally lightweight, as only one additional
> thread monitors the repair activity, updating the status to a system table
> once every few minutes or so. So, it does not appear anywhere in the CPU
> charts, etc. Unfortunately, I do not have those graphs now, but I can do a
> quick comparison if it helps!
>
> *The Repair Itself*
> As we all know, the Cassandra repair algorithm is a heavy-weight process
> due to Merkle tree/streaming, etc., no matter how we schedule it. But it is
> an orthogonal topic and folks are already discussing creating a new CEP.
>
> Jaydeep
>
>
> On Mon, Oct 21, 2024 at 10:02 AM Francisco Guerrero <fr...@apache.org>
> wrote:
>
> > Jaydeep, do you have any metrics on your clusters comparing them before
> > and after introducing repair scheduling into the Cassandra process?
> >
> > On 2024/10/21 16:57:57 "J. D. Jordan" wrote:
> > > Sounds good. Just wanted to bring it up. I agree that the scheduling bit
> > is
> > > pretty light weight and the ideal would be to bring the whole of the
> > repair
> > > external, which is a much bigger can of worms to open.
> > >
> > >
> > >
> > > -Jeremiah
> > >
> > >
> > >
> > > > On Oct 21, 2024, at 11:21 AM, Chris Lohfink <cl...@gmail.com>
> > wrote:
> > > >
> > > >
> > >
> > > >
> > > >
> > > > > I actually think we should be looking at how we can move things out
> > of the
> > > > database process.
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > While worth pursuing, I think we would need a different CEP just to
> > figure
> > > > out how to do that. Not only is there a lot of infrastructure
> > difficulty in
> > > > running multi process, the inter app communication needs to be figured
> > out
> > > > better then JMX. Even the sidecar we dont have a solid story on how to
> > > > ensure both are running or anything yet. It's up to each app owner to
> > figure
> > > > it out. Once we have a good thing in place I think we can start moving
> > > > compactions, repairs, etc out of the database. Even then it's the
> > _repairs_
> > > > that is expensive, not the scheduling.
> > > >
> > > >
> > > >
> > > >
> > > > On Mon, Oct 21, 2024 at 9:45 AM Jeremiah Jordan
> > > > <[jeremiah.jordan@gmail.com](mailto:jeremiah.jordan@gmail.com)>
> > wrote:
> > > >
> > > >
> > >
> > > >> I love the idea of a repair service being there by default for an
> > install
> > > of C*. My main concern here is that it is putting more services into
> > the main
> > > database process. I actually think we should be looking at how we can
> > move
> > > things out of the database process. The C* process being a giant
> > monolith has
> > > always been a pain point. Is there anyway it makes sense for this to be
> > an
> > > external process rather than a new thread pool inside the C* process?
> > >
> > > >>
> > >
> > > >>
> > > >
> > > >>
> > >
> > > >> -Jeremiah Jordan
> > >
> > > >>
> > >
> > > >>
> > > >
> > > >>
> > >
> > > >> On Oct 18, 2024 at 2:58:15 PM, Mick Semb Wever
> > > <[mck@apache.org](mailto:mck@apache.org)> wrote:
> > > >
> > > >>
> > >
> > > >>>
> > > >
> > > >>>
> > >
> > > >>> This is looking strong, thanks Jaydeep.
> > >
> > > >>>
> > >
> > > >>>
> > > >
> > > >>>
> > >
> > > >>> I would suggest folk take a look at the design doc and the PR in the
> > CEP.
> > > A lot is there (that I have completely missed).
> > >
> > > >>>
> > >
> > > >>>
> > > >
> > > >>>
> > >
> > > >>> I would especially ask all authors of prior art (Reaper, DSE
> > nodesync,
> > > ecchronos) to take a final review of the proposal
> > > >
> > > >>>
> > >
> > > >>>
> > > >
> > > >>>
> > >
> > > >>> Jaydeep, can we ask for a two week window while we reach out to these
> > > people ? There's a lot of prior art in this space, and it feels like
> > we're in
> > > a good place now where it's clear this has legs and we can use that to
> > bring
> > > folk in and make sure there's no remaining blindspots.
> > >
> > > >>>
> > >
> > > >>>
> > > >
> > > >>>
> > >
> > > >>>
> > > >
> > > >>>
> > >
> > > >>> On Fri, 18 Oct 2024 at 01:40, Jaydeep Chovatia
> > > <[chovatia.jaydeep@gmail.com](mailto:chovatia.jaydeep@gmail.com)>
> > wrote:
> > > >
> > > >>>
> > >
> > > >>>> Sorry, there is a typo in the CEP-37 link; here is the correct
> > > [link](
> > https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-37+Apache+Cassandra+Unified+Repair+Solution
> > )
> > >
> > > >>>>
> > >
> > > >>>>
> > > >
> > > >>>>
> > >
> > > >>>>
> > > >
> > > >>>>
> > >
> > > >>>> On Thu, Oct 17, 2024 at 4:36 PM Jaydeep Chovatia
> > > <[chovatia.jaydeep@gmail.com](mailto:chovatia.jaydeep@gmail.com)>
> > wrote:
> > > >
> > > >>>>
> > >
> > > >>>>> First, thank you for your patience while we strengthened the
> > CEP-37.
> > >
> > > >>>>>
> > >
> > > >>>>>
> > > >
> > > >>>>>
> > >
> > > >>>>> Over the last eight months, Chris Lohfink, Andy Tolbert, Josh
> > McKenzie,
> > > Dinesh Joshi, Kristijonas Zalys, and I have done tons of work (online
> > > discussions/a dedicated Slack channel
> > #cassandra-repair-scheduling-cep37) to
> > > come up with the best possible design that not only significantly
> > simplifies
> > > repair operations but also includes the most common features that
> > everyone
> > > will benefit from running at Scale.
> > >
> > > >>>>>
> > >
> > > >>>>> For example,
> > >
> > > >>>>>
> > >
> > > >>>>> * Apache Cassandra must be capable of running multiple repair
> > types,
> > > such as Full, Incremental, Paxos, and Preview - so the framework should
> > be
> > > easily extendable with no additional overhead from the operator’s point
> > of
> > > view.
> > >
> > > >>>>>
> > >
> > > >>>>> * An easy way to extend the token-split calculation algorithm
> > with a
> > > default implementation should exist.
> > >
> > > >>>>>
> > >
> > > >>>>> * Running incremental repair reliably at Scale is pretty
> > challenging,
> > > so we need to place safeguards, such as migration/rollback w/o restart
> > and
> > > stopping incremental repair automatically if the disk is about to get
> > full.
> > >
> > > >>>>>
> > >
> > > >>>>>
> > >
> > > >>>>>
> > >
> > > >>>>> We are glad to inform you that CEP-37 (i.e., Repair inside
> > Cassandra) is
> > > now officially ready for review after multiple rounds of design,
> > testing, code
> > > reviews, documentation reviews, and, more importantly, validation that
> > it runs
> > > at Scale!
> > >
> > > >>>>>
> > >
> > > >>>>>
> > > >
> > > >>>>>
> > >
> > > >>>>> Some facts about CEP-37.
> > >
> > > >>>>>
> > >
> > > >>>>> * Multiple members have verified all aspects of CEP-37 numerous
> > times.
> > >
> > > >>>>>
> > >
> > > >>>>> * The design proposed in CEP-37 has been thoroughly tried and
> > tested
> > > on an immense scale (hundreds of unique Cassandra clusters, tens of
> > thousands
> > > of Cassandra nodes, with tens of millions of QPS) on top of 4.1
> > open-source
> > > for more than five years; please see more details[
> > > here](
> > https://www.uber.com/en-US/blog/how-uber-optimized-cassandra-operations-
> > > at-scale/).
> > >
> > > >>>>>
> > >
> > > >>>>> * The following
> > > [presentation](
> > https://docs.google.com/presentation/d/1Zilww9c7LihHULk_ckErI2s4XbObxjWknKqRtbvHyZc/edit#slide=id.g30a4fd4fcf7_0_13
> > )
> > > highlights the rigorous applied to CEP-37, which was given during last
> > week’s
> > > Apache Cassandra Bay Area [Meetup](
> > https://www.meetup.com/apache-cassandra-
> > > bay-area/events/303469006/),
> > >
> > > >>>>>
> > >
> > > >>>>>
> > >
> > >
> > > >
> > > >>>>>
> > >
> > > >>>>> Since things are massively overhauled, we believe it is almost
> > ready for
> > > a final pass pre-VOTE. We would like you to please review the
> > > [CEP-37](
> > https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-37+Apache+Cassandra+Unified+Repair+Solution\
> > <https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-37+Apache+Cassandra+Unified+Repair+Solution%5C>
> > ))
> > > and the associated detailed design
> > > [doc](https://docs.google.com/document/d/1CJWxjEi-
> > > mBABPMZ3VWJ9w5KavWfJETAGxfUpsViPcPo/edit#heading=h.r112r46toau0).
> > >
> > > >>>>>
> > >
> > > >>>>>
> > > >
> > > >>>>>
> > >
> > > >>>>> Thank you everyone!
> > >
> > > >>>>>
> > >
> > > >>>>> Chris, Andy, Josh, Dinesh, Kristijonas, and Jaydeep
> > >
> > > >>>>>
> > >
> > > >>>>>
> > > >
> > > >
> > > >>>>>
> > >
> > > >>>>>
> > > >
> > > >>>>>
> > >
> > > >>>>> On Thu, Sep 19, 2024 at 11:26 AM Josh McKenzie
> > > <[jmckenzie@apache.org](mailto:jmckenzie@apache.org)> wrote:
> > > >
> > > >>>>>
> > >
> > > >>>>>> __
> > >
> > > >>>>>>
> > >
> > > >>>>>> Not quite; finishing touches on the CEP and design doc are in
> > flight
> > > (as of last / this week).
> > > >
> > > >>>>>>
> > >
> > > >>>>>>
> > > >
> > > >>>>>>
> > >
> > > >>>>>> Soon(tm).
> > >
> > > >>>>>>
> > >
> > > >>>>>>
> > > >
> > > >>>>>>
> > >
> > > >>>>>> On Thu, Sep 19, 2024, at 2:07 PM, Patrick McFadin wrote:
> > > >
> > > >>>>>>
> > >
> > > >>>>>>> Is this CEP ready for a VOTE thread?
> > > <
> > https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-37+%28DRAFT%29+Apache+Cassandra+Unified+Repair+Solution>
> >
> > > >
> > > >>>>>>>
> > >
> > > >>>>>>>
> > > >
> > > >>>>>>>
> > >
> > > >>>>>>> On Sun, Feb 25, 2024 at 12:25 PM Jaydeep Chovatia
> > > <[chovatia.jaydeep@gmail.com](mailto:chovatia.jaydeep@gmail.com)>
> > wrote:
> > > >
> > > >>>>>>>
> > >
> > > >>>>>>>> Thanks, Josh. I've just updated the
> > > [CEP](
> > https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-37+%28DRAFT%29+Apache+Cassandra+Official+Repair+Solution
> > )
> > > and included all the solutions you mentioned below.
> > > >
> > > >>>>>>>>
> > >
> > > >>>>>>>>
> > > >
> > > >>>>>>>>
> > >
> > > >>>>>>>> Jaydeep
> > > >
> > > >>>>>>>>
> > >
> > > >>>>>>>>
> > > >
> > > >>>>>>>>
> > >
> > > >>>>>>>> On Thu, Feb 22, 2024 at 9:33 AM Josh McKenzie
> > > <[jmckenzie@apache.org](mailto:jmckenzie@apache.org)> wrote:
> > > >
> > > >>>>>>>>
> > >
> > > >>>>>>>>> __
> > > >
> > > >>>>>>>>>
> > >
> > > >>>>>>>>> Very late response from me here (basically necro'ing this
> > thread).
> > > >
> > > >>>>>>>>>
> > >
> > > >>>>>>>>>
> > > >
> > > >>>>>>>>>
> > >
> > > >>>>>>>>> I think it'd be useful to get this condensed into a CEP that
> > we can
> > > then discuss in that format. It's clearly something we all agree we need
> > and
> > > having an implementation that works, even if it's not in your preferred
> > > execution domain, is vastly better than nothing IMO.
> > > >
> > > >>>>>>>>>
> > >
> > > >>>>>>>>>
> > > >
> > > >>>>>>>>>
> > >
> > > >>>>>>>>> I don't have cycles (nor background ;) ) to do that, but it
> > sounds
> > > like you do Jaydeep given the implementation you have on a private fork +
> > > design.
> > > >
> > > >>>>>>>>>
> > >
> > > >>>>>>>>>
> > > >
> > > >>>>>>>>>
> > >
> > > >>>>>>>>> A non-exhaustive list of things that might be useful
> > incorporating
> > > into or referencing from a CEP:
> > > >
> > > >>>>>>>>>
> > >
> > > >>>>>>>>> Slack thread: <https://the-
> > > asf.slack.com/archives/CK23JSY2K/p1690225062383619>
> > > >
> > > >>>>>>>>>
> > >
> > > >>>>>>>>> Joey's old C* ticket:
> > > <https://issues.apache.org/jira/browse/CASSANDRA-14346>
> > > >
> > > >>>>>>>>>
> > >
> > > >>>>>>>>> Even older automatic repair scheduling:
> > > <https://issues.apache.org/jira/browse/CASSANDRA-10070>
> > > >
> > > >>>>>>>>>
> > >
> > > >>>>>>>>> Your design gdoc: <
> > https://docs.google.com/document/d/1CJWxjEi-
> > > mBABPMZ3VWJ9w5KavWfJETAGxfUpsViPcPo/edit#heading=h.r112r46toau0>
> > > >
> > > >>>>>>>>>
> > >
> > > >>>>>>>>> PR with automated repair:
> > > <
> > https://github.com/jaydeepkumar1984/cassandra/commit/ef6456d652c0d07cf29d88dfea03b73704814c2c>
> >
> > > >
> > > >>>>>>>>>
> > >
> > > >>>>>>>>>
> > > >
> > > >>>>>>>>>
> > >
> > > >>>>>>>>> My intuition is that we're all basically in agreement that
> > this is
> > > something the DB needs, we're all willing to bikeshed for our personal
> > > preference on where it lives and how it's implemented, and at the end of
> > the
> > > day, code talks. I don't think anyone's said they'll die on the hill of
> > > implementation details, so that feels like CEP time to me.
> > > >
> > > >>>>>>>>>
> > >
> > > >>>>>>>>>
> > > >
> > > >>>>>>>>>
> > >
> > > >>>>>>>>> If you were willing and able to get a CEP together for
> > automated
> > > repair based on the above material, given you've done the work and have
> > the
> > > proof points it's working at scale, I think this would be a _huge
> > > contribution_ to the community.
> > > >
> > > >>>>>>>>>
> > >
> > > >>>>>>>>>
> > > >
> > > >>>>>>>>>
> > >
> > > >>>>>>>>> On Thu, Aug 24, 2023, at 7:26 PM, Jaydeep Chovatia wrote:
> > > >
> > > >>>>>>>>>
> > >
> > > >>>>>>>>>> Is anyone going to file an official CEP for this?
> > > >
> > > >>>>>>>>>>
> > >
> > > >>>>>>>>>> As mentioned in this email thread, here is one of the
> > solution's
> > > [design doc](https://docs.google.com/document/d/1CJWxjEi-
> > > mBABPMZ3VWJ9w5KavWfJETAGxfUpsViPcPo/edit#heading=h.r112r46toau0) and
> > source
> > > code on a private Apache Cassandra patch. Could you go through it and
> > let me
> > > know what you think?
> > > >
> > > >>>>>>>>>>
> > >
> > > >>>>>>>>>>
> > > >
> > > >>>>>>>>>>
> > >
> > > >>>>>>>>>> Jaydeep
> > > >
> > > >>>>>>>>>>
> > >
> > > >>>>>>>>>>
> > > >
> > > >>>>>>>>>>
> > >
> > > >>>>>>>>>> On Wed, Aug 2, 2023 at 3:54 PM Jon Haddad
> > > <[rustyrazorblade@apache.org](mailto:rustyrazorblade@apache.org)>
> > wrote:
> > > >
> > > >>>>>>>>>>
> > >
> > > >>>>>>>>>>> > That said I would happily support an effort to bring repair
> > > scheduling to the sidecar immediately. This has nothing blocking it, and
> > would
> > > potentially enable the sidecar to provide an official repair scheduling
> > > solution that is compatible with current or even previous versions of the
> > > database.
> > > >
> > > >>>>>>>>>>>
> > >
> > > >>>>>>>>>>>
> > > >
> > > >>>>>>>>>>>
> > >
> > > >>>>>>>>>>> This is something I hadn't thought much about, and is a
> > pretty
> > > good argument for using the sidecar initially. There's a lot of
> > deployments
> > > out there and having an official repair option would be a big win.
> > > >
> > > >>>>>>>>>>>
> > >
> > > >>>>>>>>>>>
> > > >
> > > >>>>>>>>>>>
> > >
> > > >>>>>>>>>>>
> > > >
> > > >>>>>>>>>>>
> > >
> > > >>>>>>>>>>> On 2023/07/26 23:20:07 "C. Scott Andreas" wrote:
> > > >
> > > >>>>>>>>>>>
> > >
> > > >>>>>>>>>>> > I agree that it would be ideal for Cassandra to have a
> > repair
> > > scheduler in-DB.
> > > >
> > > >>>>>>>>>>>
> > >
> > > >>>>>>>>>>> >
> > > >
> > > >>>>>>>>>>>
> > >
> > > >>>>>>>>>>> > That said I would happily support an effort to bring repair
> > > scheduling to the sidecar immediately. This has nothing blocking it, and
> > would
> > > potentially enable the sidecar to provide an official repair scheduling
> > > solution that is compatible with current or even previous versions of the
> > > database.
> > > >
> > > >>>>>>>>>>>
> > >
> > > >>>>>>>>>>> >
> > > >
> > > >>>>>>>>>>>
> > >
> > > >>>>>>>>>>> > Once TCM has landed, we’ll have much stronger primitives
> > for
> > > repair orchestration in the database itself. But I don’t think that
> > should
> > > block progress on a repair scheduling solution in the sidecar, and there
> > is
> > > nothing that would prevent someone from continuing to use a sidecar-based
> > > solution in perpetuity if they preferred.
> > > >
> > > >>>>>>>>>>>
> > >
> > > >>>>>>>>>>> >
> > > >
> > > >>>>>>>>>>>
> > >
> > > >>>>>>>>>>> > \- Scott
> > > >
> > > >>>>>>>>>>>
> > >
> > > >>>>>>>>>>> >
> > > >
> > > >>>>>>>>>>>
> > >
> > > >>>>>>>>>>> > > On Jul 26, 2023, at 3:25 PM, Jon Haddad
> > > <[rustyrazorblade@apache.org](mailto:rustyrazorblade@apache.org)>
> > wrote:
> > > >
> > > >>>>>>>>>>>
> > >
> > > >>>>>>>>>>> > >
> > > >
> > > >>>>>>>>>>>
> > >
> > > >>>>>>>>>>> > > I'm 100% in favor of repair being part of the core DB,
> > not
> > > the sidecar. The current (and past) state of things where running the DB
> > > correctly *requires* running a separate process (either community
> > maintained
> > > or official C* sidecar) is incredibly painful for folks. The idea that
> > your
> > > data integrity needs to be opt-in has never made sense to me from the
> > > perspective of either the product or the end user.
> > > >
> > > >>>>>>>>>>>
> > >
> > > >>>>>>>>>>> > >
> > > >
> > > >>>>>>>>>>>
> > >
> > > >>>>>>>>>>> > > I've worked with way too many teams that have either
> > > configured this incorrectly or not at all.
> > > >
> > > >>>>>>>>>>>
> > >
> > > >>>>>>>>>>> > >
> > > >
> > > >>>>>>>>>>>
> > >
> > > >>>>>>>>>>> > > Ideally Cassandra would ship with repair built in and on
> > by
> > > default. Power users can disable if they want to continue to maintain
> > their
> > > own repair tooling for some reason.
> > > >
> > > >>>>>>>>>>>
> > >
> > > >>>>>>>>>>> > >
> > > >
> > > >>>>>>>>>>>
> > >
> > > >>>>>>>>>>> > > Jon
> > > >
> > > >>>>>>>>>>>
> > >
> > > >>>>>>>>>>> > >
> > > >
> > > >>>>>>>>>>>
> > >
> > > >>>>>>>>>>> > >> On 2023/07/24 20:44:14 German Eichberger via dev
> > wrote:
> > > >
> > > >>>>>>>>>>>
> > >
> > > >>>>>>>>>>> > >> All,
> > > >
> > > >>>>>>>>>>>
> > >
> > > >>>>>>>>>>> > >> We had a brief discussion in [2] about the Uber article
> > [1]
> > > where they talk about having integrated repair into Cassandra and how
> > great
> > > that is. I expressed my disappointment that they didn't work with the
> > > community on that (Uber, if you are listening time to make amends 🙂)
> > and it
> > > turns out Joey already had the idea and wrote the code [3] - so I wanted
> > to
> > > start a discussion to gauge interest and maybe how to revive that
> > effort.
> > > >
> > > >>>>>>>>>>>
> > >
> > > >>>>>>>>>>> > >> Thanks,
> > > >
> > > >>>>>>>>>>>
> > >
> > > >>>>>>>>>>> > >> German
> > > >
> > > >>>>>>>>>>>
> > >
> > > >>>>>>>>>>> > >> [1] <
> > https://www.uber.com/blog/how-uber-optimized-cassandra-
> > > operations-at-scale/>
> > > >
> > > >>>>>>>>>>>
> > >
> > > >>>>>>>>>>> > >> [2] <https://the-
> > > asf.slack.com/archives/CK23JSY2K/p1690225062383619>
> > > >
> > > >>>>>>>>>>>
> > >
> > > >>>>>>>>>>> > >> [3] <
> > https://issues.apache.org/jira/browse/CASSANDRA-14346>
> > > >
> > > >>>>>>>>>>>
> > >
> > > >>>>>>>>>>> >
> > > >
> > > >>>>>>>>>
> > >
> > > >>>>>>>>>
> > > >
> > > >>>>>>
> > >
> > > >>>>>>
> > > >
> > >
> > >
> >
>
Re: [Discuss] Repair inside C*
Posted by Jaydeep Chovatia <ch...@gmail.com>.
>Jaydeep, do you have any metrics on your clusters comparing them before
and after introducing repair scheduling into the Cassandra process?
Yes, I had made some comparisons when I started rolling this feature out to
our production five years ago :) Here are the details:
*The Scheduling*
The scheduling itself is exceptionally lightweight, as only one additional
thread monitors the repair activity, updating the status to a system table
once every few minutes or so. So, it does not appear anywhere in the CPU
charts, etc. Unfortunately, I do not have those graphs now, but I can do a
quick comparison if it helps!
*The Repair Itself*
As we all know, the Cassandra repair algorithm is a heavy-weight process
due to Merkle tree/streaming, etc., no matter how we schedule it. But it is
an orthogonal topic and folks are already discussing creating a new CEP.
Jaydeep
On Mon, Oct 21, 2024 at 10:02 AM Francisco Guerrero <fr...@apache.org>
wrote:
> Jaydeep, do you have any metrics on your clusters comparing them before
> and after introducing repair scheduling into the Cassandra process?
>
> On 2024/10/21 16:57:57 "J. D. Jordan" wrote:
> > Sounds good. Just wanted to bring it up. I agree that the scheduling bit
> is
> > pretty light weight and the ideal would be to bring the whole of the
> repair
> > external, which is a much bigger can of worms to open.
> >
> >
> >
> > -Jeremiah
> >
> >
> >
> > > On Oct 21, 2024, at 11:21 AM, Chris Lohfink <cl...@gmail.com>
> wrote:
> > >
> > >
> >
> > >
> > >
> > > > I actually think we should be looking at how we can move things out
> of the
> > > database process.
> > >
> > >
> > >
> > >
> > >
> > > While worth pursuing, I think we would need a different CEP just to
> figure
> > > out how to do that. Not only is there a lot of infrastructure
> difficulty in
> > > running multi process, the inter app communication needs to be figured
> out
> > > better then JMX. Even the sidecar we dont have a solid story on how to
> > > ensure both are running or anything yet. It's up to each app owner to
> figure
> > > it out. Once we have a good thing in place I think we can start moving
> > > compactions, repairs, etc out of the database. Even then it's the
> _repairs_
> > > that is expensive, not the scheduling.
> > >
> > >
> > >
> > >
> > > On Mon, Oct 21, 2024 at 9:45 AM Jeremiah Jordan
> > > <[jeremiah.jordan@gmail.com](mailto:jeremiah.jordan@gmail.com)>
> wrote:
> > >
> > >
> >
> > >> I love the idea of a repair service being there by default for an
> install
> > of C*. My main concern here is that it is putting more services into
> the main
> > database process. I actually think we should be looking at how we can
> move
> > things out of the database process. The C* process being a giant
> monolith has
> > always been a pain point. Is there anyway it makes sense for this to be
> an
> > external process rather than a new thread pool inside the C* process?
> >
> > >>
> >
> > >>
> > >
> > >>
> >
> > >> -Jeremiah Jordan
> >
> > >>
> >
> > >>
> > >
> > >>
> >
> > >> On Oct 18, 2024 at 2:58:15 PM, Mick Semb Wever
> > <[mck@apache.org](mailto:mck@apache.org)> wrote:
> > >
> > >>
> >
> > >>>
> > >
> > >>>
> >
> > >>> This is looking strong, thanks Jaydeep.
> >
> > >>>
> >
> > >>>
> > >
> > >>>
> >
> > >>> I would suggest folk take a look at the design doc and the PR in the
> CEP.
> > A lot is there (that I have completely missed).
> >
> > >>>
> >
> > >>>
> > >
> > >>>
> >
> > >>> I would especially ask all authors of prior art (Reaper, DSE
> nodesync,
> > ecchronos) to take a final review of the proposal
> > >
> > >>>
> >
> > >>>
> > >
> > >>>
> >
> > >>> Jaydeep, can we ask for a two week window while we reach out to these
> > people ? There's a lot of prior art in this space, and it feels like
> we're in
> > a good place now where it's clear this has legs and we can use that to
> bring
> > folk in and make sure there's no remaining blindspots.
> >
> > >>>
> >
> > >>>
> > >
> > >>>
> >
> > >>>
> > >
> > >>>
> >
> > >>> On Fri, 18 Oct 2024 at 01:40, Jaydeep Chovatia
> > <[chovatia.jaydeep@gmail.com](mailto:chovatia.jaydeep@gmail.com)>
> wrote:
> > >
> > >>>
> >
> > >>>> Sorry, there is a typo in the CEP-37 link; here is the correct
> > [link](
> https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-37+Apache+Cassandra+Unified+Repair+Solution
> )
> >
> > >>>>
> >
> > >>>>
> > >
> > >>>>
> >
> > >>>>
> > >
> > >>>>
> >
> > >>>> On Thu, Oct 17, 2024 at 4:36 PM Jaydeep Chovatia
> > <[chovatia.jaydeep@gmail.com](mailto:chovatia.jaydeep@gmail.com)>
> wrote:
> > >
> > >>>>
> >
> > >>>>> First, thank you for your patience while we strengthened the
> CEP-37.
> >
> > >>>>>
> >
> > >>>>>
> > >
> > >>>>>
> >
> > >>>>> Over the last eight months, Chris Lohfink, Andy Tolbert, Josh
> McKenzie,
> > Dinesh Joshi, Kristijonas Zalys, and I have done tons of work (online
> > discussions/a dedicated Slack channel
> #cassandra-repair-scheduling-cep37) to
> > come up with the best possible design that not only significantly
> simplifies
> > repair operations but also includes the most common features that
> everyone
> > will benefit from running at Scale.
> >
> > >>>>>
> >
> > >>>>> For example,
> >
> > >>>>>
> >
> > >>>>> * Apache Cassandra must be capable of running multiple repair
> types,
> > such as Full, Incremental, Paxos, and Preview - so the framework should
> be
> > easily extendable with no additional overhead from the operator’s point
> of
> > view.
> >
> > >>>>>
> >
> > >>>>> * An easy way to extend the token-split calculation algorithm
> with a
> > default implementation should exist.
> >
> > >>>>>
> >
> > >>>>> * Running incremental repair reliably at Scale is pretty
> challenging,
> > so we need to place safeguards, such as migration/rollback w/o restart
> and
> > stopping incremental repair automatically if the disk is about to get
> full.
> >
> > >>>>>
> >
> > >>>>>
> >
> > >>>>>
> >
> > >>>>> We are glad to inform you that CEP-37 (i.e., Repair inside
> Cassandra) is
> > now officially ready for review after multiple rounds of design,
> testing, code
> > reviews, documentation reviews, and, more importantly, validation that
> it runs
> > at Scale!
> >
> > >>>>>
> >
> > >>>>>
> > >
> > >>>>>
> >
> > >>>>> Some facts about CEP-37.
> >
> > >>>>>
> >
> > >>>>> * Multiple members have verified all aspects of CEP-37 numerous
> times.
> >
> > >>>>>
> >
> > >>>>> * The design proposed in CEP-37 has been thoroughly tried and
> tested
> > on an immense scale (hundreds of unique Cassandra clusters, tens of
> thousands
> > of Cassandra nodes, with tens of millions of QPS) on top of 4.1
> open-source
> > for more than five years; please see more details[
> > here](
> https://www.uber.com/en-US/blog/how-uber-optimized-cassandra-operations-
> > at-scale/).
> >
> > >>>>>
> >
> > >>>>> * The following
> > [presentation](
> https://docs.google.com/presentation/d/1Zilww9c7LihHULk_ckErI2s4XbObxjWknKqRtbvHyZc/edit#slide=id.g30a4fd4fcf7_0_13
> )
> > highlights the rigorous applied to CEP-37, which was given during last
> week’s
> > Apache Cassandra Bay Area [Meetup](
> https://www.meetup.com/apache-cassandra-
> > bay-area/events/303469006/),
> >
> > >>>>>
> >
> > >>>>>
> >
> >
> > >
> > >>>>>
> >
> > >>>>> Since things are massively overhauled, we believe it is almost
> ready for
> > a final pass pre-VOTE. We would like you to please review the
> > [CEP-37](
> https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-37+Apache+Cassandra+Unified+Repair+Solution\
> <https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-37+Apache+Cassandra+Unified+Repair+Solution%5C>
> ))
> > and the associated detailed design
> > [doc](https://docs.google.com/document/d/1CJWxjEi-
> > mBABPMZ3VWJ9w5KavWfJETAGxfUpsViPcPo/edit#heading=h.r112r46toau0).
> >
> > >>>>>
> >
> > >>>>>
> > >
> > >>>>>
> >
> > >>>>> Thank you everyone!
> >
> > >>>>>
> >
> > >>>>> Chris, Andy, Josh, Dinesh, Kristijonas, and Jaydeep
> >
> > >>>>>
> >
> > >>>>>
> > >
> > >
> > >>>>>
> >
> > >>>>>
> > >
> > >>>>>
> >
> > >>>>> On Thu, Sep 19, 2024 at 11:26 AM Josh McKenzie
> > <[jmckenzie@apache.org](mailto:jmckenzie@apache.org)> wrote:
> > >
> > >>>>>
> >
> > >>>>>> __
> >
> > >>>>>>
> >
> > >>>>>> Not quite; finishing touches on the CEP and design doc are in
> flight
> > (as of last / this week).
> > >
> > >>>>>>
> >
> > >>>>>>
> > >
> > >>>>>>
> >
> > >>>>>> Soon(tm).
> >
> > >>>>>>
> >
> > >>>>>>
> > >
> > >>>>>>
> >
> > >>>>>> On Thu, Sep 19, 2024, at 2:07 PM, Patrick McFadin wrote:
> > >
> > >>>>>>
> >
> > >>>>>>> Is this CEP ready for a VOTE thread?
> > <
> https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-37+%28DRAFT%29+Apache+Cassandra+Unified+Repair+Solution>
>
> > >
> > >>>>>>>
> >
> > >>>>>>>
> > >
> > >>>>>>>
> >
> > >>>>>>> On Sun, Feb 25, 2024 at 12:25 PM Jaydeep Chovatia
> > <[chovatia.jaydeep@gmail.com](mailto:chovatia.jaydeep@gmail.com)>
> wrote:
> > >
> > >>>>>>>
> >
> > >>>>>>>> Thanks, Josh. I've just updated the
> > [CEP](
> https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-37+%28DRAFT%29+Apache+Cassandra+Official+Repair+Solution
> )
> > and included all the solutions you mentioned below.
> > >
> > >>>>>>>>
> >
> > >>>>>>>>
> > >
> > >>>>>>>>
> >
> > >>>>>>>> Jaydeep
> > >
> > >>>>>>>>
> >
> > >>>>>>>>
> > >
> > >>>>>>>>
> >
> > >>>>>>>> On Thu, Feb 22, 2024 at 9:33 AM Josh McKenzie
> > <[jmckenzie@apache.org](mailto:jmckenzie@apache.org)> wrote:
> > >
> > >>>>>>>>
> >
> > >>>>>>>>> __
> > >
> > >>>>>>>>>
> >
> > >>>>>>>>> Very late response from me here (basically necro'ing this
> thread).
> > >
> > >>>>>>>>>
> >
> > >>>>>>>>>
> > >
> > >>>>>>>>>
> >
> > >>>>>>>>> I think it'd be useful to get this condensed into a CEP that
> we can
> > then discuss in that format. It's clearly something we all agree we need
> and
> > having an implementation that works, even if it's not in your preferred
> > execution domain, is vastly better than nothing IMO.
> > >
> > >>>>>>>>>
> >
> > >>>>>>>>>
> > >
> > >>>>>>>>>
> >
> > >>>>>>>>> I don't have cycles (nor background ;) ) to do that, but it
> sounds
> > like you do Jaydeep given the implementation you have on a private fork +
> > design.
> > >
> > >>>>>>>>>
> >
> > >>>>>>>>>
> > >
> > >>>>>>>>>
> >
> > >>>>>>>>> A non-exhaustive list of things that might be useful
> incorporating
> > into or referencing from a CEP:
> > >
> > >>>>>>>>>
> >
> > >>>>>>>>> Slack thread: <https://the-
> > asf.slack.com/archives/CK23JSY2K/p1690225062383619>
> > >
> > >>>>>>>>>
> >
> > >>>>>>>>> Joey's old C* ticket:
> > <https://issues.apache.org/jira/browse/CASSANDRA-14346>
> > >
> > >>>>>>>>>
> >
> > >>>>>>>>> Even older automatic repair scheduling:
> > <https://issues.apache.org/jira/browse/CASSANDRA-10070>
> > >
> > >>>>>>>>>
> >
> > >>>>>>>>> Your design gdoc: <
> https://docs.google.com/document/d/1CJWxjEi-
> > mBABPMZ3VWJ9w5KavWfJETAGxfUpsViPcPo/edit#heading=h.r112r46toau0>
> > >
> > >>>>>>>>>
> >
> > >>>>>>>>> PR with automated repair:
> > <
> https://github.com/jaydeepkumar1984/cassandra/commit/ef6456d652c0d07cf29d88dfea03b73704814c2c>
>
> > >
> > >>>>>>>>>
> >
> > >>>>>>>>>
> > >
> > >>>>>>>>>
> >
> > >>>>>>>>> My intuition is that we're all basically in agreement that
> this is
> > something the DB needs, we're all willing to bikeshed for our personal
> > preference on where it lives and how it's implemented, and at the end of
> the
> > day, code talks. I don't think anyone's said they'll die on the hill of
> > implementation details, so that feels like CEP time to me.
> > >
> > >>>>>>>>>
> >
> > >>>>>>>>>
> > >
> > >>>>>>>>>
> >
> > >>>>>>>>> If you were willing and able to get a CEP together for
> automated
> > repair based on the above material, given you've done the work and have
> the
> > proof points it's working at scale, I think this would be a _huge
> > contribution_ to the community.
> > >
> > >>>>>>>>>
> >
> > >>>>>>>>>
> > >
> > >>>>>>>>>
> >
> > >>>>>>>>> On Thu, Aug 24, 2023, at 7:26 PM, Jaydeep Chovatia wrote:
> > >
> > >>>>>>>>>
> >
> > >>>>>>>>>> Is anyone going to file an official CEP for this?
> > >
> > >>>>>>>>>>
> >
> > >>>>>>>>>> As mentioned in this email thread, here is one of the
> solution's
> > [design doc](https://docs.google.com/document/d/1CJWxjEi-
> > mBABPMZ3VWJ9w5KavWfJETAGxfUpsViPcPo/edit#heading=h.r112r46toau0) and
> source
> > code on a private Apache Cassandra patch. Could you go through it and
> let me
> > know what you think?
> > >
> > >>>>>>>>>>
> >
> > >>>>>>>>>>
> > >
> > >>>>>>>>>>
> >
> > >>>>>>>>>> Jaydeep
> > >
> > >>>>>>>>>>
> >
> > >>>>>>>>>>
> > >
> > >>>>>>>>>>
> >
> > >>>>>>>>>> On Wed, Aug 2, 2023 at 3:54 PM Jon Haddad
> > <[rustyrazorblade@apache.org](mailto:rustyrazorblade@apache.org)>
> wrote:
> > >
> > >>>>>>>>>>
> >
> > >>>>>>>>>>> > That said I would happily support an effort to bring repair
> > scheduling to the sidecar immediately. This has nothing blocking it, and
> would
> > potentially enable the sidecar to provide an official repair scheduling
> > solution that is compatible with current or even previous versions of the
> > database.
> > >
> > >>>>>>>>>>>
> >
> > >>>>>>>>>>>
> > >
> > >>>>>>>>>>>
> >
> > >>>>>>>>>>> This is something I hadn't thought much about, and is a
> pretty
> > good argument for using the sidecar initially. There's a lot of
> deployments
> > out there and having an official repair option would be a big win.
> > >
> > >>>>>>>>>>>
> >
> > >>>>>>>>>>>
> > >
> > >>>>>>>>>>>
> >
> > >>>>>>>>>>>
> > >
> > >>>>>>>>>>>
> >
> > >>>>>>>>>>> On 2023/07/26 23:20:07 "C. Scott Andreas" wrote:
> > >
> > >>>>>>>>>>>
> >
> > >>>>>>>>>>> > I agree that it would be ideal for Cassandra to have a
> repair
> > scheduler in-DB.
> > >
> > >>>>>>>>>>>
> >
> > >>>>>>>>>>> >
> > >
> > >>>>>>>>>>>
> >
> > >>>>>>>>>>> > That said I would happily support an effort to bring repair
> > scheduling to the sidecar immediately. This has nothing blocking it, and
> would
> > potentially enable the sidecar to provide an official repair scheduling
> > solution that is compatible with current or even previous versions of the
> > database.
> > >
> > >>>>>>>>>>>
> >
> > >>>>>>>>>>> >
> > >
> > >>>>>>>>>>>
> >
> > >>>>>>>>>>> > Once TCM has landed, we’ll have much stronger primitives
> for
> > repair orchestration in the database itself. But I don’t think that
> should
> > block progress on a repair scheduling solution in the sidecar, and there
> is
> > nothing that would prevent someone from continuing to use a sidecar-based
> > solution in perpetuity if they preferred.
> > >
> > >>>>>>>>>>>
> >
> > >>>>>>>>>>> >
> > >
> > >>>>>>>>>>>
> >
> > >>>>>>>>>>> > \- Scott
> > >
> > >>>>>>>>>>>
> >
> > >>>>>>>>>>> >
> > >
> > >>>>>>>>>>>
> >
> > >>>>>>>>>>> > > On Jul 26, 2023, at 3:25 PM, Jon Haddad
> > <[rustyrazorblade@apache.org](mailto:rustyrazorblade@apache.org)>
> wrote:
> > >
> > >>>>>>>>>>>
> >
> > >>>>>>>>>>> > >
> > >
> > >>>>>>>>>>>
> >
> > >>>>>>>>>>> > > I'm 100% in favor of repair being part of the core DB,
> not
> > the sidecar. The current (and past) state of things where running the DB
> > correctly *requires* running a separate process (either community
> maintained
> > or official C* sidecar) is incredibly painful for folks. The idea that
> your
> > data integrity needs to be opt-in has never made sense to me from the
> > perspective of either the product or the end user.
> > >
> > >>>>>>>>>>>
> >
> > >>>>>>>>>>> > >
> > >
> > >>>>>>>>>>>
> >
> > >>>>>>>>>>> > > I've worked with way too many teams that have either
> > configured this incorrectly or not at all.
> > >
> > >>>>>>>>>>>
> >
> > >>>>>>>>>>> > >
> > >
> > >>>>>>>>>>>
> >
> > >>>>>>>>>>> > > Ideally Cassandra would ship with repair built in and on
> by
> > default. Power users can disable if they want to continue to maintain
> their
> > own repair tooling for some reason.
> > >
> > >>>>>>>>>>>
> >
> > >>>>>>>>>>> > >
> > >
> > >>>>>>>>>>>
> >
> > >>>>>>>>>>> > > Jon
> > >
> > >>>>>>>>>>>
> >
> > >>>>>>>>>>> > >
> > >
> > >>>>>>>>>>>
> >
> > >>>>>>>>>>> > >> On 2023/07/24 20:44:14 German Eichberger via dev
> wrote:
> > >
> > >>>>>>>>>>>
> >
> > >>>>>>>>>>> > >> All,
> > >
> > >>>>>>>>>>>
> >
> > >>>>>>>>>>> > >> We had a brief discussion in [2] about the Uber article
> [1]
> > where they talk about having integrated repair into Cassandra and how
> great
> > that is. I expressed my disappointment that they didn't work with the
> > community on that (Uber, if you are listening time to make amends 🙂)
> and it
> > turns out Joey already had the idea and wrote the code [3] - so I wanted
> to
> > start a discussion to gauge interest and maybe how to revive that
> effort.
> > >
> > >>>>>>>>>>>
> >
> > >>>>>>>>>>> > >> Thanks,
> > >
> > >>>>>>>>>>>
> >
> > >>>>>>>>>>> > >> German
> > >
> > >>>>>>>>>>>
> >
> > >>>>>>>>>>> > >> [1] <
> https://www.uber.com/blog/how-uber-optimized-cassandra-
> > operations-at-scale/>
> > >
> > >>>>>>>>>>>
> >
> > >>>>>>>>>>> > >> [2] <https://the-
> > asf.slack.com/archives/CK23JSY2K/p1690225062383619>
> > >
> > >>>>>>>>>>>
> >
> > >>>>>>>>>>> > >> [3] <
> https://issues.apache.org/jira/browse/CASSANDRA-14346>
> > >
> > >>>>>>>>>>>
> >
> > >>>>>>>>>>> >
> > >
> > >>>>>>>>>
> >
> > >>>>>>>>>
> > >
> > >>>>>>
> >
> > >>>>>>
> > >
> >
> >
>
Re: [Discuss] Repair inside C*
Posted by Francisco Guerrero <fr...@apache.org>.
Jaydeep, do you have any metrics on your clusters comparing them before
and after introducing repair scheduling into the Cassandra process?
On 2024/10/21 16:57:57 "J. D. Jordan" wrote:
> Sounds good. Just wanted to bring it up. I agree that the scheduling bit is
> pretty light weight and the ideal would be to bring the whole of the repair
> external, which is a much bigger can of worms to open.
>
>
>
> -Jeremiah
>
>
>
> > On Oct 21, 2024, at 11:21 AM, Chris Lohfink <cl...@gmail.com> wrote:
> >
> >
>
> >
> >
> > > I actually think we should be looking at how we can move things out of the
> > database process.
> >
> >
> >
> >
> >
> > While worth pursuing, I think we would need a different CEP just to figure
> > out how to do that. Not only is there a lot of infrastructure difficulty in
> > running multi process, the inter app communication needs to be figured out
> > better then JMX. Even the sidecar we dont have a solid story on how to
> > ensure both are running or anything yet. It's up to each app owner to figure
> > it out. Once we have a good thing in place I think we can start moving
> > compactions, repairs, etc out of the database. Even then it's the _repairs_
> > that is expensive, not the scheduling.
> >
> >
> >
> >
> > On Mon, Oct 21, 2024 at 9:45 AM Jeremiah Jordan
> > <[jeremiah.jordan@gmail.com](mailto:jeremiah.jordan@gmail.com)> wrote:
> >
> >
>
> >> I love the idea of a repair service being there by default for an install
> of C*. My main concern here is that it is putting more services into the main
> database process. I actually think we should be looking at how we can move
> things out of the database process. The C* process being a giant monolith has
> always been a pain point. Is there anyway it makes sense for this to be an
> external process rather than a new thread pool inside the C* process?
>
> >>
>
> >>
> >
> >>
>
> >> -Jeremiah Jordan
>
> >>
>
> >>
> >
> >>
>
> >> On Oct 18, 2024 at 2:58:15 PM, Mick Semb Wever
> <[mck@apache.org](mailto:mck@apache.org)> wrote:
> >
> >>
>
> >>>
> >
> >>>
>
> >>> This is looking strong, thanks Jaydeep.
>
> >>>
>
> >>>
> >
> >>>
>
> >>> I would suggest folk take a look at the design doc and the PR in the CEP.
> A lot is there (that I have completely missed).
>
> >>>
>
> >>>
> >
> >>>
>
> >>> I would especially ask all authors of prior art (Reaper, DSE nodesync,
> ecchronos) to take a final review of the proposal
> >
> >>>
>
> >>>
> >
> >>>
>
> >>> Jaydeep, can we ask for a two week window while we reach out to these
> people ? There's a lot of prior art in this space, and it feels like we're in
> a good place now where it's clear this has legs and we can use that to bring
> folk in and make sure there's no remaining blindspots.
>
> >>>
>
> >>>
> >
> >>>
>
> >>>
> >
> >>>
>
> >>> On Fri, 18 Oct 2024 at 01:40, Jaydeep Chovatia
> <[chovatia.jaydeep@gmail.com](mailto:chovatia.jaydeep@gmail.com)> wrote:
> >
> >>>
>
> >>>> Sorry, there is a typo in the CEP-37 link; here is the correct
> [link](https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-37+Apache+Cassandra+Unified+Repair+Solution)
>
> >>>>
>
> >>>>
> >
> >>>>
>
> >>>>
> >
> >>>>
>
> >>>> On Thu, Oct 17, 2024 at 4:36 PM Jaydeep Chovatia
> <[chovatia.jaydeep@gmail.com](mailto:chovatia.jaydeep@gmail.com)> wrote:
> >
> >>>>
>
> >>>>> First, thank you for your patience while we strengthened the CEP-37.
>
> >>>>>
>
> >>>>>
> >
> >>>>>
>
> >>>>> Over the last eight months, Chris Lohfink, Andy Tolbert, Josh McKenzie,
> Dinesh Joshi, Kristijonas Zalys, and I have done tons of work (online
> discussions/a dedicated Slack channel #cassandra-repair-scheduling-cep37) to
> come up with the best possible design that not only significantly simplifies
> repair operations but also includes the most common features that everyone
> will benefit from running at Scale.
>
> >>>>>
>
> >>>>> For example,
>
> >>>>>
>
> >>>>> * Apache Cassandra must be capable of running multiple repair types,
> such as Full, Incremental, Paxos, and Preview - so the framework should be
> easily extendable with no additional overhead from the operator’s point of
> view.
>
> >>>>>
>
> >>>>> * An easy way to extend the token-split calculation algorithm with a
> default implementation should exist.
>
> >>>>>
>
> >>>>> * Running incremental repair reliably at Scale is pretty challenging,
> so we need to place safeguards, such as migration/rollback w/o restart and
> stopping incremental repair automatically if the disk is about to get full.
>
> >>>>>
>
> >>>>>
>
> >>>>>
>
> >>>>> We are glad to inform you that CEP-37 (i.e., Repair inside Cassandra) is
> now officially ready for review after multiple rounds of design, testing, code
> reviews, documentation reviews, and, more importantly, validation that it runs
> at Scale!
>
> >>>>>
>
> >>>>>
> >
> >>>>>
>
> >>>>> Some facts about CEP-37.
>
> >>>>>
>
> >>>>> * Multiple members have verified all aspects of CEP-37 numerous times.
>
> >>>>>
>
> >>>>> * The design proposed in CEP-37 has been thoroughly tried and tested
> on an immense scale (hundreds of unique Cassandra clusters, tens of thousands
> of Cassandra nodes, with tens of millions of QPS) on top of 4.1 open-source
> for more than five years; please see more details[
> here](https://www.uber.com/en-US/blog/how-uber-optimized-cassandra-operations-
> at-scale/).
>
> >>>>>
>
> >>>>> * The following
> [presentation](https://docs.google.com/presentation/d/1Zilww9c7LihHULk_ckErI2s4XbObxjWknKqRtbvHyZc/edit#slide=id.g30a4fd4fcf7_0_13)
> highlights the rigorous applied to CEP-37, which was given during last week’s
> Apache Cassandra Bay Area [Meetup](https://www.meetup.com/apache-cassandra-
> bay-area/events/303469006/),
>
> >>>>>
>
> >>>>>
>
>
> >
> >>>>>
>
> >>>>> Since things are massively overhauled, we believe it is almost ready for
> a final pass pre-VOTE. We would like you to please review the
> [CEP-37](https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-37+Apache+Cassandra+Unified+Repair+Solution\))
> and the associated detailed design
> [doc](https://docs.google.com/document/d/1CJWxjEi-
> mBABPMZ3VWJ9w5KavWfJETAGxfUpsViPcPo/edit#heading=h.r112r46toau0).
>
> >>>>>
>
> >>>>>
> >
> >>>>>
>
> >>>>> Thank you everyone!
>
> >>>>>
>
> >>>>> Chris, Andy, Josh, Dinesh, Kristijonas, and Jaydeep
>
> >>>>>
>
> >>>>>
> >
> >
> >>>>>
>
> >>>>>
> >
> >>>>>
>
> >>>>> On Thu, Sep 19, 2024 at 11:26 AM Josh McKenzie
> <[jmckenzie@apache.org](mailto:jmckenzie@apache.org)> wrote:
> >
> >>>>>
>
> >>>>>> __
>
> >>>>>>
>
> >>>>>> Not quite; finishing touches on the CEP and design doc are in flight
> (as of last / this week).
> >
> >>>>>>
>
> >>>>>>
> >
> >>>>>>
>
> >>>>>> Soon(tm).
>
> >>>>>>
>
> >>>>>>
> >
> >>>>>>
>
> >>>>>> On Thu, Sep 19, 2024, at 2:07 PM, Patrick McFadin wrote:
> >
> >>>>>>
>
> >>>>>>> Is this CEP ready for a VOTE thread?
> <https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-37+%28DRAFT%29+Apache+Cassandra+Unified+Repair+Solution>
> >
> >>>>>>>
>
> >>>>>>>
> >
> >>>>>>>
>
> >>>>>>> On Sun, Feb 25, 2024 at 12:25 PM Jaydeep Chovatia
> <[chovatia.jaydeep@gmail.com](mailto:chovatia.jaydeep@gmail.com)> wrote:
> >
> >>>>>>>
>
> >>>>>>>> Thanks, Josh. I've just updated the
> [CEP](https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-37+%28DRAFT%29+Apache+Cassandra+Official+Repair+Solution)
> and included all the solutions you mentioned below.
> >
> >>>>>>>>
>
> >>>>>>>>
> >
> >>>>>>>>
>
> >>>>>>>> Jaydeep
> >
> >>>>>>>>
>
> >>>>>>>>
> >
> >>>>>>>>
>
> >>>>>>>> On Thu, Feb 22, 2024 at 9:33 AM Josh McKenzie
> <[jmckenzie@apache.org](mailto:jmckenzie@apache.org)> wrote:
> >
> >>>>>>>>
>
> >>>>>>>>> __
> >
> >>>>>>>>>
>
> >>>>>>>>> Very late response from me here (basically necro'ing this thread).
> >
> >>>>>>>>>
>
> >>>>>>>>>
> >
> >>>>>>>>>
>
> >>>>>>>>> I think it'd be useful to get this condensed into a CEP that we can
> then discuss in that format. It's clearly something we all agree we need and
> having an implementation that works, even if it's not in your preferred
> execution domain, is vastly better than nothing IMO.
> >
> >>>>>>>>>
>
> >>>>>>>>>
> >
> >>>>>>>>>
>
> >>>>>>>>> I don't have cycles (nor background ;) ) to do that, but it sounds
> like you do Jaydeep given the implementation you have on a private fork +
> design.
> >
> >>>>>>>>>
>
> >>>>>>>>>
> >
> >>>>>>>>>
>
> >>>>>>>>> A non-exhaustive list of things that might be useful incorporating
> into or referencing from a CEP:
> >
> >>>>>>>>>
>
> >>>>>>>>> Slack thread: <https://the-
> asf.slack.com/archives/CK23JSY2K/p1690225062383619>
> >
> >>>>>>>>>
>
> >>>>>>>>> Joey's old C* ticket:
> <https://issues.apache.org/jira/browse/CASSANDRA-14346>
> >
> >>>>>>>>>
>
> >>>>>>>>> Even older automatic repair scheduling:
> <https://issues.apache.org/jira/browse/CASSANDRA-10070>
> >
> >>>>>>>>>
>
> >>>>>>>>> Your design gdoc: <https://docs.google.com/document/d/1CJWxjEi-
> mBABPMZ3VWJ9w5KavWfJETAGxfUpsViPcPo/edit#heading=h.r112r46toau0>
> >
> >>>>>>>>>
>
> >>>>>>>>> PR with automated repair:
> <https://github.com/jaydeepkumar1984/cassandra/commit/ef6456d652c0d07cf29d88dfea03b73704814c2c>
> >
> >>>>>>>>>
>
> >>>>>>>>>
> >
> >>>>>>>>>
>
> >>>>>>>>> My intuition is that we're all basically in agreement that this is
> something the DB needs, we're all willing to bikeshed for our personal
> preference on where it lives and how it's implemented, and at the end of the
> day, code talks. I don't think anyone's said they'll die on the hill of
> implementation details, so that feels like CEP time to me.
> >
> >>>>>>>>>
>
> >>>>>>>>>
> >
> >>>>>>>>>
>
> >>>>>>>>> If you were willing and able to get a CEP together for automated
> repair based on the above material, given you've done the work and have the
> proof points it's working at scale, I think this would be a _huge
> contribution_ to the community.
> >
> >>>>>>>>>
>
> >>>>>>>>>
> >
> >>>>>>>>>
>
> >>>>>>>>> On Thu, Aug 24, 2023, at 7:26 PM, Jaydeep Chovatia wrote:
> >
> >>>>>>>>>
>
> >>>>>>>>>> Is anyone going to file an official CEP for this?
> >
> >>>>>>>>>>
>
> >>>>>>>>>> As mentioned in this email thread, here is one of the solution's
> [design doc](https://docs.google.com/document/d/1CJWxjEi-
> mBABPMZ3VWJ9w5KavWfJETAGxfUpsViPcPo/edit#heading=h.r112r46toau0) and source
> code on a private Apache Cassandra patch. Could you go through it and let me
> know what you think?
> >
> >>>>>>>>>>
>
> >>>>>>>>>>
> >
> >>>>>>>>>>
>
> >>>>>>>>>> Jaydeep
> >
> >>>>>>>>>>
>
> >>>>>>>>>>
> >
> >>>>>>>>>>
>
> >>>>>>>>>> On Wed, Aug 2, 2023 at 3:54 PM Jon Haddad
> <[rustyrazorblade@apache.org](mailto:rustyrazorblade@apache.org)> wrote:
> >
> >>>>>>>>>>
>
> >>>>>>>>>>> > That said I would happily support an effort to bring repair
> scheduling to the sidecar immediately. This has nothing blocking it, and would
> potentially enable the sidecar to provide an official repair scheduling
> solution that is compatible with current or even previous versions of the
> database.
> >
> >>>>>>>>>>>
>
> >>>>>>>>>>>
> >
> >>>>>>>>>>>
>
> >>>>>>>>>>> This is something I hadn't thought much about, and is a pretty
> good argument for using the sidecar initially. There's a lot of deployments
> out there and having an official repair option would be a big win.
> >
> >>>>>>>>>>>
>
> >>>>>>>>>>>
> >
> >>>>>>>>>>>
>
> >>>>>>>>>>>
> >
> >>>>>>>>>>>
>
> >>>>>>>>>>> On 2023/07/26 23:20:07 "C. Scott Andreas" wrote:
> >
> >>>>>>>>>>>
>
> >>>>>>>>>>> > I agree that it would be ideal for Cassandra to have a repair
> scheduler in-DB.
> >
> >>>>>>>>>>>
>
> >>>>>>>>>>> >
> >
> >>>>>>>>>>>
>
> >>>>>>>>>>> > That said I would happily support an effort to bring repair
> scheduling to the sidecar immediately. This has nothing blocking it, and would
> potentially enable the sidecar to provide an official repair scheduling
> solution that is compatible with current or even previous versions of the
> database.
> >
> >>>>>>>>>>>
>
> >>>>>>>>>>> >
> >
> >>>>>>>>>>>
>
> >>>>>>>>>>> > Once TCM has landed, we’ll have much stronger primitives for
> repair orchestration in the database itself. But I don’t think that should
> block progress on a repair scheduling solution in the sidecar, and there is
> nothing that would prevent someone from continuing to use a sidecar-based
> solution in perpetuity if they preferred.
> >
> >>>>>>>>>>>
>
> >>>>>>>>>>> >
> >
> >>>>>>>>>>>
>
> >>>>>>>>>>> > \- Scott
> >
> >>>>>>>>>>>
>
> >>>>>>>>>>> >
> >
> >>>>>>>>>>>
>
> >>>>>>>>>>> > > On Jul 26, 2023, at 3:25 PM, Jon Haddad
> <[rustyrazorblade@apache.org](mailto:rustyrazorblade@apache.org)> wrote:
> >
> >>>>>>>>>>>
>
> >>>>>>>>>>> > >
> >
> >>>>>>>>>>>
>
> >>>>>>>>>>> > > I'm 100% in favor of repair being part of the core DB, not
> the sidecar. The current (and past) state of things where running the DB
> correctly *requires* running a separate process (either community maintained
> or official C* sidecar) is incredibly painful for folks. The idea that your
> data integrity needs to be opt-in has never made sense to me from the
> perspective of either the product or the end user.
> >
> >>>>>>>>>>>
>
> >>>>>>>>>>> > >
> >
> >>>>>>>>>>>
>
> >>>>>>>>>>> > > I've worked with way too many teams that have either
> configured this incorrectly or not at all.
> >
> >>>>>>>>>>>
>
> >>>>>>>>>>> > >
> >
> >>>>>>>>>>>
>
> >>>>>>>>>>> > > Ideally Cassandra would ship with repair built in and on by
> default. Power users can disable if they want to continue to maintain their
> own repair tooling for some reason.
> >
> >>>>>>>>>>>
>
> >>>>>>>>>>> > >
> >
> >>>>>>>>>>>
>
> >>>>>>>>>>> > > Jon
> >
> >>>>>>>>>>>
>
> >>>>>>>>>>> > >
> >
> >>>>>>>>>>>
>
> >>>>>>>>>>> > >> On 2023/07/24 20:44:14 German Eichberger via dev wrote:
> >
> >>>>>>>>>>>
>
> >>>>>>>>>>> > >> All,
> >
> >>>>>>>>>>>
>
> >>>>>>>>>>> > >> We had a brief discussion in [2] about the Uber article [1]
> where they talk about having integrated repair into Cassandra and how great
> that is. I expressed my disappointment that they didn't work with the
> community on that (Uber, if you are listening time to make amends 🙂) and it
> turns out Joey already had the idea and wrote the code [3] - so I wanted to
> start a discussion to gauge interest and maybe how to revive that effort.
> >
> >>>>>>>>>>>
>
> >>>>>>>>>>> > >> Thanks,
> >
> >>>>>>>>>>>
>
> >>>>>>>>>>> > >> German
> >
> >>>>>>>>>>>
>
> >>>>>>>>>>> > >> [1] <https://www.uber.com/blog/how-uber-optimized-cassandra-
> operations-at-scale/>
> >
> >>>>>>>>>>>
>
> >>>>>>>>>>> > >> [2] <https://the-
> asf.slack.com/archives/CK23JSY2K/p1690225062383619>
> >
> >>>>>>>>>>>
>
> >>>>>>>>>>> > >> [3] <https://issues.apache.org/jira/browse/CASSANDRA-14346>
> >
> >>>>>>>>>>>
>
> >>>>>>>>>>> >
> >
> >>>>>>>>>
>
> >>>>>>>>>
> >
> >>>>>>
>
> >>>>>>
> >
>
>
Re: [Discuss] Repair inside C*
Posted by "J. D. Jordan" <je...@gmail.com>.
Sounds good. Just wanted to bring it up. I agree that the scheduling bit is
pretty light weight and the ideal would be to bring the whole of the repair
external, which is a much bigger can of worms to open.
-Jeremiah
> On Oct 21, 2024, at 11:21 AM, Chris Lohfink <cl...@gmail.com> wrote:
>
>
>
>
> > I actually think we should be looking at how we can move things out of the
> database process.
>
>
>
>
>
> While worth pursuing, I think we would need a different CEP just to figure
> out how to do that. Not only is there a lot of infrastructure difficulty in
> running multi process, the inter app communication needs to be figured out
> better then JMX. Even the sidecar we dont have a solid story on how to
> ensure both are running or anything yet. It's up to each app owner to figure
> it out. Once we have a good thing in place I think we can start moving
> compactions, repairs, etc out of the database. Even then it's the _repairs_
> that is expensive, not the scheduling.
>
>
>
>
> On Mon, Oct 21, 2024 at 9:45 AM Jeremiah Jordan
> <[jeremiah.jordan@gmail.com](mailto:jeremiah.jordan@gmail.com)> wrote:
>
>
>> I love the idea of a repair service being there by default for an install
of C*. My main concern here is that it is putting more services into the main
database process. I actually think we should be looking at how we can move
things out of the database process. The C* process being a giant monolith has
always been a pain point. Is there anyway it makes sense for this to be an
external process rather than a new thread pool inside the C* process?
>>
>>
>
>>
>> -Jeremiah Jordan
>>
>>
>
>>
>> On Oct 18, 2024 at 2:58:15 PM, Mick Semb Wever
<[mck@apache.org](mailto:mck@apache.org)> wrote:
>
>>
>>>
>
>>>
>>> This is looking strong, thanks Jaydeep.
>>>
>>>
>
>>>
>>> I would suggest folk take a look at the design doc and the PR in the CEP.
A lot is there (that I have completely missed).
>>>
>>>
>
>>>
>>> I would especially ask all authors of prior art (Reaper, DSE nodesync,
ecchronos) to take a final review of the proposal
>
>>>
>>>
>
>>>
>>> Jaydeep, can we ask for a two week window while we reach out to these
people ? There's a lot of prior art in this space, and it feels like we're in
a good place now where it's clear this has legs and we can use that to bring
folk in and make sure there's no remaining blindspots.
>>>
>>>
>
>>>
>>>
>
>>>
>>> On Fri, 18 Oct 2024 at 01:40, Jaydeep Chovatia
<[chovatia.jaydeep@gmail.com](mailto:chovatia.jaydeep@gmail.com)> wrote:
>
>>>
>>>> Sorry, there is a typo in the CEP-37 link; here is the correct
[link](https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-37+Apache+Cassandra+Unified+Repair+Solution)
>>>>
>>>>
>
>>>>
>>>>
>
>>>>
>>>> On Thu, Oct 17, 2024 at 4:36 PM Jaydeep Chovatia
<[chovatia.jaydeep@gmail.com](mailto:chovatia.jaydeep@gmail.com)> wrote:
>
>>>>
>>>>> First, thank you for your patience while we strengthened the CEP-37.
>>>>>
>>>>>
>
>>>>>
>>>>> Over the last eight months, Chris Lohfink, Andy Tolbert, Josh McKenzie,
Dinesh Joshi, Kristijonas Zalys, and I have done tons of work (online
discussions/a dedicated Slack channel #cassandra-repair-scheduling-cep37) to
come up with the best possible design that not only significantly simplifies
repair operations but also includes the most common features that everyone
will benefit from running at Scale.
>>>>>
>>>>> For example,
>>>>>
>>>>> * Apache Cassandra must be capable of running multiple repair types,
such as Full, Incremental, Paxos, and Preview - so the framework should be
easily extendable with no additional overhead from the operator’s point of
view.
>>>>>
>>>>> * An easy way to extend the token-split calculation algorithm with a
default implementation should exist.
>>>>>
>>>>> * Running incremental repair reliably at Scale is pretty challenging,
so we need to place safeguards, such as migration/rollback w/o restart and
stopping incremental repair automatically if the disk is about to get full.
>>>>>
>>>>>
>>>>>
>>>>> We are glad to inform you that CEP-37 (i.e., Repair inside Cassandra) is
now officially ready for review after multiple rounds of design, testing, code
reviews, documentation reviews, and, more importantly, validation that it runs
at Scale!
>>>>>
>>>>>
>
>>>>>
>>>>> Some facts about CEP-37.
>>>>>
>>>>> * Multiple members have verified all aspects of CEP-37 numerous times.
>>>>>
>>>>> * The design proposed in CEP-37 has been thoroughly tried and tested
on an immense scale (hundreds of unique Cassandra clusters, tens of thousands
of Cassandra nodes, with tens of millions of QPS) on top of 4.1 open-source
for more than five years; please see more details[
here](https://www.uber.com/en-US/blog/how-uber-optimized-cassandra-operations-
at-scale/).
>>>>>
>>>>> * The following
[presentation](https://docs.google.com/presentation/d/1Zilww9c7LihHULk_ckErI2s4XbObxjWknKqRtbvHyZc/edit#slide=id.g30a4fd4fcf7_0_13)
highlights the rigorous applied to CEP-37, which was given during last week’s
Apache Cassandra Bay Area [Meetup](https://www.meetup.com/apache-cassandra-
bay-area/events/303469006/),
>>>>>
>>>>>
>
>>>>>
>>>>> Since things are massively overhauled, we believe it is almost ready for
a final pass pre-VOTE. We would like you to please review the
[CEP-37](https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-37+Apache+Cassandra+Unified+Repair+Solution\))
and the associated detailed design
[doc](https://docs.google.com/document/d/1CJWxjEi-
mBABPMZ3VWJ9w5KavWfJETAGxfUpsViPcPo/edit#heading=h.r112r46toau0).
>>>>>
>>>>>
>
>>>>>
>>>>> Thank you everyone!
>>>>>
>>>>> Chris, Andy, Josh, Dinesh, Kristijonas, and Jaydeep
>>>>>
>>>>>
>
>
>>>>>
>>>>>
>
>>>>>
>>>>> On Thu, Sep 19, 2024 at 11:26 AM Josh McKenzie
<[jmckenzie@apache.org](mailto:jmckenzie@apache.org)> wrote:
>
>>>>>
>>>>>> __
>>>>>>
>>>>>> Not quite; finishing touches on the CEP and design doc are in flight
(as of last / this week).
>
>>>>>>
>>>>>>
>
>>>>>>
>>>>>> Soon(tm).
>>>>>>
>>>>>>
>
>>>>>>
>>>>>> On Thu, Sep 19, 2024, at 2:07 PM, Patrick McFadin wrote:
>
>>>>>>
>>>>>>> Is this CEP ready for a VOTE thread?
<https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-37+%28DRAFT%29+Apache+Cassandra+Unified+Repair+Solution>
>
>>>>>>>
>>>>>>>
>
>>>>>>>
>>>>>>> On Sun, Feb 25, 2024 at 12:25 PM Jaydeep Chovatia
<[chovatia.jaydeep@gmail.com](mailto:chovatia.jaydeep@gmail.com)> wrote:
>
>>>>>>>
>>>>>>>> Thanks, Josh. I've just updated the
[CEP](https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-37+%28DRAFT%29+Apache+Cassandra+Official+Repair+Solution)
and included all the solutions you mentioned below.
>
>>>>>>>>
>>>>>>>>
>
>>>>>>>>
>>>>>>>> Jaydeep
>
>>>>>>>>
>>>>>>>>
>
>>>>>>>>
>>>>>>>> On Thu, Feb 22, 2024 at 9:33 AM Josh McKenzie
<[jmckenzie@apache.org](mailto:jmckenzie@apache.org)> wrote:
>
>>>>>>>>
>>>>>>>>> __
>
>>>>>>>>>
>>>>>>>>> Very late response from me here (basically necro'ing this thread).
>
>>>>>>>>>
>>>>>>>>>
>
>>>>>>>>>
>>>>>>>>> I think it'd be useful to get this condensed into a CEP that we can
then discuss in that format. It's clearly something we all agree we need and
having an implementation that works, even if it's not in your preferred
execution domain, is vastly better than nothing IMO.
>
>>>>>>>>>
>>>>>>>>>
>
>>>>>>>>>
>>>>>>>>> I don't have cycles (nor background ;) ) to do that, but it sounds
like you do Jaydeep given the implementation you have on a private fork +
design.
>
>>>>>>>>>
>>>>>>>>>
>
>>>>>>>>>
>>>>>>>>> A non-exhaustive list of things that might be useful incorporating
into or referencing from a CEP:
>
>>>>>>>>>
>>>>>>>>> Slack thread: <https://the-
asf.slack.com/archives/CK23JSY2K/p1690225062383619>
>
>>>>>>>>>
>>>>>>>>> Joey's old C* ticket:
<https://issues.apache.org/jira/browse/CASSANDRA-14346>
>
>>>>>>>>>
>>>>>>>>> Even older automatic repair scheduling:
<https://issues.apache.org/jira/browse/CASSANDRA-10070>
>
>>>>>>>>>
>>>>>>>>> Your design gdoc: <https://docs.google.com/document/d/1CJWxjEi-
mBABPMZ3VWJ9w5KavWfJETAGxfUpsViPcPo/edit#heading=h.r112r46toau0>
>
>>>>>>>>>
>>>>>>>>> PR with automated repair:
<https://github.com/jaydeepkumar1984/cassandra/commit/ef6456d652c0d07cf29d88dfea03b73704814c2c>
>
>>>>>>>>>
>>>>>>>>>
>
>>>>>>>>>
>>>>>>>>> My intuition is that we're all basically in agreement that this is
something the DB needs, we're all willing to bikeshed for our personal
preference on where it lives and how it's implemented, and at the end of the
day, code talks. I don't think anyone's said they'll die on the hill of
implementation details, so that feels like CEP time to me.
>
>>>>>>>>>
>>>>>>>>>
>
>>>>>>>>>
>>>>>>>>> If you were willing and able to get a CEP together for automated
repair based on the above material, given you've done the work and have the
proof points it's working at scale, I think this would be a _huge
contribution_ to the community.
>
>>>>>>>>>
>>>>>>>>>
>
>>>>>>>>>
>>>>>>>>> On Thu, Aug 24, 2023, at 7:26 PM, Jaydeep Chovatia wrote:
>
>>>>>>>>>
>>>>>>>>>> Is anyone going to file an official CEP for this?
>
>>>>>>>>>>
>>>>>>>>>> As mentioned in this email thread, here is one of the solution's
[design doc](https://docs.google.com/document/d/1CJWxjEi-
mBABPMZ3VWJ9w5KavWfJETAGxfUpsViPcPo/edit#heading=h.r112r46toau0) and source
code on a private Apache Cassandra patch. Could you go through it and let me
know what you think?
>
>>>>>>>>>>
>>>>>>>>>>
>
>>>>>>>>>>
>>>>>>>>>> Jaydeep
>
>>>>>>>>>>
>>>>>>>>>>
>
>>>>>>>>>>
>>>>>>>>>> On Wed, Aug 2, 2023 at 3:54 PM Jon Haddad
<[rustyrazorblade@apache.org](mailto:rustyrazorblade@apache.org)> wrote:
>
>>>>>>>>>>
>>>>>>>>>>> > That said I would happily support an effort to bring repair
scheduling to the sidecar immediately. This has nothing blocking it, and would
potentially enable the sidecar to provide an official repair scheduling
solution that is compatible with current or even previous versions of the
database.
>
>>>>>>>>>>>
>>>>>>>>>>>
>
>>>>>>>>>>>
>>>>>>>>>>> This is something I hadn't thought much about, and is a pretty
good argument for using the sidecar initially. There's a lot of deployments
out there and having an official repair option would be a big win.
>
>>>>>>>>>>>
>>>>>>>>>>>
>
>>>>>>>>>>>
>>>>>>>>>>>
>
>>>>>>>>>>>
>>>>>>>>>>> On 2023/07/26 23:20:07 "C. Scott Andreas" wrote:
>
>>>>>>>>>>>
>>>>>>>>>>> > I agree that it would be ideal for Cassandra to have a repair
scheduler in-DB.
>
>>>>>>>>>>>
>>>>>>>>>>> >
>
>>>>>>>>>>>
>>>>>>>>>>> > That said I would happily support an effort to bring repair
scheduling to the sidecar immediately. This has nothing blocking it, and would
potentially enable the sidecar to provide an official repair scheduling
solution that is compatible with current or even previous versions of the
database.
>
>>>>>>>>>>>
>>>>>>>>>>> >
>
>>>>>>>>>>>
>>>>>>>>>>> > Once TCM has landed, we’ll have much stronger primitives for
repair orchestration in the database itself. But I don’t think that should
block progress on a repair scheduling solution in the sidecar, and there is
nothing that would prevent someone from continuing to use a sidecar-based
solution in perpetuity if they preferred.
>
>>>>>>>>>>>
>>>>>>>>>>> >
>
>>>>>>>>>>>
>>>>>>>>>>> > \- Scott
>
>>>>>>>>>>>
>>>>>>>>>>> >
>
>>>>>>>>>>>
>>>>>>>>>>> > > On Jul 26, 2023, at 3:25 PM, Jon Haddad
<[rustyrazorblade@apache.org](mailto:rustyrazorblade@apache.org)> wrote:
>
>>>>>>>>>>>
>>>>>>>>>>> > >
>
>>>>>>>>>>>
>>>>>>>>>>> > > I'm 100% in favor of repair being part of the core DB, not
the sidecar. The current (and past) state of things where running the DB
correctly *requires* running a separate process (either community maintained
or official C* sidecar) is incredibly painful for folks. The idea that your
data integrity needs to be opt-in has never made sense to me from the
perspective of either the product or the end user.
>
>>>>>>>>>>>
>>>>>>>>>>> > >
>
>>>>>>>>>>>
>>>>>>>>>>> > > I've worked with way too many teams that have either
configured this incorrectly or not at all.
>
>>>>>>>>>>>
>>>>>>>>>>> > >
>
>>>>>>>>>>>
>>>>>>>>>>> > > Ideally Cassandra would ship with repair built in and on by
default. Power users can disable if they want to continue to maintain their
own repair tooling for some reason.
>
>>>>>>>>>>>
>>>>>>>>>>> > >
>
>>>>>>>>>>>
>>>>>>>>>>> > > Jon
>
>>>>>>>>>>>
>>>>>>>>>>> > >
>
>>>>>>>>>>>
>>>>>>>>>>> > >> On 2023/07/24 20:44:14 German Eichberger via dev wrote:
>
>>>>>>>>>>>
>>>>>>>>>>> > >> All,
>
>>>>>>>>>>>
>>>>>>>>>>> > >> We had a brief discussion in [2] about the Uber article [1]
where they talk about having integrated repair into Cassandra and how great
that is. I expressed my disappointment that they didn't work with the
community on that (Uber, if you are listening time to make amends 🙂) and it
turns out Joey already had the idea and wrote the code [3] - so I wanted to
start a discussion to gauge interest and maybe how to revive that effort.
>
>>>>>>>>>>>
>>>>>>>>>>> > >> Thanks,
>
>>>>>>>>>>>
>>>>>>>>>>> > >> German
>
>>>>>>>>>>>
>>>>>>>>>>> > >> [1] <https://www.uber.com/blog/how-uber-optimized-cassandra-
operations-at-scale/>
>
>>>>>>>>>>>
>>>>>>>>>>> > >> [2] <https://the-
asf.slack.com/archives/CK23JSY2K/p1690225062383619>
>
>>>>>>>>>>>
>>>>>>>>>>> > >> [3] <https://issues.apache.org/jira/browse/CASSANDRA-14346>
>
>>>>>>>>>>>
>>>>>>>>>>> >
>
>>>>>>>>>
>>>>>>>>>
>
>>>>>>
>>>>>>
>
Re: [Discuss] Repair inside C*
Posted by Patrick McFadin <pm...@gmail.com>.
> While worth pursuing, I think we would need a different CEP just to figure out how to do that. Not only is there a lot of infrastructure difficulty in running multi process, the inter app communication needs to be figured out better then JMX.
I strongly agree and this is a good time to start considering this
with TCM pending. The subtle difference is how much is in process vs
coordinated by the Cassandra core. Alex and I were talking about the
idea of ETCD in Kubernetes and in the database role, stands as the
source of truth for cluster operations. How many pods? Where are they?
What services map to what pod? etc etc. Separate processes could
operate independently by letting the database be the database, and
Cassandra could be the coordinator and source of truth with a
transactional guarantee of correctness.
On Mon, Oct 21, 2024 at 9:22 AM Chris Lohfink <cl...@gmail.com> wrote:
>
> > I actually think we should be looking at how we can move things out of the database process.
>
> While worth pursuing, I think we would need a different CEP just to figure out how to do that. Not only is there a lot of infrastructure difficulty in running multi process, the inter app communication needs to be figured out better then JMX. Even the sidecar we dont have a solid story on how to ensure both are running or anything yet. It's up to each app owner to figure it out. Once we have a good thing in place I think we can start moving compactions, repairs, etc out of the database. Even then it's the _repairs_ that is expensive, not the scheduling.
>
> On Mon, Oct 21, 2024 at 9:45 AM Jeremiah Jordan <je...@gmail.com> wrote:
>>
>> I love the idea of a repair service being there by default for an install of C*. My main concern here is that it is putting more services into the main database process. I actually think we should be looking at how we can move things out of the database process. The C* process being a giant monolith has always been a pain point. Is there anyway it makes sense for this to be an external process rather than a new thread pool inside the C* process?
>>
>> -Jeremiah Jordan
>>
>> On Oct 18, 2024 at 2:58:15 PM, Mick Semb Wever <mc...@apache.org> wrote:
>>>
>>>
>>> This is looking strong, thanks Jaydeep.
>>>
>>> I would suggest folk take a look at the design doc and the PR in the CEP. A lot is there (that I have completely missed).
>>>
>>> I would especially ask all authors of prior art (Reaper, DSE nodesync, ecchronos) to take a final review of the proposal
>>>
>>> Jaydeep, can we ask for a two week window while we reach out to these people ? There's a lot of prior art in this space, and it feels like we're in a good place now where it's clear this has legs and we can use that to bring folk in and make sure there's no remaining blindspots.
>>>
>>>
>>> On Fri, 18 Oct 2024 at 01:40, Jaydeep Chovatia <ch...@gmail.com> wrote:
>>>>
>>>> Sorry, there is a typo in the CEP-37 link; here is the correct link
>>>>
>>>>
>>>> On Thu, Oct 17, 2024 at 4:36 PM Jaydeep Chovatia <ch...@gmail.com> wrote:
>>>>>
>>>>> First, thank you for your patience while we strengthened the CEP-37.
>>>>>
>>>>>
>>>>> Over the last eight months, Chris Lohfink, Andy Tolbert, Josh McKenzie, Dinesh Joshi, Kristijonas Zalys, and I have done tons of work (online discussions/a dedicated Slack channel #cassandra-repair-scheduling-cep37) to come up with the best possible design that not only significantly simplifies repair operations but also includes the most common features that everyone will benefit from running at Scale.
>>>>>
>>>>> For example,
>>>>>
>>>>> Apache Cassandra must be capable of running multiple repair types, such as Full, Incremental, Paxos, and Preview - so the framework should be easily extendable with no additional overhead from the operator’s point of view.
>>>>>
>>>>> An easy way to extend the token-split calculation algorithm with a default implementation should exist.
>>>>>
>>>>> Running incremental repair reliably at Scale is pretty challenging, so we need to place safeguards, such as migration/rollback w/o restart and stopping incremental repair automatically if the disk is about to get full.
>>>>>
>>>>> We are glad to inform you that CEP-37 (i.e., Repair inside Cassandra) is now officially ready for review after multiple rounds of design, testing, code reviews, documentation reviews, and, more importantly, validation that it runs at Scale!
>>>>>
>>>>>
>>>>> Some facts about CEP-37.
>>>>>
>>>>> Multiple members have verified all aspects of CEP-37 numerous times.
>>>>>
>>>>> The design proposed in CEP-37 has been thoroughly tried and tested on an immense scale (hundreds of unique Cassandra clusters, tens of thousands of Cassandra nodes, with tens of millions of QPS) on top of 4.1 open-source for more than five years; please see more details here.
>>>>>
>>>>> The following presentation highlights the rigorous applied to CEP-37, which was given during last week’s Apache Cassandra Bay Area Meetup,
>>>>>
>>>>>
>>>>> Since things are massively overhauled, we believe it is almost ready for a final pass pre-VOTE. We would like you to please review the CEP-37 and the associated detailed design doc.
>>>>>
>>>>>
>>>>> Thank you everyone!
>>>>>
>>>>> Chris, Andy, Josh, Dinesh, Kristijonas, and Jaydeep
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Thu, Sep 19, 2024 at 11:26 AM Josh McKenzie <jm...@apache.org> wrote:
>>>>>>
>>>>>> Not quite; finishing touches on the CEP and design doc are in flight (as of last / this week).
>>>>>>
>>>>>> Soon(tm).
>>>>>>
>>>>>> On Thu, Sep 19, 2024, at 2:07 PM, Patrick McFadin wrote:
>>>>>>
>>>>>> Is this CEP ready for a VOTE thread? https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-37+%28DRAFT%29+Apache+Cassandra+Unified+Repair+Solution
>>>>>>
>>>>>> On Sun, Feb 25, 2024 at 12:25 PM Jaydeep Chovatia <ch...@gmail.com> wrote:
>>>>>>
>>>>>> Thanks, Josh. I've just updated the CEP and included all the solutions you mentioned below.
>>>>>>
>>>>>> Jaydeep
>>>>>>
>>>>>> On Thu, Feb 22, 2024 at 9:33 AM Josh McKenzie <jm...@apache.org> wrote:
>>>>>>
>>>>>>
>>>>>> Very late response from me here (basically necro'ing this thread).
>>>>>>
>>>>>> I think it'd be useful to get this condensed into a CEP that we can then discuss in that format. It's clearly something we all agree we need and having an implementation that works, even if it's not in your preferred execution domain, is vastly better than nothing IMO.
>>>>>>
>>>>>> I don't have cycles (nor background ;) ) to do that, but it sounds like you do Jaydeep given the implementation you have on a private fork + design.
>>>>>>
>>>>>> A non-exhaustive list of things that might be useful incorporating into or referencing from a CEP:
>>>>>> Slack thread: https://the-asf.slack.com/archives/CK23JSY2K/p1690225062383619
>>>>>> Joey's old C* ticket: https://issues.apache.org/jira/browse/CASSANDRA-14346
>>>>>> Even older automatic repair scheduling: https://issues.apache.org/jira/browse/CASSANDRA-10070
>>>>>> Your design gdoc: https://docs.google.com/document/d/1CJWxjEi-mBABPMZ3VWJ9w5KavWfJETAGxfUpsViPcPo/edit#heading=h.r112r46toau0
>>>>>> PR with automated repair: https://github.com/jaydeepkumar1984/cassandra/commit/ef6456d652c0d07cf29d88dfea03b73704814c2c
>>>>>>
>>>>>> My intuition is that we're all basically in agreement that this is something the DB needs, we're all willing to bikeshed for our personal preference on where it lives and how it's implemented, and at the end of the day, code talks. I don't think anyone's said they'll die on the hill of implementation details, so that feels like CEP time to me.
>>>>>>
>>>>>> If you were willing and able to get a CEP together for automated repair based on the above material, given you've done the work and have the proof points it's working at scale, I think this would be a huge contribution to the community.
>>>>>>
>>>>>> On Thu, Aug 24, 2023, at 7:26 PM, Jaydeep Chovatia wrote:
>>>>>>
>>>>>> Is anyone going to file an official CEP for this?
>>>>>> As mentioned in this email thread, here is one of the solution's design doc and source code on a private Apache Cassandra patch. Could you go through it and let me know what you think?
>>>>>>
>>>>>> Jaydeep
>>>>>>
>>>>>> On Wed, Aug 2, 2023 at 3:54 PM Jon Haddad <ru...@apache.org> wrote:
>>>>>>
>>>>>> > That said I would happily support an effort to bring repair scheduling to the sidecar immediately. This has nothing blocking it, and would potentially enable the sidecar to provide an official repair scheduling solution that is compatible with current or even previous versions of the database.
>>>>>>
>>>>>> This is something I hadn't thought much about, and is a pretty good argument for using the sidecar initially. There's a lot of deployments out there and having an official repair option would be a big win.
>>>>>>
>>>>>>
>>>>>> On 2023/07/26 23:20:07 "C. Scott Andreas" wrote:
>>>>>> > I agree that it would be ideal for Cassandra to have a repair scheduler in-DB.
>>>>>> >
>>>>>> > That said I would happily support an effort to bring repair scheduling to the sidecar immediately. This has nothing blocking it, and would potentially enable the sidecar to provide an official repair scheduling solution that is compatible with current or even previous versions of the database.
>>>>>> >
>>>>>> > Once TCM has landed, we’ll have much stronger primitives for repair orchestration in the database itself. But I don’t think that should block progress on a repair scheduling solution in the sidecar, and there is nothing that would prevent someone from continuing to use a sidecar-based solution in perpetuity if they preferred.
>>>>>> >
>>>>>> > - Scott
>>>>>> >
>>>>>> > > On Jul 26, 2023, at 3:25 PM, Jon Haddad <ru...@apache.org> wrote:
>>>>>> > >
>>>>>> > > I'm 100% in favor of repair being part of the core DB, not the sidecar. The current (and past) state of things where running the DB correctly *requires* running a separate process (either community maintained or official C* sidecar) is incredibly painful for folks. The idea that your data integrity needs to be opt-in has never made sense to me from the perspective of either the product or the end user.
>>>>>> > >
>>>>>> > > I've worked with way too many teams that have either configured this incorrectly or not at all.
>>>>>> > >
>>>>>> > > Ideally Cassandra would ship with repair built in and on by default. Power users can disable if they want to continue to maintain their own repair tooling for some reason.
>>>>>> > >
>>>>>> > > Jon
>>>>>> > >
>>>>>> > >> On 2023/07/24 20:44:14 German Eichberger via dev wrote:
>>>>>> > >> All,
>>>>>> > >> We had a brief discussion in [2] about the Uber article [1] where they talk about having integrated repair into Cassandra and how great that is. I expressed my disappointment that they didn't work with the community on that (Uber, if you are listening time to make amends 🙂) and it turns out Joey already had the idea and wrote the code [3] - so I wanted to start a discussion to gauge interest and maybe how to revive that effort.
>>>>>> > >> Thanks,
>>>>>> > >> German
>>>>>> > >> [1] https://www.uber.com/blog/how-uber-optimized-cassandra-operations-at-scale/
>>>>>> > >> [2] https://the-asf.slack.com/archives/CK23JSY2K/p1690225062383619
>>>>>> > >> [3] https://issues.apache.org/jira/browse/CASSANDRA-14346
>>>>>> >
>>>>>>
>>>>>>
>>>>>>
Re: [Discuss] Repair inside C*
Posted by Jaydeep Chovatia <ch...@gmail.com>.
Just to keep all discussions here, I am copying/pasting the comments
<https://github.com/apache/cassandra/pull/3598#issuecomment-2425935493>
received from @masokol on the PR
[image: image.png]
And my response
<https://github.com/apache/cassandra/pull/3598#issuecomment-2427154086> to
the comments
[image: image.png]
Jaydeep
On Mon, Oct 21, 2024 at 9:22 AM Chris Lohfink <cl...@gmail.com> wrote:
> > I actually think we should be looking at how we can move things out of
> the database process.
>
> While worth pursuing, I think we would need a different CEP just to figure
> out how to do that. Not only is there a lot of infrastructure difficulty in
> running multi process, the inter app communication needs to be figured out
> better then JMX. Even the sidecar we dont have a solid story on how to
> ensure both are running or anything yet. It's up to each app owner to
> figure it out. Once we have a good thing in place I think we can start
> moving compactions, repairs, etc out of the database. Even then it's the
> _repairs_ that is expensive, not the scheduling.
>
> On Mon, Oct 21, 2024 at 9:45 AM Jeremiah Jordan <je...@gmail.com>
> wrote:
>
>> I love the idea of a repair service being there by default for an install
>> of C*. My main concern here is that it is putting more services into the
>> main database process. I actually think we should be looking at how we can
>> move things out of the database process. The C* process being a giant
>> monolith has always been a pain point. Is there anyway it makes sense for
>> this to be an external process rather than a new thread pool inside the C*
>> process?
>>
>> -Jeremiah Jordan
>>
>> On Oct 18, 2024 at 2:58:15 PM, Mick Semb Wever <mc...@apache.org> wrote:
>>
>>>
>>> This is looking strong, thanks Jaydeep.
>>>
>>> I would suggest folk take a look at the design doc and the PR in the
>>> CEP. A lot is there (that I have completely missed).
>>>
>>> I would especially ask all authors of prior art (Reaper, DSE nodesync,
>>> ecchronos) to take a final review of the proposal
>>>
>>> Jaydeep, can we ask for a two week window while we reach out to these
>>> people ? There's a lot of prior art in this space, and it feels like we're
>>> in a good place now where it's clear this has legs and we can use that to
>>> bring folk in and make sure there's no remaining blindspots.
>>>
>>>
>>> On Fri, 18 Oct 2024 at 01:40, Jaydeep Chovatia <
>>> chovatia.jaydeep@gmail.com> wrote:
>>>
>>>> Sorry, there is a typo in the CEP-37 link; here is the correct link
>>>> <https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-37+Apache+Cassandra+Unified+Repair+Solution>
>>>>
>>>>
>>>> On Thu, Oct 17, 2024 at 4:36 PM Jaydeep Chovatia <
>>>> chovatia.jaydeep@gmail.com> wrote:
>>>>
>>>>> First, thank you for your patience while we strengthened the CEP-37.
>>>>>
>>>>>
>>>>> Over the last eight months, Chris Lohfink, Andy Tolbert, Josh
>>>>> McKenzie, Dinesh Joshi, Kristijonas Zalys, and I have done tons of work
>>>>> (online discussions/a dedicated Slack channel
>>>>> #cassandra-repair-scheduling-cep37) to come up with the best possible
>>>>> design that not only significantly simplifies repair operations but also
>>>>> includes the most common features that everyone will benefit from running
>>>>> at Scale.
>>>>>
>>>>> For example,
>>>>>
>>>>> -
>>>>>
>>>>> Apache Cassandra must be capable of running multiple repair types,
>>>>> such as Full, Incremental, Paxos, and Preview - so the framework should be
>>>>> easily extendable with no additional overhead from the operator’s point of
>>>>> view.
>>>>> -
>>>>>
>>>>> An easy way to extend the token-split calculation algorithm with a
>>>>> default implementation should exist.
>>>>> -
>>>>>
>>>>> Running incremental repair reliably at Scale is pretty
>>>>> challenging, so we need to place safeguards, such as migration/rollback w/o
>>>>> restart and stopping incremental repair automatically if the disk is about
>>>>> to get full.
>>>>>
>>>>> We are glad to inform you that CEP-37 (i.e., Repair inside Cassandra)
>>>>> is now officially ready for review after multiple rounds of design,
>>>>> testing, code reviews, documentation reviews, and, more importantly,
>>>>> validation that it runs at Scale!
>>>>>
>>>>>
>>>>> Some facts about CEP-37.
>>>>>
>>>>> -
>>>>>
>>>>> Multiple members have verified all aspects of CEP-37 numerous
>>>>> times.
>>>>> -
>>>>>
>>>>> The design proposed in CEP-37 has been thoroughly tried and tested
>>>>> on an immense scale (hundreds of unique Cassandra clusters, tens of
>>>>> thousands of Cassandra nodes, with tens of millions of QPS) on top of 4.1
>>>>> open-source for more than five years; please see more details here
>>>>> <https://www.uber.com/en-US/blog/how-uber-optimized-cassandra-operations-at-scale/>
>>>>> .
>>>>> -
>>>>>
>>>>> The following presentation
>>>>> <https://docs.google.com/presentation/d/1Zilww9c7LihHULk_ckErI2s4XbObxjWknKqRtbvHyZc/edit#slide=id.g30a4fd4fcf7_0_13>
>>>>> highlights the rigorous applied to CEP-37, which was given during last
>>>>> week’s Apache Cassandra Bay Area Meetup
>>>>> <https://www.meetup.com/apache-cassandra-bay-area/events/303469006/>
>>>>> ,
>>>>>
>>>>>
>>>>> Since things are massively overhauled, we believe it is almost ready
>>>>> for a final pass pre-VOTE. We would like you to please review the
>>>>> CEP-37
>>>>> <https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-37+Apache+Cassandra+Unified+Repair+Solution)>
>>>>> and the associated detailed design doc
>>>>> <https://docs.google.com/document/d/1CJWxjEi-mBABPMZ3VWJ9w5KavWfJETAGxfUpsViPcPo/edit#heading=h.r112r46toau0>
>>>>> .
>>>>>
>>>>> Thank you everyone!
>>>>>
>>>>> Chris, Andy, Josh, Dinesh, Kristijonas, and Jaydeep
>>>>>
>>>>>
>>>>>
>>>>> On Thu, Sep 19, 2024 at 11:26 AM Josh McKenzie <jm...@apache.org>
>>>>> wrote:
>>>>>
>>>>>> Not quite; finishing touches on the CEP and design doc are in flight
>>>>>> (as of last / this week).
>>>>>>
>>>>>> Soon(tm).
>>>>>>
>>>>>> On Thu, Sep 19, 2024, at 2:07 PM, Patrick McFadin wrote:
>>>>>>
>>>>>> Is this CEP ready for a VOTE thread?
>>>>>> https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-37+%28DRAFT%29+Apache+Cassandra+Unified+Repair+Solution
>>>>>>
>>>>>> On Sun, Feb 25, 2024 at 12:25 PM Jaydeep Chovatia <
>>>>>> chovatia.jaydeep@gmail.com> wrote:
>>>>>>
>>>>>> Thanks, Josh. I've just updated the CEP
>>>>>> <https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-37+%28DRAFT%29+Apache+Cassandra+Official+Repair+Solution>
>>>>>> and included all the solutions you mentioned below.
>>>>>>
>>>>>> Jaydeep
>>>>>>
>>>>>> On Thu, Feb 22, 2024 at 9:33 AM Josh McKenzie <jm...@apache.org>
>>>>>> wrote:
>>>>>>
>>>>>>
>>>>>> Very late response from me here (basically necro'ing this thread).
>>>>>>
>>>>>> I think it'd be useful to get this condensed into a CEP that we can
>>>>>> then discuss in that format. It's clearly something we all agree we need
>>>>>> and having an implementation that works, even if it's not in your preferred
>>>>>> execution domain, is vastly better than nothing IMO.
>>>>>>
>>>>>> I don't have cycles (nor background ;) ) to do that, but it sounds
>>>>>> like you do Jaydeep given the implementation you have on a private fork +
>>>>>> design.
>>>>>>
>>>>>> A non-exhaustive list of things that might be useful incorporating
>>>>>> into or referencing from a CEP:
>>>>>> Slack thread:
>>>>>> https://the-asf.slack.com/archives/CK23JSY2K/p1690225062383619
>>>>>> Joey's old C* ticket:
>>>>>> https://issues.apache.org/jira/browse/CASSANDRA-14346
>>>>>> Even older automatic repair scheduling:
>>>>>> https://issues.apache.org/jira/browse/CASSANDRA-10070
>>>>>> Your design gdoc:
>>>>>> https://docs.google.com/document/d/1CJWxjEi-mBABPMZ3VWJ9w5KavWfJETAGxfUpsViPcPo/edit#heading=h.r112r46toau0
>>>>>> PR with automated repair:
>>>>>> https://github.com/jaydeepkumar1984/cassandra/commit/ef6456d652c0d07cf29d88dfea03b73704814c2c
>>>>>>
>>>>>> My intuition is that we're all basically in agreement that this is
>>>>>> something the DB needs, we're all willing to bikeshed for our personal
>>>>>> preference on where it lives and how it's implemented, and at the end of
>>>>>> the day, code talks. I don't think anyone's said they'll die on the hill of
>>>>>> implementation details, so that feels like CEP time to me.
>>>>>>
>>>>>> If you were willing and able to get a CEP together for automated
>>>>>> repair based on the above material, given you've done the work and have the
>>>>>> proof points it's working at scale, I think this would be a *huge
>>>>>> contribution* to the community.
>>>>>>
>>>>>> On Thu, Aug 24, 2023, at 7:26 PM, Jaydeep Chovatia wrote:
>>>>>>
>>>>>> Is anyone going to file an official CEP for this?
>>>>>> As mentioned in this email thread, here is one of the solution's design
>>>>>> doc
>>>>>> <https://docs.google.com/document/d/1CJWxjEi-mBABPMZ3VWJ9w5KavWfJETAGxfUpsViPcPo/edit#heading=h.r112r46toau0>
>>>>>> and source code on a private Apache Cassandra patch. Could you go through
>>>>>> it and let me know what you think?
>>>>>>
>>>>>> Jaydeep
>>>>>>
>>>>>> On Wed, Aug 2, 2023 at 3:54 PM Jon Haddad <ru...@apache.org>
>>>>>> wrote:
>>>>>>
>>>>>> > That said I would happily support an effort to bring repair
>>>>>> scheduling to the sidecar immediately. This has nothing blocking it, and
>>>>>> would potentially enable the sidecar to provide an official repair
>>>>>> scheduling solution that is compatible with current or even previous
>>>>>> versions of the database.
>>>>>>
>>>>>> This is something I hadn't thought much about, and is a pretty good
>>>>>> argument for using the sidecar initially. There's a lot of deployments out
>>>>>> there and having an official repair option would be a big win.
>>>>>>
>>>>>>
>>>>>> On 2023/07/26 23:20:07 "C. Scott Andreas" wrote:
>>>>>> > I agree that it would be ideal for Cassandra to have a repair
>>>>>> scheduler in-DB.
>>>>>> >
>>>>>> > That said I would happily support an effort to bring repair
>>>>>> scheduling to the sidecar immediately. This has nothing blocking it, and
>>>>>> would potentially enable the sidecar to provide an official repair
>>>>>> scheduling solution that is compatible with current or even previous
>>>>>> versions of the database.
>>>>>> >
>>>>>> > Once TCM has landed, we’ll have much stronger primitives for repair
>>>>>> orchestration in the database itself. But I don’t think that should block
>>>>>> progress on a repair scheduling solution in the sidecar, and there is
>>>>>> nothing that would prevent someone from continuing to use a sidecar-based
>>>>>> solution in perpetuity if they preferred.
>>>>>> >
>>>>>> > - Scott
>>>>>> >
>>>>>> > > On Jul 26, 2023, at 3:25 PM, Jon Haddad <
>>>>>> rustyrazorblade@apache.org> wrote:
>>>>>> > >
>>>>>> > > I'm 100% in favor of repair being part of the core DB, not the
>>>>>> sidecar. The current (and past) state of things where running the DB
>>>>>> correctly *requires* running a separate process (either community
>>>>>> maintained or official C* sidecar) is incredibly painful for folks. The
>>>>>> idea that your data integrity needs to be opt-in has never made sense to me
>>>>>> from the perspective of either the product or the end user.
>>>>>> > >
>>>>>> > > I've worked with way too many teams that have either configured
>>>>>> this incorrectly or not at all.
>>>>>> > >
>>>>>> > > Ideally Cassandra would ship with repair built in and on by
>>>>>> default. Power users can disable if they want to continue to maintain
>>>>>> their own repair tooling for some reason.
>>>>>> > >
>>>>>> > > Jon
>>>>>> > >
>>>>>> > >> On 2023/07/24 20:44:14 German Eichberger via dev wrote:
>>>>>> > >> All,
>>>>>> > >> We had a brief discussion in [2] about the Uber article [1]
>>>>>> where they talk about having integrated repair into Cassandra and how great
>>>>>> that is. I expressed my disappointment that they didn't work with the
>>>>>> community on that (Uber, if you are listening time to make amends 🙂) and
>>>>>> it turns out Joey already had the idea and wrote the code [3] - so I wanted
>>>>>> to start a discussion to gauge interest and maybe how to revive that effort.
>>>>>> > >> Thanks,
>>>>>> > >> German
>>>>>> > >> [1]
>>>>>> https://www.uber.com/blog/how-uber-optimized-cassandra-operations-at-scale/
>>>>>> > >> [2]
>>>>>> https://the-asf.slack.com/archives/CK23JSY2K/p1690225062383619
>>>>>> > >> [3] https://issues.apache.org/jira/browse/CASSANDRA-14346
>>>>>> >
>>>>>>
>>>>>>
>>>>>>
>>>>>>
Re: [Discuss] Repair inside C*
Posted by Chris Lohfink <cl...@gmail.com>.
> I actually think we should be looking at how we can move things out of
the database process.
While worth pursuing, I think we would need a different CEP just to figure
out how to do that. Not only is there a lot of infrastructure difficulty in
running multi process, the inter app communication needs to be figured out
better then JMX. Even the sidecar we dont have a solid story on how to
ensure both are running or anything yet. It's up to each app owner to
figure it out. Once we have a good thing in place I think we can start
moving compactions, repairs, etc out of the database. Even then it's the
_repairs_ that is expensive, not the scheduling.
On Mon, Oct 21, 2024 at 9:45 AM Jeremiah Jordan <je...@gmail.com>
wrote:
> I love the idea of a repair service being there by default for an install
> of C*. My main concern here is that it is putting more services into the
> main database process. I actually think we should be looking at how we can
> move things out of the database process. The C* process being a giant
> monolith has always been a pain point. Is there anyway it makes sense for
> this to be an external process rather than a new thread pool inside the C*
> process?
>
> -Jeremiah Jordan
>
> On Oct 18, 2024 at 2:58:15 PM, Mick Semb Wever <mc...@apache.org> wrote:
>
>>
>> This is looking strong, thanks Jaydeep.
>>
>> I would suggest folk take a look at the design doc and the PR in the
>> CEP. A lot is there (that I have completely missed).
>>
>> I would especially ask all authors of prior art (Reaper, DSE nodesync,
>> ecchronos) to take a final review of the proposal
>>
>> Jaydeep, can we ask for a two week window while we reach out to these
>> people ? There's a lot of prior art in this space, and it feels like we're
>> in a good place now where it's clear this has legs and we can use that to
>> bring folk in and make sure there's no remaining blindspots.
>>
>>
>> On Fri, 18 Oct 2024 at 01:40, Jaydeep Chovatia <
>> chovatia.jaydeep@gmail.com> wrote:
>>
>>> Sorry, there is a typo in the CEP-37 link; here is the correct link
>>> <https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-37+Apache+Cassandra+Unified+Repair+Solution>
>>>
>>>
>>> On Thu, Oct 17, 2024 at 4:36 PM Jaydeep Chovatia <
>>> chovatia.jaydeep@gmail.com> wrote:
>>>
>>>> First, thank you for your patience while we strengthened the CEP-37.
>>>>
>>>>
>>>> Over the last eight months, Chris Lohfink, Andy Tolbert, Josh McKenzie,
>>>> Dinesh Joshi, Kristijonas Zalys, and I have done tons of work (online
>>>> discussions/a dedicated Slack channel #cassandra-repair-scheduling-cep37)
>>>> to come up with the best possible design that not only significantly
>>>> simplifies repair operations but also includes the most common features
>>>> that everyone will benefit from running at Scale.
>>>>
>>>> For example,
>>>>
>>>> -
>>>>
>>>> Apache Cassandra must be capable of running multiple repair types,
>>>> such as Full, Incremental, Paxos, and Preview - so the framework should be
>>>> easily extendable with no additional overhead from the operator’s point of
>>>> view.
>>>> -
>>>>
>>>> An easy way to extend the token-split calculation algorithm with a
>>>> default implementation should exist.
>>>> -
>>>>
>>>> Running incremental repair reliably at Scale is pretty challenging,
>>>> so we need to place safeguards, such as migration/rollback w/o restart and
>>>> stopping incremental repair automatically if the disk is about to get full.
>>>>
>>>> We are glad to inform you that CEP-37 (i.e., Repair inside Cassandra)
>>>> is now officially ready for review after multiple rounds of design,
>>>> testing, code reviews, documentation reviews, and, more importantly,
>>>> validation that it runs at Scale!
>>>>
>>>>
>>>> Some facts about CEP-37.
>>>>
>>>> -
>>>>
>>>> Multiple members have verified all aspects of CEP-37 numerous times.
>>>> -
>>>>
>>>> The design proposed in CEP-37 has been thoroughly tried and tested
>>>> on an immense scale (hundreds of unique Cassandra clusters, tens of
>>>> thousands of Cassandra nodes, with tens of millions of QPS) on top of 4.1
>>>> open-source for more than five years; please see more details here
>>>> <https://www.uber.com/en-US/blog/how-uber-optimized-cassandra-operations-at-scale/>
>>>> .
>>>> -
>>>>
>>>> The following presentation
>>>> <https://docs.google.com/presentation/d/1Zilww9c7LihHULk_ckErI2s4XbObxjWknKqRtbvHyZc/edit#slide=id.g30a4fd4fcf7_0_13>
>>>> highlights the rigorous applied to CEP-37, which was given during last
>>>> week’s Apache Cassandra Bay Area Meetup
>>>> <https://www.meetup.com/apache-cassandra-bay-area/events/303469006/>
>>>> ,
>>>>
>>>>
>>>> Since things are massively overhauled, we believe it is almost ready
>>>> for a final pass pre-VOTE. We would like you to please review the
>>>> CEP-37
>>>> <https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-37+Apache+Cassandra+Unified+Repair+Solution)>
>>>> and the associated detailed design doc
>>>> <https://docs.google.com/document/d/1CJWxjEi-mBABPMZ3VWJ9w5KavWfJETAGxfUpsViPcPo/edit#heading=h.r112r46toau0>
>>>> .
>>>>
>>>> Thank you everyone!
>>>>
>>>> Chris, Andy, Josh, Dinesh, Kristijonas, and Jaydeep
>>>>
>>>>
>>>>
>>>> On Thu, Sep 19, 2024 at 11:26 AM Josh McKenzie <jm...@apache.org>
>>>> wrote:
>>>>
>>>>> Not quite; finishing touches on the CEP and design doc are in flight
>>>>> (as of last / this week).
>>>>>
>>>>> Soon(tm).
>>>>>
>>>>> On Thu, Sep 19, 2024, at 2:07 PM, Patrick McFadin wrote:
>>>>>
>>>>> Is this CEP ready for a VOTE thread?
>>>>> https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-37+%28DRAFT%29+Apache+Cassandra+Unified+Repair+Solution
>>>>>
>>>>> On Sun, Feb 25, 2024 at 12:25 PM Jaydeep Chovatia <
>>>>> chovatia.jaydeep@gmail.com> wrote:
>>>>>
>>>>> Thanks, Josh. I've just updated the CEP
>>>>> <https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-37+%28DRAFT%29+Apache+Cassandra+Official+Repair+Solution>
>>>>> and included all the solutions you mentioned below.
>>>>>
>>>>> Jaydeep
>>>>>
>>>>> On Thu, Feb 22, 2024 at 9:33 AM Josh McKenzie <jm...@apache.org>
>>>>> wrote:
>>>>>
>>>>>
>>>>> Very late response from me here (basically necro'ing this thread).
>>>>>
>>>>> I think it'd be useful to get this condensed into a CEP that we can
>>>>> then discuss in that format. It's clearly something we all agree we need
>>>>> and having an implementation that works, even if it's not in your preferred
>>>>> execution domain, is vastly better than nothing IMO.
>>>>>
>>>>> I don't have cycles (nor background ;) ) to do that, but it sounds
>>>>> like you do Jaydeep given the implementation you have on a private fork +
>>>>> design.
>>>>>
>>>>> A non-exhaustive list of things that might be useful incorporating
>>>>> into or referencing from a CEP:
>>>>> Slack thread:
>>>>> https://the-asf.slack.com/archives/CK23JSY2K/p1690225062383619
>>>>> Joey's old C* ticket:
>>>>> https://issues.apache.org/jira/browse/CASSANDRA-14346
>>>>> Even older automatic repair scheduling:
>>>>> https://issues.apache.org/jira/browse/CASSANDRA-10070
>>>>> Your design gdoc:
>>>>> https://docs.google.com/document/d/1CJWxjEi-mBABPMZ3VWJ9w5KavWfJETAGxfUpsViPcPo/edit#heading=h.r112r46toau0
>>>>> PR with automated repair:
>>>>> https://github.com/jaydeepkumar1984/cassandra/commit/ef6456d652c0d07cf29d88dfea03b73704814c2c
>>>>>
>>>>> My intuition is that we're all basically in agreement that this is
>>>>> something the DB needs, we're all willing to bikeshed for our personal
>>>>> preference on where it lives and how it's implemented, and at the end of
>>>>> the day, code talks. I don't think anyone's said they'll die on the hill of
>>>>> implementation details, so that feels like CEP time to me.
>>>>>
>>>>> If you were willing and able to get a CEP together for automated
>>>>> repair based on the above material, given you've done the work and have the
>>>>> proof points it's working at scale, I think this would be a *huge
>>>>> contribution* to the community.
>>>>>
>>>>> On Thu, Aug 24, 2023, at 7:26 PM, Jaydeep Chovatia wrote:
>>>>>
>>>>> Is anyone going to file an official CEP for this?
>>>>> As mentioned in this email thread, here is one of the solution's design
>>>>> doc
>>>>> <https://docs.google.com/document/d/1CJWxjEi-mBABPMZ3VWJ9w5KavWfJETAGxfUpsViPcPo/edit#heading=h.r112r46toau0>
>>>>> and source code on a private Apache Cassandra patch. Could you go through
>>>>> it and let me know what you think?
>>>>>
>>>>> Jaydeep
>>>>>
>>>>> On Wed, Aug 2, 2023 at 3:54 PM Jon Haddad <ru...@apache.org>
>>>>> wrote:
>>>>>
>>>>> > That said I would happily support an effort to bring repair
>>>>> scheduling to the sidecar immediately. This has nothing blocking it, and
>>>>> would potentially enable the sidecar to provide an official repair
>>>>> scheduling solution that is compatible with current or even previous
>>>>> versions of the database.
>>>>>
>>>>> This is something I hadn't thought much about, and is a pretty good
>>>>> argument for using the sidecar initially. There's a lot of deployments out
>>>>> there and having an official repair option would be a big win.
>>>>>
>>>>>
>>>>> On 2023/07/26 23:20:07 "C. Scott Andreas" wrote:
>>>>> > I agree that it would be ideal for Cassandra to have a repair
>>>>> scheduler in-DB.
>>>>> >
>>>>> > That said I would happily support an effort to bring repair
>>>>> scheduling to the sidecar immediately. This has nothing blocking it, and
>>>>> would potentially enable the sidecar to provide an official repair
>>>>> scheduling solution that is compatible with current or even previous
>>>>> versions of the database.
>>>>> >
>>>>> > Once TCM has landed, we’ll have much stronger primitives for repair
>>>>> orchestration in the database itself. But I don’t think that should block
>>>>> progress on a repair scheduling solution in the sidecar, and there is
>>>>> nothing that would prevent someone from continuing to use a sidecar-based
>>>>> solution in perpetuity if they preferred.
>>>>> >
>>>>> > - Scott
>>>>> >
>>>>> > > On Jul 26, 2023, at 3:25 PM, Jon Haddad <
>>>>> rustyrazorblade@apache.org> wrote:
>>>>> > >
>>>>> > > I'm 100% in favor of repair being part of the core DB, not the
>>>>> sidecar. The current (and past) state of things where running the DB
>>>>> correctly *requires* running a separate process (either community
>>>>> maintained or official C* sidecar) is incredibly painful for folks. The
>>>>> idea that your data integrity needs to be opt-in has never made sense to me
>>>>> from the perspective of either the product or the end user.
>>>>> > >
>>>>> > > I've worked with way too many teams that have either configured
>>>>> this incorrectly or not at all.
>>>>> > >
>>>>> > > Ideally Cassandra would ship with repair built in and on by
>>>>> default. Power users can disable if they want to continue to maintain
>>>>> their own repair tooling for some reason.
>>>>> > >
>>>>> > > Jon
>>>>> > >
>>>>> > >> On 2023/07/24 20:44:14 German Eichberger via dev wrote:
>>>>> > >> All,
>>>>> > >> We had a brief discussion in [2] about the Uber article [1] where
>>>>> they talk about having integrated repair into Cassandra and how great that
>>>>> is. I expressed my disappointment that they didn't work with the community
>>>>> on that (Uber, if you are listening time to make amends 🙂) and it turns
>>>>> out Joey already had the idea and wrote the code [3] - so I wanted to start
>>>>> a discussion to gauge interest and maybe how to revive that effort.
>>>>> > >> Thanks,
>>>>> > >> German
>>>>> > >> [1]
>>>>> https://www.uber.com/blog/how-uber-optimized-cassandra-operations-at-scale/
>>>>> > >> [2]
>>>>> https://the-asf.slack.com/archives/CK23JSY2K/p1690225062383619
>>>>> > >> [3] https://issues.apache.org/jira/browse/CASSANDRA-14346
>>>>> >
>>>>>
>>>>>
>>>>>
>>>>>
Re: [Discuss] Repair inside C*
Posted by Dinesh Joshi <dj...@apache.org>.
On Mon, Oct 21, 2024 at 9:18 AM David Capwell <dc...@apple.com> wrote:
> One thing to keep in mind is that larger clusters require you “smartly”
> split the ranges else you nuke your cluster… knowing how to split requires
> internal knowledge from the database which we could expose, but then we
> need to expose a new public API (most likely a set of APIs) just to do
> this. When you do the scheduling internal to the database you can make
> “breaking” changes that improve stability into a patch fix rather than have
> to wait for the next major…
>
As the project and its ecosystem grows we need to have a conversation on
what is a public API? I do not want to derail this thread but very briefly,
we should make a distinction between `project internal` private API that is
exposed to Cassandra's components (which very well could run as a separate
local or remote process) and public API that the rest of the world outside
of the project uses. The backward compatibility expectations will be
different for `project internal` private API and public API.
Re: [Discuss] Repair inside C*
Posted by David Capwell <dc...@apple.com>.
> Is there anyway it makes sense for this to be an external process rather than a new thread pool inside the C* process?
One thing to keep in mind is that larger clusters require you “smartly” split the ranges else you nuke your cluster… knowing how to split requires internal knowledge from the database which we could expose, but then we need to expose a new public API (most likely a set of APIs) just to do this. When you do the scheduling internal to the database you can make “breaking” changes that improve stability into a patch fix rather than have to wait for the next major…
To me this problem is the main reason I am in favor of repair scheduling being inside the database…
> On Oct 21, 2024, at 8:55 AM, Josh McKenzie <jm...@apache.org> wrote:
>
>> Is there anyway it makes sense for this to be an external process rather than a new thread pool inside the C* process?
> I'm personally more irked by the merkle tree building / streaming / merging / etc resource utilization being in the primary C* process. My intuition is that the scheduling of things is so lightweight as to be a non-issue when it comes to impact on reads and writes.
>
> That said, if you're more alluding to a meta conversation about the architecture of the DB and whether having a monolithic :allthethings: process is preferable to breaking things apart, well, that's an entirely different conversation on which I have... different thoughts. :D
>
> On Mon, Oct 21, 2024, at 10:44 AM, Jeremiah Jordan wrote:
>> I love the idea of a repair service being there by default for an install of C*. My main concern here is that it is putting more services into the main database process. I actually think we should be looking at how we can move things out of the database process. The C* process being a giant monolith has always been a pain point. Is there anyway it makes sense for this to be an external process rather than a new thread pool inside the C* process?
>>
>> -Jeremiah Jordan
>>
>> On Oct 18, 2024 at 2:58:15 PM, Mick Semb Wever <mck@apache.org <ma...@apache.org>> wrote:
>>>
>>> This is looking strong, thanks Jaydeep.
>>>
>>> I would suggest folk take a look at the design doc and the PR in the CEP. A lot is there (that I have completely missed).
>>>
>>> I would especially ask all authors of prior art (Reaper, DSE nodesync, ecchronos) to take a final review of the proposal
>>>
>>> Jaydeep, can we ask for a two week window while we reach out to these people ? There's a lot of prior art in this space, and it feels like we're in a good place now where it's clear this has legs and we can use that to bring folk in and make sure there's no remaining blindspots.
>>>
>>>
>>> On Fri, 18 Oct 2024 at 01:40, Jaydeep Chovatia <chovatia.jaydeep@gmail.com <ma...@gmail.com>> wrote:
>>> Sorry, there is a typo in the CEP-37 link; here is the correct link <https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-37+Apache+Cassandra+Unified+Repair+Solution>
>>>
>>>
>>> On Thu, Oct 17, 2024 at 4:36 PM Jaydeep Chovatia <chovatia.jaydeep@gmail.com <ma...@gmail.com>> wrote:
>>> First, thank you for your patience while we strengthened the CEP-37.
>>>
>>> Over the last eight months, Chris Lohfink, Andy Tolbert, Josh McKenzie, Dinesh Joshi, Kristijonas Zalys, and I have done tons of work (online discussions/a dedicated Slack channel #cassandra-repair-scheduling-cep37) to come up with the best possible design that not only significantly simplifies repair operations but also includes the most common features that everyone will benefit from running at Scale.
>>> For example,
>>> Apache Cassandra must be capable of running multiple repair types, such as Full, Incremental, Paxos, and Preview - so the framework should be easily extendable with no additional overhead from the operator’s point of view.
>>> An easy way to extend the token-split calculation algorithm with a default implementation should exist.
>>> Running incremental repair reliably at Scale is pretty challenging, so we need to place safeguards, such as migration/rollback w/o restart and stopping incremental repair automatically if the disk is about to get full.
>>>
>>> We are glad to inform you that CEP-37 (i.e., Repair inside Cassandra) is now officially ready for review after multiple rounds of design, testing, code reviews, documentation reviews, and, more importantly, validation that it runs at Scale!
>>>
>>> Some facts about CEP-37.
>>> Multiple members have verified all aspects of CEP-37 numerous times.
>>> The design proposed in CEP-37 has been thoroughly tried and tested on an immense scale (hundreds of unique Cassandra clusters, tens of thousands of Cassandra nodes, with tens of millions of QPS) on top of 4.1 open-source for more than five years; please see more detailshere <https://www.uber.com/en-US/blog/how-uber-optimized-cassandra-operations-at-scale/>.
>>> The following presentation <https://docs.google.com/presentation/d/1Zilww9c7LihHULk_ckErI2s4XbObxjWknKqRtbvHyZc/edit#slide=id.g30a4fd4fcf7_0_13> highlights the rigorous applied to CEP-37, which was given during last week’s Apache Cassandra Bay Area Meetup <https://www.meetup.com/apache-cassandra-bay-area/events/303469006/>,
>>>
>>> Since things are massively overhauled, we believe it is almost ready for a final pass pre-VOTE. We would like you to please review the CEP-37 <https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-37+Apache+Cassandra+Unified+Repair+Solution)> and the associated detailed design doc <https://docs.google.com/document/d/1CJWxjEi-mBABPMZ3VWJ9w5KavWfJETAGxfUpsViPcPo/edit#heading=h.r112r46toau0>.
>>>
>>> Thank you everyone!
>>> Chris, Andy, Josh, Dinesh, Kristijonas, and Jaydeep
>>>
>>>
>>>
>>> On Thu, Sep 19, 2024 at 11:26 AM Josh McKenzie <jmckenzie@apache.org <ma...@apache.org>> wrote:
>>>
>>> Not quite; finishing touches on the CEP and design doc are in flight (as of last / this week).
>>>
>>> Soon(tm).
>>>
>>> On Thu, Sep 19, 2024, at 2:07 PM, Patrick McFadin wrote:
>>>> Is this CEP ready for a VOTE thread? https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-37+%28DRAFT%29+Apache+Cassandra+Unified+Repair+Solution
>>>>
>>>> On Sun, Feb 25, 2024 at 12:25 PM Jaydeep Chovatia <chovatia.jaydeep@gmail.com <ma...@gmail.com>> wrote:
>>>> Thanks, Josh. I've just updated the CEP <https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-37+%28DRAFT%29+Apache+Cassandra+Official+Repair+Solution> and included all the solutions you mentioned below.
>>>>
>>>> Jaydeep
>>>>
>>>> On Thu, Feb 22, 2024 at 9:33 AM Josh McKenzie <jmckenzie@apache.org <ma...@apache.org>> wrote:
>>>>
>>>> Very late response from me here (basically necro'ing this thread).
>>>>
>>>> I think it'd be useful to get this condensed into a CEP that we can then discuss in that format. It's clearly something we all agree we need and having an implementation that works, even if it's not in your preferred execution domain, is vastly better than nothing IMO.
>>>>
>>>> I don't have cycles (nor background ;) ) to do that, but it sounds like you do Jaydeep given the implementation you have on a private fork + design.
>>>>
>>>> A non-exhaustive list of things that might be useful incorporating into or referencing from a CEP:
>>>> Slack thread: https://the-asf.slack.com/archives/CK23JSY2K/p1690225062383619
>>>> Joey's old C* ticket: https://issues.apache.org/jira/browse/CASSANDRA-14346
>>>> Even older automatic repair scheduling: https://issues.apache.org/jira/browse/CASSANDRA-10070
>>>> Your design gdoc: https://docs.google.com/document/d/1CJWxjEi-mBABPMZ3VWJ9w5KavWfJETAGxfUpsViPcPo/edit#heading=h.r112r46toau0
>>>> PR with automated repair: https://github.com/jaydeepkumar1984/cassandra/commit/ef6456d652c0d07cf29d88dfea03b73704814c2c
>>>>
>>>> My intuition is that we're all basically in agreement that this is something the DB needs, we're all willing to bikeshed for our personal preference on where it lives and how it's implemented, and at the end of the day, code talks. I don't think anyone's said they'll die on the hill of implementation details, so that feels like CEP time to me.
>>>>
>>>> If you were willing and able to get a CEP together for automated repair based on the above material, given you've done the work and have the proof points it's working at scale, I think this would be a huge contribution to the community.
>>>>
>>>> On Thu, Aug 24, 2023, at 7:26 PM, Jaydeep Chovatia wrote:
>>>>> Is anyone going to file an official CEP for this?
>>>>> As mentioned in this email thread, here is one of the solution's design doc <https://docs.google.com/document/d/1CJWxjEi-mBABPMZ3VWJ9w5KavWfJETAGxfUpsViPcPo/edit#heading=h.r112r46toau0> and source code on a private Apache Cassandra patch. Could you go through it and let me know what you think?
>>>>>
>>>>> Jaydeep
>>>>>
>>>>> On Wed, Aug 2, 2023 at 3:54 PM Jon Haddad <rustyrazorblade@apache.org <ma...@apache.org>> wrote:
>>>>> > That said I would happily support an effort to bring repair scheduling to the sidecar immediately. This has nothing blocking it, and would potentially enable the sidecar to provide an official repair scheduling solution that is compatible with current or even previous versions of the database.
>>>>>
>>>>> This is something I hadn't thought much about, and is a pretty good argument for using the sidecar initially. There's a lot of deployments out there and having an official repair option would be a big win.
>>>>>
>>>>>
>>>>> On 2023/07/26 23:20:07 "C. Scott Andreas" wrote:
>>>>> > I agree that it would be ideal for Cassandra to have a repair scheduler in-DB.
>>>>> >
>>>>> > That said I would happily support an effort to bring repair scheduling to the sidecar immediately. This has nothing blocking it, and would potentially enable the sidecar to provide an official repair scheduling solution that is compatible with current or even previous versions of the database.
>>>>> >
>>>>> > Once TCM has landed, we’ll have much stronger primitives for repair orchestration in the database itself. But I don’t think that should block progress on a repair scheduling solution in the sidecar, and there is nothing that would prevent someone from continuing to use a sidecar-based solution in perpetuity if they preferred.
>>>>> >
>>>>> > - Scott
>>>>> >
>>>>> > > On Jul 26, 2023, at 3:25 PM, Jon Haddad <rustyrazorblade@apache.org <ma...@apache.org>> wrote:
>>>>> > >
>>>>> > > I'm 100% in favor of repair being part of the core DB, not the sidecar. The current (and past) state of things where running the DB correctly *requires* running a separate process (either community maintained or official C* sidecar) is incredibly painful for folks. The idea that your data integrity needs to be opt-in has never made sense to me from the perspective of either the product or the end user.
>>>>> > >
>>>>> > > I've worked with way too many teams that have either configured this incorrectly or not at all.
>>>>> > >
>>>>> > > Ideally Cassandra would ship with repair built in and on by default. Power users can disable if they want to continue to maintain their own repair tooling for some reason.
>>>>> > >
>>>>> > > Jon
>>>>> > >
>>>>> > >> On 2023/07/24 20:44:14 German Eichberger via dev wrote:
>>>>> > >> All,
>>>>> > >> We had a brief discussion in [2] about the Uber article [1] where they talk about having integrated repair into Cassandra and how great that is. I expressed my disappointment that they didn't work with the community on that (Uber, if you are listening time to make amends 🙂) and it turns out Joey already had the idea and wrote the code [3] - so I wanted to start a discussion to gauge interest and maybe how to revive that effort.
>>>>> > >> Thanks,
>>>>> > >> German
>>>>> > >> [1] https://www.uber.com/blog/how-uber-optimized-cassandra-operations-at-scale/
>>>>> > >> [2] https://the-asf.slack.com/archives/CK23JSY2K/p1690225062383619
>>>>> > >> [3] https://issues.apache.org/jira/browse/CASSANDRA-14346
>>>>> >
Re: [Discuss] Repair inside C*
Posted by Josh McKenzie <jm...@apache.org>.
> Is there anyway it makes sense for this to be an external process rather than a new thread pool inside the C* process?
I'm personally more irked by the merkle tree building / streaming / merging / etc resource utilization being in the primary C* process. My intuition is that the *scheduling* of things is so lightweight as to be a non-issue when it comes to impact on reads and writes.
That said, if you're more alluding to a meta conversation about the *architecture* of the DB and whether having a monolithic :allthethings: process is preferable to breaking things apart, well, that's an entirely different conversation on which I have... different thoughts. :D
On Mon, Oct 21, 2024, at 10:44 AM, Jeremiah Jordan wrote:
> I love the idea of a repair service being there by default for an install of C*. My main concern here is that it is putting more services into the main database process. I actually think we should be looking at how we can move things out of the database process. The C* process being a giant monolith has always been a pain point. Is there anyway it makes sense for this to be an external process rather than a new thread pool inside the C* process?
>
> -Jeremiah Jordan
>
> On Oct 18, 2024 at 2:58:15 PM, Mick Semb Wever <mc...@apache.org> wrote:
>>
>> This is looking strong, thanks Jaydeep.
>>
>> I would suggest folk take a look at the design doc and the PR in the CEP. A lot is there (that I have completely missed).
>>
>> I would especially ask all authors of prior art (Reaper, DSE nodesync, ecchronos) to take a final review of the proposal
>>
>> Jaydeep, can we ask for a two week window while we reach out to these people ? There's a lot of prior art in this space, and it feels like we're in a good place now where it's clear this has legs and we can use that to bring folk in and make sure there's no remaining blindspots.
>>
>>
>> On Fri, 18 Oct 2024 at 01:40, Jaydeep Chovatia <ch...@gmail.com> wrote:
>>> Sorry, there is a typo in the CEP-37 link; here is the correct link <https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-37+Apache+Cassandra+Unified+Repair+Solution>
>>>
>>>
>>> On Thu, Oct 17, 2024 at 4:36 PM Jaydeep Chovatia <ch...@gmail.com> wrote:
>>>> First, thank you for your patience while we strengthened the CEP-37.
>>>>
>>>>
>>>>
>>>> Over the last eight months, Chris Lohfink, Andy Tolbert, Josh McKenzie, Dinesh Joshi, Kristijonas Zalys, and I have done tons of work (online discussions/a dedicated Slack channel #cassandra-repair-scheduling-cep37) to come up with the best possible design that not only significantly simplifies repair operations but also includes the most common features that everyone will benefit from running at Scale.
>>>>
>>>> For example,
>>>>
>>>> • Apache Cassandra must be capable of running multiple repair types, such as Full, Incremental, Paxos, and Preview - so the framework should be easily extendable with no additional overhead from the operator’s point of view.
>>>>
>>>> • An easy way to extend the token-split calculation algorithm with a default implementation should exist.
>>>>
>>>> • Running incremental repair reliably at Scale is pretty challenging, so we need to place safeguards, such as migration/rollback w/o restart and stopping incremental repair automatically if the disk is about to get full.
>>>>
>>>> We are glad to inform you that CEP-37 (i.e., Repair inside Cassandra) is now officially ready for review after multiple rounds of design, testing, code reviews, documentation reviews, and, more importantly, validation that it runs at Scale!
>>>>
>>>>
>>>>
>>>> Some facts about CEP-37.
>>>>
>>>> • Multiple members have verified all aspects of CEP-37 numerous times.
>>>>
>>>> • The design proposed in CEP-37 has been thoroughly tried and tested on an immense scale (hundreds of unique Cassandra clusters, tens of thousands of Cassandra nodes, with tens of millions of QPS) on top of 4.1 open-source for more than five years; please see more details _here_ <https://www.uber.com/en-US/blog/how-uber-optimized-cassandra-operations-at-scale/>.
>>>>
>>>> • The following _presentation_ <https://docs.google.com/presentation/d/1Zilww9c7LihHULk_ckErI2s4XbObxjWknKqRtbvHyZc/edit#slide=id.g30a4fd4fcf7_0_13> highlights the rigorous applied to CEP-37, which was given during last week’s Apache Cassandra Bay Area _Meetup_ <https://www.meetup.com/apache-cassandra-bay-area/events/303469006/>,
>>>>
>>>>
>>>> Since things are massively overhauled, we believe it is almost ready for a final pass pre-VOTE. We would like you to please review the _CEP-37_ <https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-37+Apache+Cassandra+Unified+Repair+Solution)> and the associated detailed design _doc_ <https://docs.google.com/document/d/1CJWxjEi-mBABPMZ3VWJ9w5KavWfJETAGxfUpsViPcPo/edit#heading=h.r112r46toau0>.
>>>>
>>>>
>>>> Thank you everyone!
>>>>
>>>> Chris, Andy, Josh, Dinesh, Kristijonas, and Jaydeep
>>>>
>>>>
>>>>
>>>>
>>>> On Thu, Sep 19, 2024 at 11:26 AM Josh McKenzie <jm...@apache.org> wrote:
>>>>> __
>>>>> Not quite; finishing touches on the CEP and design doc are in flight (as of last / this week).
>>>>>
>>>>> Soon(tm).
>>>>>
>>>>> On Thu, Sep 19, 2024, at 2:07 PM, Patrick McFadin wrote:
>>>>>> Is this CEP ready for a VOTE thread? https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-37+%28DRAFT%29+Apache+Cassandra+Unified+Repair+Solution
>>>>>>
>>>>>> On Sun, Feb 25, 2024 at 12:25 PM Jaydeep Chovatia <ch...@gmail.com> wrote:
>>>>>>> Thanks, Josh. I've just updated the CEP <https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-37+%28DRAFT%29+Apache+Cassandra+Official+Repair+Solution> and included all the solutions you mentioned below.
>>>>>>>
>>>>>>> Jaydeep
>>>>>>>
>>>>>>> On Thu, Feb 22, 2024 at 9:33 AM Josh McKenzie <jm...@apache.org> wrote:
>>>>>>>> __
>>>>>>>> Very late response from me here (basically necro'ing this thread).
>>>>>>>>
>>>>>>>> I think it'd be useful to get this condensed into a CEP that we can then discuss in that format. It's clearly something we all agree we need and having an implementation that works, even if it's not in your preferred execution domain, is vastly better than nothing IMO.
>>>>>>>>
>>>>>>>> I don't have cycles (nor background ;) ) to do that, but it sounds like you do Jaydeep given the implementation you have on a private fork + design.
>>>>>>>>
>>>>>>>> A non-exhaustive list of things that might be useful incorporating into or referencing from a CEP:
>>>>>>>> Slack thread: https://the-asf.slack.com/archives/CK23JSY2K/p1690225062383619
>>>>>>>> Joey's old C* ticket: https://issues.apache.org/jira/browse/CASSANDRA-14346
>>>>>>>> Even older automatic repair scheduling: https://issues.apache.org/jira/browse/CASSANDRA-10070
>>>>>>>> Your design gdoc: https://docs.google.com/document/d/1CJWxjEi-mBABPMZ3VWJ9w5KavWfJETAGxfUpsViPcPo/edit#heading=h.r112r46toau0
>>>>>>>> PR with automated repair: https://github.com/jaydeepkumar1984/cassandra/commit/ef6456d652c0d07cf29d88dfea03b73704814c2c
>>>>>>>>
>>>>>>>> My intuition is that we're all basically in agreement that this is something the DB needs, we're all willing to bikeshed for our personal preference on where it lives and how it's implemented, and at the end of the day, code talks. I don't think anyone's said they'll die on the hill of implementation details, so that feels like CEP time to me.
>>>>>>>>
>>>>>>>> If you were willing and able to get a CEP together for automated repair based on the above material, given you've done the work and have the proof points it's working at scale, I think this would be a *huge contribution* to the community.
>>>>>>>>
>>>>>>>> On Thu, Aug 24, 2023, at 7:26 PM, Jaydeep Chovatia wrote:
>>>>>>>>> Is anyone going to file an official CEP for this?
>>>>>>>>> As mentioned in this email thread, here is one of the solution's design doc <https://docs.google.com/document/d/1CJWxjEi-mBABPMZ3VWJ9w5KavWfJETAGxfUpsViPcPo/edit#heading=h.r112r46toau0> and source code on a private Apache Cassandra patch. Could you go through it and let me know what you think?
>>>>>>>>>
>>>>>>>>> Jaydeep
>>>>>>>>>
>>>>>>>>> On Wed, Aug 2, 2023 at 3:54 PM Jon Haddad <ru...@apache.org> wrote:
>>>>>>>>>> > That said I would happily support an effort to bring repair scheduling to the sidecar immediately. This has nothing blocking it, and would potentially enable the sidecar to provide an official repair scheduling solution that is compatible with current or even previous versions of the database.
>>>>>>>>>>
>>>>>>>>>> This is something I hadn't thought much about, and is a pretty good argument for using the sidecar initially. There's a lot of deployments out there and having an official repair option would be a big win.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On 2023/07/26 23:20:07 "C. Scott Andreas" wrote:
>>>>>>>>>> > I agree that it would be ideal for Cassandra to have a repair scheduler in-DB.
>>>>>>>>>> >
>>>>>>>>>> > That said I would happily support an effort to bring repair scheduling to the sidecar immediately. This has nothing blocking it, and would potentially enable the sidecar to provide an official repair scheduling solution that is compatible with current or even previous versions of the database.
>>>>>>>>>> >
>>>>>>>>>> > Once TCM has landed, we’ll have much stronger primitives for repair orchestration in the database itself. But I don’t think that should block progress on a repair scheduling solution in the sidecar, and there is nothing that would prevent someone from continuing to use a sidecar-based solution in perpetuity if they preferred.
>>>>>>>>>> >
>>>>>>>>>> > - Scott
>>>>>>>>>> >
>>>>>>>>>> > > On Jul 26, 2023, at 3:25 PM, Jon Haddad <ru...@apache.org> wrote:
>>>>>>>>>> > >
>>>>>>>>>> > > I'm 100% in favor of repair being part of the core DB, not the sidecar. The current (and past) state of things where running the DB correctly *requires* running a separate process (either community maintained or official C* sidecar) is incredibly painful for folks. The idea that your data integrity needs to be opt-in has never made sense to me from the perspective of either the product or the end user.
>>>>>>>>>> > >
>>>>>>>>>> > > I've worked with way too many teams that have either configured this incorrectly or not at all.
>>>>>>>>>> > >
>>>>>>>>>> > > Ideally Cassandra would ship with repair built in and on by default. Power users can disable if they want to continue to maintain their own repair tooling for some reason.
>>>>>>>>>> > >
>>>>>>>>>> > > Jon
>>>>>>>>>> > >
>>>>>>>>>> > >> On 2023/07/24 20:44:14 German Eichberger via dev wrote:
>>>>>>>>>> > >> All,
>>>>>>>>>> > >> We had a brief discussion in [2] about the Uber article [1] where they talk about having integrated repair into Cassandra and how great that is. I expressed my disappointment that they didn't work with the community on that (Uber, if you are listening time to make amends 🙂) and it turns out Joey already had the idea and wrote the code [3] - so I wanted to start a discussion to gauge interest and maybe how to revive that effort.
>>>>>>>>>> > >> Thanks,
>>>>>>>>>> > >> German
>>>>>>>>>> > >> [1] https://www.uber.com/blog/how-uber-optimized-cassandra-operations-at-scale/
>>>>>>>>>> > >> [2] https://the-asf.slack.com/archives/CK23JSY2K/p1690225062383619
>>>>>>>>>> > >> [3] https://issues.apache.org/jira/browse/CASSANDRA-14346
>>>>>>>>>> >
>>>>>>>>
>>>>>
Re: [Discuss] Repair inside C*
Posted by Jeremiah Jordan <je...@gmail.com>.
I love the idea of a repair service being there by default for an install
of C*. My main concern here is that it is putting more services into the
main database process. I actually think we should be looking at how we can
move things out of the database process. The C* process being a giant
monolith has always been a pain point. Is there anyway it makes sense for
this to be an external process rather than a new thread pool inside the C*
process?
-Jeremiah Jordan
On Oct 18, 2024 at 2:58:15 PM, Mick Semb Wever <mc...@apache.org> wrote:
>
> This is looking strong, thanks Jaydeep.
>
> I would suggest folk take a look at the design doc and the PR in the CEP.
> A lot is there (that I have completely missed).
>
> I would especially ask all authors of prior art (Reaper, DSE nodesync,
> ecchronos) to take a final review of the proposal
>
> Jaydeep, can we ask for a two week window while we reach out to these
> people ? There's a lot of prior art in this space, and it feels like we're
> in a good place now where it's clear this has legs and we can use that to
> bring folk in and make sure there's no remaining blindspots.
>
>
> On Fri, 18 Oct 2024 at 01:40, Jaydeep Chovatia <ch...@gmail.com>
> wrote:
>
>> Sorry, there is a typo in the CEP-37 link; here is the correct link
>> <https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-37+Apache+Cassandra+Unified+Repair+Solution>
>>
>>
>> On Thu, Oct 17, 2024 at 4:36 PM Jaydeep Chovatia <
>> chovatia.jaydeep@gmail.com> wrote:
>>
>>> First, thank you for your patience while we strengthened the CEP-37.
>>>
>>>
>>> Over the last eight months, Chris Lohfink, Andy Tolbert, Josh McKenzie,
>>> Dinesh Joshi, Kristijonas Zalys, and I have done tons of work (online
>>> discussions/a dedicated Slack channel #cassandra-repair-scheduling-cep37)
>>> to come up with the best possible design that not only significantly
>>> simplifies repair operations but also includes the most common features
>>> that everyone will benefit from running at Scale.
>>>
>>> For example,
>>>
>>> -
>>>
>>> Apache Cassandra must be capable of running multiple repair types,
>>> such as Full, Incremental, Paxos, and Preview - so the framework should be
>>> easily extendable with no additional overhead from the operator’s point of
>>> view.
>>> -
>>>
>>> An easy way to extend the token-split calculation algorithm with a
>>> default implementation should exist.
>>> -
>>>
>>> Running incremental repair reliably at Scale is pretty challenging,
>>> so we need to place safeguards, such as migration/rollback w/o restart and
>>> stopping incremental repair automatically if the disk is about to get full.
>>>
>>> We are glad to inform you that CEP-37 (i.e., Repair inside Cassandra) is
>>> now officially ready for review after multiple rounds of design, testing,
>>> code reviews, documentation reviews, and, more importantly, validation that
>>> it runs at Scale!
>>>
>>>
>>> Some facts about CEP-37.
>>>
>>> -
>>>
>>> Multiple members have verified all aspects of CEP-37 numerous times.
>>> -
>>>
>>> The design proposed in CEP-37 has been thoroughly tried and tested
>>> on an immense scale (hundreds of unique Cassandra clusters, tens of
>>> thousands of Cassandra nodes, with tens of millions of QPS) on top of 4.1
>>> open-source for more than five years; please see more details here
>>> <https://www.uber.com/en-US/blog/how-uber-optimized-cassandra-operations-at-scale/>
>>> .
>>> -
>>>
>>> The following presentation
>>> <https://docs.google.com/presentation/d/1Zilww9c7LihHULk_ckErI2s4XbObxjWknKqRtbvHyZc/edit#slide=id.g30a4fd4fcf7_0_13>
>>> highlights the rigorous applied to CEP-37, which was given during last
>>> week’s Apache Cassandra Bay Area Meetup
>>> <https://www.meetup.com/apache-cassandra-bay-area/events/303469006/>,
>>>
>>>
>>> Since things are massively overhauled, we believe it is almost ready for
>>> a final pass pre-VOTE. We would like you to please review the CEP-37
>>> <https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-37+Apache+Cassandra+Unified+Repair+Solution)>
>>> and the associated detailed design doc
>>> <https://docs.google.com/document/d/1CJWxjEi-mBABPMZ3VWJ9w5KavWfJETAGxfUpsViPcPo/edit#heading=h.r112r46toau0>
>>> .
>>>
>>> Thank you everyone!
>>>
>>> Chris, Andy, Josh, Dinesh, Kristijonas, and Jaydeep
>>>
>>>
>>>
>>> On Thu, Sep 19, 2024 at 11:26 AM Josh McKenzie <jm...@apache.org>
>>> wrote:
>>>
>>>> Not quite; finishing touches on the CEP and design doc are in flight
>>>> (as of last / this week).
>>>>
>>>> Soon(tm).
>>>>
>>>> On Thu, Sep 19, 2024, at 2:07 PM, Patrick McFadin wrote:
>>>>
>>>> Is this CEP ready for a VOTE thread?
>>>> https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-37+%28DRAFT%29+Apache+Cassandra+Unified+Repair+Solution
>>>>
>>>> On Sun, Feb 25, 2024 at 12:25 PM Jaydeep Chovatia <
>>>> chovatia.jaydeep@gmail.com> wrote:
>>>>
>>>> Thanks, Josh. I've just updated the CEP
>>>> <https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-37+%28DRAFT%29+Apache+Cassandra+Official+Repair+Solution>
>>>> and included all the solutions you mentioned below.
>>>>
>>>> Jaydeep
>>>>
>>>> On Thu, Feb 22, 2024 at 9:33 AM Josh McKenzie <jm...@apache.org>
>>>> wrote:
>>>>
>>>>
>>>> Very late response from me here (basically necro'ing this thread).
>>>>
>>>> I think it'd be useful to get this condensed into a CEP that we can
>>>> then discuss in that format. It's clearly something we all agree we need
>>>> and having an implementation that works, even if it's not in your preferred
>>>> execution domain, is vastly better than nothing IMO.
>>>>
>>>> I don't have cycles (nor background ;) ) to do that, but it sounds like
>>>> you do Jaydeep given the implementation you have on a private fork + design.
>>>>
>>>> A non-exhaustive list of things that might be useful incorporating into
>>>> or referencing from a CEP:
>>>> Slack thread:
>>>> https://the-asf.slack.com/archives/CK23JSY2K/p1690225062383619
>>>> Joey's old C* ticket:
>>>> https://issues.apache.org/jira/browse/CASSANDRA-14346
>>>> Even older automatic repair scheduling:
>>>> https://issues.apache.org/jira/browse/CASSANDRA-10070
>>>> Your design gdoc:
>>>> https://docs.google.com/document/d/1CJWxjEi-mBABPMZ3VWJ9w5KavWfJETAGxfUpsViPcPo/edit#heading=h.r112r46toau0
>>>> PR with automated repair:
>>>> https://github.com/jaydeepkumar1984/cassandra/commit/ef6456d652c0d07cf29d88dfea03b73704814c2c
>>>>
>>>> My intuition is that we're all basically in agreement that this is
>>>> something the DB needs, we're all willing to bikeshed for our personal
>>>> preference on where it lives and how it's implemented, and at the end of
>>>> the day, code talks. I don't think anyone's said they'll die on the hill of
>>>> implementation details, so that feels like CEP time to me.
>>>>
>>>> If you were willing and able to get a CEP together for automated repair
>>>> based on the above material, given you've done the work and have the proof
>>>> points it's working at scale, I think this would be a *huge
>>>> contribution* to the community.
>>>>
>>>> On Thu, Aug 24, 2023, at 7:26 PM, Jaydeep Chovatia wrote:
>>>>
>>>> Is anyone going to file an official CEP for this?
>>>> As mentioned in this email thread, here is one of the solution's design
>>>> doc
>>>> <https://docs.google.com/document/d/1CJWxjEi-mBABPMZ3VWJ9w5KavWfJETAGxfUpsViPcPo/edit#heading=h.r112r46toau0>
>>>> and source code on a private Apache Cassandra patch. Could you go through
>>>> it and let me know what you think?
>>>>
>>>> Jaydeep
>>>>
>>>> On Wed, Aug 2, 2023 at 3:54 PM Jon Haddad <ru...@apache.org>
>>>> wrote:
>>>>
>>>> > That said I would happily support an effort to bring repair
>>>> scheduling to the sidecar immediately. This has nothing blocking it, and
>>>> would potentially enable the sidecar to provide an official repair
>>>> scheduling solution that is compatible with current or even previous
>>>> versions of the database.
>>>>
>>>> This is something I hadn't thought much about, and is a pretty good
>>>> argument for using the sidecar initially. There's a lot of deployments out
>>>> there and having an official repair option would be a big win.
>>>>
>>>>
>>>> On 2023/07/26 23:20:07 "C. Scott Andreas" wrote:
>>>> > I agree that it would be ideal for Cassandra to have a repair
>>>> scheduler in-DB.
>>>> >
>>>> > That said I would happily support an effort to bring repair
>>>> scheduling to the sidecar immediately. This has nothing blocking it, and
>>>> would potentially enable the sidecar to provide an official repair
>>>> scheduling solution that is compatible with current or even previous
>>>> versions of the database.
>>>> >
>>>> > Once TCM has landed, we’ll have much stronger primitives for repair
>>>> orchestration in the database itself. But I don’t think that should block
>>>> progress on a repair scheduling solution in the sidecar, and there is
>>>> nothing that would prevent someone from continuing to use a sidecar-based
>>>> solution in perpetuity if they preferred.
>>>> >
>>>> > - Scott
>>>> >
>>>> > > On Jul 26, 2023, at 3:25 PM, Jon Haddad <ru...@apache.org>
>>>> wrote:
>>>> > >
>>>> > > I'm 100% in favor of repair being part of the core DB, not the
>>>> sidecar. The current (and past) state of things where running the DB
>>>> correctly *requires* running a separate process (either community
>>>> maintained or official C* sidecar) is incredibly painful for folks. The
>>>> idea that your data integrity needs to be opt-in has never made sense to me
>>>> from the perspective of either the product or the end user.
>>>> > >
>>>> > > I've worked with way too many teams that have either configured
>>>> this incorrectly or not at all.
>>>> > >
>>>> > > Ideally Cassandra would ship with repair built in and on by
>>>> default. Power users can disable if they want to continue to maintain
>>>> their own repair tooling for some reason.
>>>> > >
>>>> > > Jon
>>>> > >
>>>> > >> On 2023/07/24 20:44:14 German Eichberger via dev wrote:
>>>> > >> All,
>>>> > >> We had a brief discussion in [2] about the Uber article [1] where
>>>> they talk about having integrated repair into Cassandra and how great that
>>>> is. I expressed my disappointment that they didn't work with the community
>>>> on that (Uber, if you are listening time to make amends 🙂) and it turns
>>>> out Joey already had the idea and wrote the code [3] - so I wanted to start
>>>> a discussion to gauge interest and maybe how to revive that effort.
>>>> > >> Thanks,
>>>> > >> German
>>>> > >> [1]
>>>> https://www.uber.com/blog/how-uber-optimized-cassandra-operations-at-scale/
>>>> > >> [2] https://the-asf.slack.com/archives/CK23JSY2K/p1690225062383619
>>>> > >> [3] https://issues.apache.org/jira/browse/CASSANDRA-14346
>>>> >
>>>>
>>>>
>>>>
>>>>
Re: [Discuss] Repair inside C*
Posted by Mick Semb Wever <mc...@apache.org>.
This is looking strong, thanks Jaydeep.
I would suggest folk take a look at the design doc and the PR in the CEP.
A lot is there (that I have completely missed).
I would especially ask all authors of prior art (Reaper, DSE nodesync,
ecchronos) to take a final review of the proposal
Jaydeep, can we ask for a two week window while we reach out to these
people ? There's a lot of prior art in this space, and it feels like we're
in a good place now where it's clear this has legs and we can use that to
bring folk in and make sure there's no remaining blindspots.
On Fri, 18 Oct 2024 at 01:40, Jaydeep Chovatia <ch...@gmail.com>
wrote:
> Sorry, there is a typo in the CEP-37 link; here is the correct link
> <https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-37+Apache+Cassandra+Unified+Repair+Solution>
>
>
> On Thu, Oct 17, 2024 at 4:36 PM Jaydeep Chovatia <
> chovatia.jaydeep@gmail.com> wrote:
>
>> First, thank you for your patience while we strengthened the CEP-37.
>>
>>
>> Over the last eight months, Chris Lohfink, Andy Tolbert, Josh McKenzie,
>> Dinesh Joshi, Kristijonas Zalys, and I have done tons of work (online
>> discussions/a dedicated Slack channel #cassandra-repair-scheduling-cep37)
>> to come up with the best possible design that not only significantly
>> simplifies repair operations but also includes the most common features
>> that everyone will benefit from running at Scale.
>>
>> For example,
>>
>> -
>>
>> Apache Cassandra must be capable of running multiple repair types,
>> such as Full, Incremental, Paxos, and Preview - so the framework should be
>> easily extendable with no additional overhead from the operator’s point of
>> view.
>> -
>>
>> An easy way to extend the token-split calculation algorithm with a
>> default implementation should exist.
>> -
>>
>> Running incremental repair reliably at Scale is pretty challenging,
>> so we need to place safeguards, such as migration/rollback w/o restart and
>> stopping incremental repair automatically if the disk is about to get full.
>>
>> We are glad to inform you that CEP-37 (i.e., Repair inside Cassandra) is
>> now officially ready for review after multiple rounds of design, testing,
>> code reviews, documentation reviews, and, more importantly, validation that
>> it runs at Scale!
>>
>>
>> Some facts about CEP-37.
>>
>> -
>>
>> Multiple members have verified all aspects of CEP-37 numerous times.
>> -
>>
>> The design proposed in CEP-37 has been thoroughly tried and tested on
>> an immense scale (hundreds of unique Cassandra clusters, tens of thousands
>> of Cassandra nodes, with tens of millions of QPS) on top of 4.1 open-source
>> for more than five years; please see more details here
>> <https://www.uber.com/en-US/blog/how-uber-optimized-cassandra-operations-at-scale/>
>> .
>> -
>>
>> The following presentation
>> <https://docs.google.com/presentation/d/1Zilww9c7LihHULk_ckErI2s4XbObxjWknKqRtbvHyZc/edit#slide=id.g30a4fd4fcf7_0_13>
>> highlights the rigorous applied to CEP-37, which was given during last
>> week’s Apache Cassandra Bay Area Meetup
>> <https://www.meetup.com/apache-cassandra-bay-area/events/303469006/>,
>>
>>
>> Since things are massively overhauled, we believe it is almost ready for
>> a final pass pre-VOTE. We would like you to please review the CEP-37
>> <https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-37+Apache+Cassandra+Unified+Repair+Solution)>
>> and the associated detailed design doc
>> <https://docs.google.com/document/d/1CJWxjEi-mBABPMZ3VWJ9w5KavWfJETAGxfUpsViPcPo/edit#heading=h.r112r46toau0>
>> .
>>
>> Thank you everyone!
>>
>> Chris, Andy, Josh, Dinesh, Kristijonas, and Jaydeep
>>
>>
>>
>> On Thu, Sep 19, 2024 at 11:26 AM Josh McKenzie <jm...@apache.org>
>> wrote:
>>
>>> Not quite; finishing touches on the CEP and design doc are in flight (as
>>> of last / this week).
>>>
>>> Soon(tm).
>>>
>>> On Thu, Sep 19, 2024, at 2:07 PM, Patrick McFadin wrote:
>>>
>>> Is this CEP ready for a VOTE thread?
>>> https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-37+%28DRAFT%29+Apache+Cassandra+Unified+Repair+Solution
>>>
>>> On Sun, Feb 25, 2024 at 12:25 PM Jaydeep Chovatia <
>>> chovatia.jaydeep@gmail.com> wrote:
>>>
>>> Thanks, Josh. I've just updated the CEP
>>> <https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-37+%28DRAFT%29+Apache+Cassandra+Official+Repair+Solution>
>>> and included all the solutions you mentioned below.
>>>
>>> Jaydeep
>>>
>>> On Thu, Feb 22, 2024 at 9:33 AM Josh McKenzie <jm...@apache.org>
>>> wrote:
>>>
>>>
>>> Very late response from me here (basically necro'ing this thread).
>>>
>>> I think it'd be useful to get this condensed into a CEP that we can then
>>> discuss in that format. It's clearly something we all agree we need and
>>> having an implementation that works, even if it's not in your preferred
>>> execution domain, is vastly better than nothing IMO.
>>>
>>> I don't have cycles (nor background ;) ) to do that, but it sounds like
>>> you do Jaydeep given the implementation you have on a private fork + design.
>>>
>>> A non-exhaustive list of things that might be useful incorporating into
>>> or referencing from a CEP:
>>> Slack thread:
>>> https://the-asf.slack.com/archives/CK23JSY2K/p1690225062383619
>>> Joey's old C* ticket:
>>> https://issues.apache.org/jira/browse/CASSANDRA-14346
>>> Even older automatic repair scheduling:
>>> https://issues.apache.org/jira/browse/CASSANDRA-10070
>>> Your design gdoc:
>>> https://docs.google.com/document/d/1CJWxjEi-mBABPMZ3VWJ9w5KavWfJETAGxfUpsViPcPo/edit#heading=h.r112r46toau0
>>> PR with automated repair:
>>> https://github.com/jaydeepkumar1984/cassandra/commit/ef6456d652c0d07cf29d88dfea03b73704814c2c
>>>
>>> My intuition is that we're all basically in agreement that this is
>>> something the DB needs, we're all willing to bikeshed for our personal
>>> preference on where it lives and how it's implemented, and at the end of
>>> the day, code talks. I don't think anyone's said they'll die on the hill of
>>> implementation details, so that feels like CEP time to me.
>>>
>>> If you were willing and able to get a CEP together for automated repair
>>> based on the above material, given you've done the work and have the proof
>>> points it's working at scale, I think this would be a *huge
>>> contribution* to the community.
>>>
>>> On Thu, Aug 24, 2023, at 7:26 PM, Jaydeep Chovatia wrote:
>>>
>>> Is anyone going to file an official CEP for this?
>>> As mentioned in this email thread, here is one of the solution's design
>>> doc
>>> <https://docs.google.com/document/d/1CJWxjEi-mBABPMZ3VWJ9w5KavWfJETAGxfUpsViPcPo/edit#heading=h.r112r46toau0>
>>> and source code on a private Apache Cassandra patch. Could you go through
>>> it and let me know what you think?
>>>
>>> Jaydeep
>>>
>>> On Wed, Aug 2, 2023 at 3:54 PM Jon Haddad <ru...@apache.org>
>>> wrote:
>>>
>>> > That said I would happily support an effort to bring repair scheduling
>>> to the sidecar immediately. This has nothing blocking it, and would
>>> potentially enable the sidecar to provide an official repair scheduling
>>> solution that is compatible with current or even previous versions of the
>>> database.
>>>
>>> This is something I hadn't thought much about, and is a pretty good
>>> argument for using the sidecar initially. There's a lot of deployments out
>>> there and having an official repair option would be a big win.
>>>
>>>
>>> On 2023/07/26 23:20:07 "C. Scott Andreas" wrote:
>>> > I agree that it would be ideal for Cassandra to have a repair
>>> scheduler in-DB.
>>> >
>>> > That said I would happily support an effort to bring repair scheduling
>>> to the sidecar immediately. This has nothing blocking it, and would
>>> potentially enable the sidecar to provide an official repair scheduling
>>> solution that is compatible with current or even previous versions of the
>>> database.
>>> >
>>> > Once TCM has landed, we’ll have much stronger primitives for repair
>>> orchestration in the database itself. But I don’t think that should block
>>> progress on a repair scheduling solution in the sidecar, and there is
>>> nothing that would prevent someone from continuing to use a sidecar-based
>>> solution in perpetuity if they preferred.
>>> >
>>> > - Scott
>>> >
>>> > > On Jul 26, 2023, at 3:25 PM, Jon Haddad <ru...@apache.org>
>>> wrote:
>>> > >
>>> > > I'm 100% in favor of repair being part of the core DB, not the
>>> sidecar. The current (and past) state of things where running the DB
>>> correctly *requires* running a separate process (either community
>>> maintained or official C* sidecar) is incredibly painful for folks. The
>>> idea that your data integrity needs to be opt-in has never made sense to me
>>> from the perspective of either the product or the end user.
>>> > >
>>> > > I've worked with way too many teams that have either configured this
>>> incorrectly or not at all.
>>> > >
>>> > > Ideally Cassandra would ship with repair built in and on by
>>> default. Power users can disable if they want to continue to maintain
>>> their own repair tooling for some reason.
>>> > >
>>> > > Jon
>>> > >
>>> > >> On 2023/07/24 20:44:14 German Eichberger via dev wrote:
>>> > >> All,
>>> > >> We had a brief discussion in [2] about the Uber article [1] where
>>> they talk about having integrated repair into Cassandra and how great that
>>> is. I expressed my disappointment that they didn't work with the community
>>> on that (Uber, if you are listening time to make amends 🙂) and it turns
>>> out Joey already had the idea and wrote the code [3] - so I wanted to start
>>> a discussion to gauge interest and maybe how to revive that effort.
>>> > >> Thanks,
>>> > >> German
>>> > >> [1]
>>> https://www.uber.com/blog/how-uber-optimized-cassandra-operations-at-scale/
>>> > >> [2] https://the-asf.slack.com/archives/CK23JSY2K/p1690225062383619
>>> > >> [3] https://issues.apache.org/jira/browse/CASSANDRA-14346
>>> >
>>>
>>>
>>>
>>>
Re: [Discuss] Repair inside C*
Posted by Jaydeep Chovatia <ch...@gmail.com>.
Sorry, there is a typo in the CEP-37 link; here is the correct link
<https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-37+Apache+Cassandra+Unified+Repair+Solution>
On Thu, Oct 17, 2024 at 4:36 PM Jaydeep Chovatia <ch...@gmail.com>
wrote:
> First, thank you for your patience while we strengthened the CEP-37.
>
>
> Over the last eight months, Chris Lohfink, Andy Tolbert, Josh McKenzie,
> Dinesh Joshi, Kristijonas Zalys, and I have done tons of work (online
> discussions/a dedicated Slack channel #cassandra-repair-scheduling-cep37)
> to come up with the best possible design that not only significantly
> simplifies repair operations but also includes the most common features
> that everyone will benefit from running at Scale.
>
> For example,
>
> -
>
> Apache Cassandra must be capable of running multiple repair types,
> such as Full, Incremental, Paxos, and Preview - so the framework should be
> easily extendable with no additional overhead from the operator’s point of
> view.
> -
>
> An easy way to extend the token-split calculation algorithm with a
> default implementation should exist.
> -
>
> Running incremental repair reliably at Scale is pretty challenging, so
> we need to place safeguards, such as migration/rollback w/o restart and
> stopping incremental repair automatically if the disk is about to get full.
>
> We are glad to inform you that CEP-37 (i.e., Repair inside Cassandra) is
> now officially ready for review after multiple rounds of design, testing,
> code reviews, documentation reviews, and, more importantly, validation that
> it runs at Scale!
>
>
> Some facts about CEP-37.
>
> -
>
> Multiple members have verified all aspects of CEP-37 numerous times.
> -
>
> The design proposed in CEP-37 has been thoroughly tried and tested on
> an immense scale (hundreds of unique Cassandra clusters, tens of thousands
> of Cassandra nodes, with tens of millions of QPS) on top of 4.1 open-source
> for more than five years; please see more details here
> <https://www.uber.com/en-US/blog/how-uber-optimized-cassandra-operations-at-scale/>
> .
> -
>
> The following presentation
> <https://docs.google.com/presentation/d/1Zilww9c7LihHULk_ckErI2s4XbObxjWknKqRtbvHyZc/edit#slide=id.g30a4fd4fcf7_0_13>
> highlights the rigorous applied to CEP-37, which was given during last
> week’s Apache Cassandra Bay Area Meetup
> <https://www.meetup.com/apache-cassandra-bay-area/events/303469006/>,
>
>
> Since things are massively overhauled, we believe it is almost ready for a
> final pass pre-VOTE. We would like you to please review the CEP-37
> <https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-37+Apache+Cassandra+Unified+Repair+Solution)>
> and the associated detailed design doc
> <https://docs.google.com/document/d/1CJWxjEi-mBABPMZ3VWJ9w5KavWfJETAGxfUpsViPcPo/edit#heading=h.r112r46toau0>
> .
>
> Thank you everyone!
>
> Chris, Andy, Josh, Dinesh, Kristijonas, and Jaydeep
>
>
>
> On Thu, Sep 19, 2024 at 11:26 AM Josh McKenzie <jm...@apache.org>
> wrote:
>
>> Not quite; finishing touches on the CEP and design doc are in flight (as
>> of last / this week).
>>
>> Soon(tm).
>>
>> On Thu, Sep 19, 2024, at 2:07 PM, Patrick McFadin wrote:
>>
>> Is this CEP ready for a VOTE thread?
>> https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-37+%28DRAFT%29+Apache+Cassandra+Unified+Repair+Solution
>>
>> On Sun, Feb 25, 2024 at 12:25 PM Jaydeep Chovatia <
>> chovatia.jaydeep@gmail.com> wrote:
>>
>> Thanks, Josh. I've just updated the CEP
>> <https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-37+%28DRAFT%29+Apache+Cassandra+Official+Repair+Solution>
>> and included all the solutions you mentioned below.
>>
>> Jaydeep
>>
>> On Thu, Feb 22, 2024 at 9:33 AM Josh McKenzie <jm...@apache.org>
>> wrote:
>>
>>
>> Very late response from me here (basically necro'ing this thread).
>>
>> I think it'd be useful to get this condensed into a CEP that we can then
>> discuss in that format. It's clearly something we all agree we need and
>> having an implementation that works, even if it's not in your preferred
>> execution domain, is vastly better than nothing IMO.
>>
>> I don't have cycles (nor background ;) ) to do that, but it sounds like
>> you do Jaydeep given the implementation you have on a private fork + design.
>>
>> A non-exhaustive list of things that might be useful incorporating into
>> or referencing from a CEP:
>> Slack thread:
>> https://the-asf.slack.com/archives/CK23JSY2K/p1690225062383619
>> Joey's old C* ticket:
>> https://issues.apache.org/jira/browse/CASSANDRA-14346
>> Even older automatic repair scheduling:
>> https://issues.apache.org/jira/browse/CASSANDRA-10070
>> Your design gdoc:
>> https://docs.google.com/document/d/1CJWxjEi-mBABPMZ3VWJ9w5KavWfJETAGxfUpsViPcPo/edit#heading=h.r112r46toau0
>> PR with automated repair:
>> https://github.com/jaydeepkumar1984/cassandra/commit/ef6456d652c0d07cf29d88dfea03b73704814c2c
>>
>> My intuition is that we're all basically in agreement that this is
>> something the DB needs, we're all willing to bikeshed for our personal
>> preference on where it lives and how it's implemented, and at the end of
>> the day, code talks. I don't think anyone's said they'll die on the hill of
>> implementation details, so that feels like CEP time to me.
>>
>> If you were willing and able to get a CEP together for automated repair
>> based on the above material, given you've done the work and have the proof
>> points it's working at scale, I think this would be a *huge contribution*
>> to the community.
>>
>> On Thu, Aug 24, 2023, at 7:26 PM, Jaydeep Chovatia wrote:
>>
>> Is anyone going to file an official CEP for this?
>> As mentioned in this email thread, here is one of the solution's design
>> doc
>> <https://docs.google.com/document/d/1CJWxjEi-mBABPMZ3VWJ9w5KavWfJETAGxfUpsViPcPo/edit#heading=h.r112r46toau0>
>> and source code on a private Apache Cassandra patch. Could you go through
>> it and let me know what you think?
>>
>> Jaydeep
>>
>> On Wed, Aug 2, 2023 at 3:54 PM Jon Haddad <ru...@apache.org>
>> wrote:
>>
>> > That said I would happily support an effort to bring repair scheduling
>> to the sidecar immediately. This has nothing blocking it, and would
>> potentially enable the sidecar to provide an official repair scheduling
>> solution that is compatible with current or even previous versions of the
>> database.
>>
>> This is something I hadn't thought much about, and is a pretty good
>> argument for using the sidecar initially. There's a lot of deployments out
>> there and having an official repair option would be a big win.
>>
>>
>> On 2023/07/26 23:20:07 "C. Scott Andreas" wrote:
>> > I agree that it would be ideal for Cassandra to have a repair scheduler
>> in-DB.
>> >
>> > That said I would happily support an effort to bring repair scheduling
>> to the sidecar immediately. This has nothing blocking it, and would
>> potentially enable the sidecar to provide an official repair scheduling
>> solution that is compatible with current or even previous versions of the
>> database.
>> >
>> > Once TCM has landed, we’ll have much stronger primitives for repair
>> orchestration in the database itself. But I don’t think that should block
>> progress on a repair scheduling solution in the sidecar, and there is
>> nothing that would prevent someone from continuing to use a sidecar-based
>> solution in perpetuity if they preferred.
>> >
>> > - Scott
>> >
>> > > On Jul 26, 2023, at 3:25 PM, Jon Haddad <ru...@apache.org>
>> wrote:
>> > >
>> > > I'm 100% in favor of repair being part of the core DB, not the
>> sidecar. The current (and past) state of things where running the DB
>> correctly *requires* running a separate process (either community
>> maintained or official C* sidecar) is incredibly painful for folks. The
>> idea that your data integrity needs to be opt-in has never made sense to me
>> from the perspective of either the product or the end user.
>> > >
>> > > I've worked with way too many teams that have either configured this
>> incorrectly or not at all.
>> > >
>> > > Ideally Cassandra would ship with repair built in and on by default.
>> Power users can disable if they want to continue to maintain their own
>> repair tooling for some reason.
>> > >
>> > > Jon
>> > >
>> > >> On 2023/07/24 20:44:14 German Eichberger via dev wrote:
>> > >> All,
>> > >> We had a brief discussion in [2] about the Uber article [1] where
>> they talk about having integrated repair into Cassandra and how great that
>> is. I expressed my disappointment that they didn't work with the community
>> on that (Uber, if you are listening time to make amends 🙂) and it turns
>> out Joey already had the idea and wrote the code [3] - so I wanted to start
>> a discussion to gauge interest and maybe how to revive that effort.
>> > >> Thanks,
>> > >> German
>> > >> [1]
>> https://www.uber.com/blog/how-uber-optimized-cassandra-operations-at-scale/
>> > >> [2] https://the-asf.slack.com/archives/CK23JSY2K/p1690225062383619
>> > >> [3] https://issues.apache.org/jira/browse/CASSANDRA-14346
>> >
>>
>>
>>
>>
Re: [Discuss] Repair inside C*
Posted by Jaydeep Chovatia <ch...@gmail.com>.
First, thank you for your patience while we strengthened the CEP-37.
Over the last eight months, Chris Lohfink, Andy Tolbert, Josh McKenzie,
Dinesh Joshi, Kristijonas Zalys, and I have done tons of work (online
discussions/a dedicated Slack channel #cassandra-repair-scheduling-cep37)
to come up with the best possible design that not only significantly
simplifies repair operations but also includes the most common features
that everyone will benefit from running at Scale.
For example,
-
Apache Cassandra must be capable of running multiple repair types, such
as Full, Incremental, Paxos, and Preview - so the framework should be
easily extendable with no additional overhead from the operator’s point of
view.
-
An easy way to extend the token-split calculation algorithm with a
default implementation should exist.
-
Running incremental repair reliably at Scale is pretty challenging, so
we need to place safeguards, such as migration/rollback w/o restart and
stopping incremental repair automatically if the disk is about to get full.
We are glad to inform you that CEP-37 (i.e., Repair inside Cassandra) is
now officially ready for review after multiple rounds of design, testing,
code reviews, documentation reviews, and, more importantly, validation that
it runs at Scale!
Some facts about CEP-37.
-
Multiple members have verified all aspects of CEP-37 numerous times.
-
The design proposed in CEP-37 has been thoroughly tried and tested on an
immense scale (hundreds of unique Cassandra clusters, tens of thousands of
Cassandra nodes, with tens of millions of QPS) on top of 4.1 open-source
for more than five years; please see more details here
<https://www.uber.com/en-US/blog/how-uber-optimized-cassandra-operations-at-scale/>
.
-
The following presentation
<https://docs.google.com/presentation/d/1Zilww9c7LihHULk_ckErI2s4XbObxjWknKqRtbvHyZc/edit#slide=id.g30a4fd4fcf7_0_13>
highlights the rigorous applied to CEP-37, which was given during last
week’s Apache Cassandra Bay Area Meetup
<https://www.meetup.com/apache-cassandra-bay-area/events/303469006/>,
Since things are massively overhauled, we believe it is almost ready for a
final pass pre-VOTE. We would like you to please review the CEP-37
<https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-37+Apache+Cassandra+Unified+Repair+Solution)>
and the associated detailed design doc
<https://docs.google.com/document/d/1CJWxjEi-mBABPMZ3VWJ9w5KavWfJETAGxfUpsViPcPo/edit#heading=h.r112r46toau0>
.
Thank you everyone!
Chris, Andy, Josh, Dinesh, Kristijonas, and Jaydeep
On Thu, Sep 19, 2024 at 11:26 AM Josh McKenzie <jm...@apache.org> wrote:
> Not quite; finishing touches on the CEP and design doc are in flight (as
> of last / this week).
>
> Soon(tm).
>
> On Thu, Sep 19, 2024, at 2:07 PM, Patrick McFadin wrote:
>
> Is this CEP ready for a VOTE thread?
> https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-37+%28DRAFT%29+Apache+Cassandra+Unified+Repair+Solution
>
> On Sun, Feb 25, 2024 at 12:25 PM Jaydeep Chovatia <
> chovatia.jaydeep@gmail.com> wrote:
>
> Thanks, Josh. I've just updated the CEP
> <https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-37+%28DRAFT%29+Apache+Cassandra+Official+Repair+Solution>
> and included all the solutions you mentioned below.
>
> Jaydeep
>
> On Thu, Feb 22, 2024 at 9:33 AM Josh McKenzie <jm...@apache.org>
> wrote:
>
>
> Very late response from me here (basically necro'ing this thread).
>
> I think it'd be useful to get this condensed into a CEP that we can then
> discuss in that format. It's clearly something we all agree we need and
> having an implementation that works, even if it's not in your preferred
> execution domain, is vastly better than nothing IMO.
>
> I don't have cycles (nor background ;) ) to do that, but it sounds like
> you do Jaydeep given the implementation you have on a private fork + design.
>
> A non-exhaustive list of things that might be useful incorporating into or
> referencing from a CEP:
> Slack thread:
> https://the-asf.slack.com/archives/CK23JSY2K/p1690225062383619
> Joey's old C* ticket:
> https://issues.apache.org/jira/browse/CASSANDRA-14346
> Even older automatic repair scheduling:
> https://issues.apache.org/jira/browse/CASSANDRA-10070
> Your design gdoc:
> https://docs.google.com/document/d/1CJWxjEi-mBABPMZ3VWJ9w5KavWfJETAGxfUpsViPcPo/edit#heading=h.r112r46toau0
> PR with automated repair:
> https://github.com/jaydeepkumar1984/cassandra/commit/ef6456d652c0d07cf29d88dfea03b73704814c2c
>
> My intuition is that we're all basically in agreement that this is
> something the DB needs, we're all willing to bikeshed for our personal
> preference on where it lives and how it's implemented, and at the end of
> the day, code talks. I don't think anyone's said they'll die on the hill of
> implementation details, so that feels like CEP time to me.
>
> If you were willing and able to get a CEP together for automated repair
> based on the above material, given you've done the work and have the proof
> points it's working at scale, I think this would be a *huge contribution*
> to the community.
>
> On Thu, Aug 24, 2023, at 7:26 PM, Jaydeep Chovatia wrote:
>
> Is anyone going to file an official CEP for this?
> As mentioned in this email thread, here is one of the solution's design
> doc
> <https://docs.google.com/document/d/1CJWxjEi-mBABPMZ3VWJ9w5KavWfJETAGxfUpsViPcPo/edit#heading=h.r112r46toau0>
> and source code on a private Apache Cassandra patch. Could you go through
> it and let me know what you think?
>
> Jaydeep
>
> On Wed, Aug 2, 2023 at 3:54 PM Jon Haddad <ru...@apache.org>
> wrote:
>
> > That said I would happily support an effort to bring repair scheduling
> to the sidecar immediately. This has nothing blocking it, and would
> potentially enable the sidecar to provide an official repair scheduling
> solution that is compatible with current or even previous versions of the
> database.
>
> This is something I hadn't thought much about, and is a pretty good
> argument for using the sidecar initially. There's a lot of deployments out
> there and having an official repair option would be a big win.
>
>
> On 2023/07/26 23:20:07 "C. Scott Andreas" wrote:
> > I agree that it would be ideal for Cassandra to have a repair scheduler
> in-DB.
> >
> > That said I would happily support an effort to bring repair scheduling
> to the sidecar immediately. This has nothing blocking it, and would
> potentially enable the sidecar to provide an official repair scheduling
> solution that is compatible with current or even previous versions of the
> database.
> >
> > Once TCM has landed, we’ll have much stronger primitives for repair
> orchestration in the database itself. But I don’t think that should block
> progress on a repair scheduling solution in the sidecar, and there is
> nothing that would prevent someone from continuing to use a sidecar-based
> solution in perpetuity if they preferred.
> >
> > - Scott
> >
> > > On Jul 26, 2023, at 3:25 PM, Jon Haddad <ru...@apache.org>
> wrote:
> > >
> > > I'm 100% in favor of repair being part of the core DB, not the
> sidecar. The current (and past) state of things where running the DB
> correctly *requires* running a separate process (either community
> maintained or official C* sidecar) is incredibly painful for folks. The
> idea that your data integrity needs to be opt-in has never made sense to me
> from the perspective of either the product or the end user.
> > >
> > > I've worked with way too many teams that have either configured this
> incorrectly or not at all.
> > >
> > > Ideally Cassandra would ship with repair built in and on by default.
> Power users can disable if they want to continue to maintain their own
> repair tooling for some reason.
> > >
> > > Jon
> > >
> > >> On 2023/07/24 20:44:14 German Eichberger via dev wrote:
> > >> All,
> > >> We had a brief discussion in [2] about the Uber article [1] where
> they talk about having integrated repair into Cassandra and how great that
> is. I expressed my disappointment that they didn't work with the community
> on that (Uber, if you are listening time to make amends 🙂) and it turns
> out Joey already had the idea and wrote the code [3] - so I wanted to start
> a discussion to gauge interest and maybe how to revive that effort.
> > >> Thanks,
> > >> German
> > >> [1]
> https://www.uber.com/blog/how-uber-optimized-cassandra-operations-at-scale/
> > >> [2] https://the-asf.slack.com/archives/CK23JSY2K/p1690225062383619
> > >> [3] https://issues.apache.org/jira/browse/CASSANDRA-14346
> >
>
>
>
>
Re: [Discuss] Repair inside C*
Posted by Josh McKenzie <jm...@apache.org>.
Not quite; finishing touches on the CEP and design doc are in flight (as of last / this week).
Soon(tm).
On Thu, Sep 19, 2024, at 2:07 PM, Patrick McFadin wrote:
> Is this CEP ready for a VOTE thread? https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-37+%28DRAFT%29+Apache+Cassandra+Unified+Repair+Solution
>
> On Sun, Feb 25, 2024 at 12:25 PM Jaydeep Chovatia <ch...@gmail.com> wrote:
>> Thanks, Josh. I've just updated the CEP <https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-37+%28DRAFT%29+Apache+Cassandra+Official+Repair+Solution> and included all the solutions you mentioned below.
>>
>> Jaydeep
>>
>> On Thu, Feb 22, 2024 at 9:33 AM Josh McKenzie <jm...@apache.org> wrote:
>>> __
>>> Very late response from me here (basically necro'ing this thread).
>>>
>>> I think it'd be useful to get this condensed into a CEP that we can then discuss in that format. It's clearly something we all agree we need and having an implementation that works, even if it's not in your preferred execution domain, is vastly better than nothing IMO.
>>>
>>> I don't have cycles (nor background ;) ) to do that, but it sounds like you do Jaydeep given the implementation you have on a private fork + design.
>>>
>>> A non-exhaustive list of things that might be useful incorporating into or referencing from a CEP:
>>> Slack thread: https://the-asf.slack.com/archives/CK23JSY2K/p1690225062383619
>>> Joey's old C* ticket: https://issues.apache.org/jira/browse/CASSANDRA-14346
>>> Even older automatic repair scheduling: https://issues.apache.org/jira/browse/CASSANDRA-10070
>>> Your design gdoc: https://docs.google.com/document/d/1CJWxjEi-mBABPMZ3VWJ9w5KavWfJETAGxfUpsViPcPo/edit#heading=h.r112r46toau0
>>> PR with automated repair: https://github.com/jaydeepkumar1984/cassandra/commit/ef6456d652c0d07cf29d88dfea03b73704814c2c
>>>
>>> My intuition is that we're all basically in agreement that this is something the DB needs, we're all willing to bikeshed for our personal preference on where it lives and how it's implemented, and at the end of the day, code talks. I don't think anyone's said they'll die on the hill of implementation details, so that feels like CEP time to me.
>>>
>>> If you were willing and able to get a CEP together for automated repair based on the above material, given you've done the work and have the proof points it's working at scale, I think this would be a *huge contribution* to the community.
>>>
>>> On Thu, Aug 24, 2023, at 7:26 PM, Jaydeep Chovatia wrote:
>>>> Is anyone going to file an official CEP for this?
>>>> As mentioned in this email thread, here is one of the solution's design doc <https://docs.google.com/document/d/1CJWxjEi-mBABPMZ3VWJ9w5KavWfJETAGxfUpsViPcPo/edit#heading=h.r112r46toau0> and source code on a private Apache Cassandra patch. Could you go through it and let me know what you think?
>>>>
>>>> Jaydeep
>>>>
>>>> On Wed, Aug 2, 2023 at 3:54 PM Jon Haddad <ru...@apache.org> wrote:
>>>>> > That said I would happily support an effort to bring repair scheduling to the sidecar immediately. This has nothing blocking it, and would potentially enable the sidecar to provide an official repair scheduling solution that is compatible with current or even previous versions of the database.
>>>>>
>>>>> This is something I hadn't thought much about, and is a pretty good argument for using the sidecar initially. There's a lot of deployments out there and having an official repair option would be a big win.
>>>>>
>>>>>
>>>>> On 2023/07/26 23:20:07 "C. Scott Andreas" wrote:
>>>>> > I agree that it would be ideal for Cassandra to have a repair scheduler in-DB.
>>>>> >
>>>>> > That said I would happily support an effort to bring repair scheduling to the sidecar immediately. This has nothing blocking it, and would potentially enable the sidecar to provide an official repair scheduling solution that is compatible with current or even previous versions of the database.
>>>>> >
>>>>> > Once TCM has landed, we’ll have much stronger primitives for repair orchestration in the database itself. But I don’t think that should block progress on a repair scheduling solution in the sidecar, and there is nothing that would prevent someone from continuing to use a sidecar-based solution in perpetuity if they preferred.
>>>>> >
>>>>> > - Scott
>>>>> >
>>>>> > > On Jul 26, 2023, at 3:25 PM, Jon Haddad <ru...@apache.org> wrote:
>>>>> > >
>>>>> > > I'm 100% in favor of repair being part of the core DB, not the sidecar. The current (and past) state of things where running the DB correctly *requires* running a separate process (either community maintained or official C* sidecar) is incredibly painful for folks. The idea that your data integrity needs to be opt-in has never made sense to me from the perspective of either the product or the end user.
>>>>> > >
>>>>> > > I've worked with way too many teams that have either configured this incorrectly or not at all.
>>>>> > >
>>>>> > > Ideally Cassandra would ship with repair built in and on by default. Power users can disable if they want to continue to maintain their own repair tooling for some reason.
>>>>> > >
>>>>> > > Jon
>>>>> > >
>>>>> > >> On 2023/07/24 20:44:14 German Eichberger via dev wrote:
>>>>> > >> All,
>>>>> > >> We had a brief discussion in [2] about the Uber article [1] where they talk about having integrated repair into Cassandra and how great that is. I expressed my disappointment that they didn't work with the community on that (Uber, if you are listening time to make amends 🙂) and it turns out Joey already had the idea and wrote the code [3] - so I wanted to start a discussion to gauge interest and maybe how to revive that effort.
>>>>> > >> Thanks,
>>>>> > >> German
>>>>> > >> [1] https://www.uber.com/blog/how-uber-optimized-cassandra-operations-at-scale/
>>>>> > >> [2] https://the-asf.slack.com/archives/CK23JSY2K/p1690225062383619
>>>>> > >> [3] https://issues.apache.org/jira/browse/CASSANDRA-14346
>>>>> >
>>>