You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@cassandra.apache.org by Scott Carey <sc...@apache.org> on 2021/06/29 22:49:06 UTC
[DISCUSS] CASSANDRA-16767, CASSANDRA-16768, and CASSANDRA-16769 for 3.11.x
I'd like to discuss the inclusion of the above tickets for a 3.11.x
release. These are not a pure 'bug fix' so I'll need a waiver to get them
into 3.11.x (and implicitly, 4.0.x).
The first two are straightforward oversights: neither *nodetool
garbagecollect *nor *nodetool scrub* currently accept a *--user-defined*
parameter list of SSTables in the same way that *nodetool compact* does.
This is an operational problem for large tables.
I often need to scrub just one file that is corrupted for some reason, and
not scrub an entire 1TB+ of data for a table on a node. This renders
'nodetool scrub' operationally useless for large tables.
For *garbagecollect* it is often operationally easy to identify which
tables are likely to be full of bloa- and operationally useful to do this
task in small increments. The existing order that garbagecollect processes
SSTables prevents it from being useful in any incremental fashion -- if you
stop it and later restart, it will first process the SSTables you just
garbage collected.
The third ticket adds an option for* nodetool garbagecollect*,
*--oldest-fraction* that can select a fraction of the oldest table data in
bytes, and garbagecollect only the SSTables that 'cover' that percentage of
data. Operationally, this lends itself to easy automation -- for example
running this once a week on 10% of a table's data would imply that there is
no data on disk that has been overwritten within the last 10 weeks. This
caps data bloat in ways neither LCS nor STCS can currently achieve without
regular major compactions or full-pass garbagecollect.
I have a large LCS table that has existed in steady state for about two
years. Its oldest SSTable files were about 20 months old. These old tables
were 95% bloated by that time -- 'garbagecollect' was able to shrink those
to 5% of their original size.
Being able to automate garbagecollect on a small fraction of the older data
would be a big disk space and performance win, without the downsides of a
major compaction.
The overall risk of these additions is low:
- They do not modify any existing behavior, only add new options.
- They re-use existing machinery for most of the work, and only adds
logic in areas that are already well tested. The areas that need the most
scrutiny in review have good test coverage.
- scripts that worked with nodetool before should continue to work
except for the case where a keyspace is named --user-defined or
--oldest-fraction, but this flaw already exists with other nodetool
commands.
- Three is no modification to sensitive areas like the read, write, or
autocompaction path. This merely does the same thing that is already done,
just on a subset of SSTables rather than all of them.
Thanks for considering this proposal,
-Scott Carey
P.S.
You might wonder why the --oldest-fraction is necessary when one can use
--user-defined and some OS level scripting.
1. --oldest-fraction calculates the SSTable fraction based on the total
data size, not file count.
2. nodetool can avoid race conditions with autocompaction on sstable
selection
3. nodetool has access to the current state of active SSTables, a script
just sees files on disk, files that might be scheduled for delete or files
that are actively being written to.
4. Even if used at a 100% fraction, it processes from oldest to newest
by the SSTable generation number, meaning that if it is interrupted half
way through, then re-started, it won't immediately work on the files that
were just processed, as those will have the largest generation number.
Re: [DISCUSS] CASSANDRA-16767, CASSANDRA-16768, and CASSANDRA-16769
for 3.11.x
Posted by "benedict@apache.org" <be...@apache.org>.
Hi Scott,
I wonder if it’s possible that too few people who saw your email consider themselves sufficiently involved in this part of the codebase. People tend to keep quiet about stuff they don’t participate in deeply, which is why I haven’t responded – and I wonder if this might explain the tumbleweed. I wonder how we might generally track if areas of the codebase are adequately covered by active contributors.
To answer your question, I don’t personally believe it is problematic to add additional features to command line tools in a patch version – they’re not scary systems where new features introduce much risk of high impact bugs. Others have stricter interpretations of the rules, but if they haven’t spoken up yet I’d say you’re clear to post some patches – but you might want to first make sure there’s somebody able and willing to review them.
From: Scott Carey <sc...@apache.org>
Date: Monday, 9 August 2021 at 20:12
To: dev@cassandra.apache.org <de...@cassandra.apache.org>
Subject: Re: [DISCUSS] CASSANDRA-16767, CASSANDRA-16768, and CASSANDRA-16769 for 3.11.x
Thank you Brandon, for answering my questions on slack, and providing early
feedback on these ideas more than a month before I created the patches and
replying here.
Does anyone else have any comments or opinions? Can a decision be reached
one way or another? It is my understanding that we'll need more than one
+1 to move forward here.
I understand that the 4.0 release was a busy time, and that many probably
saw this, thought about replying, but got too busy and never did.
However, in light of the recent discussions around attracting new
contributors, I would like to highlight that being left in limbo with no
resolution is worse than being told "no", especially for new contributors.
On Fri, Jul 2, 2021 at 1:23 PM Brandon Williams <dr...@gmail.com> wrote:
> On Tue, Jun 29, 2021 at 5:49 PM Scott Carey <sc...@apache.org> wrote:
> >
> > I'd like to discuss the inclusion of the above tickets for a 3.11.x
> > release. These are not a pure 'bug fix' so I'll need a waiver to get
> them
> > into 3.11.x (and implicitly, 4.0.x).
> >
> > The first two are straightforward oversights: neither *nodetool
> > garbagecollect *nor *nodetool scrub* currently accept a *--user-defined*
> > parameter list of SSTables in the same way that *nodetool compact* does.
> >
> > This is an operational problem for large tables.
> >
> > I often need to scrub just one file that is corrupted for some reason,
> and
> > not scrub an entire 1TB+ of data for a table on a node. This renders
> > 'nodetool scrub' operationally useless for large tables.
>
> I think that given not having user defined options for these
> compaction types is clearly an oversight, and that the alternative of
> deleting the large 1TB+ sstable and then repairing is a cure worse
> than the disease, this should be added to 3.11.x and 4.0.x. I am +1
> here.
>
> > For *garbagecollect* it is often operationally easy to identify which
> > tables are likely to be full of bloa- and operationally useful to do this
> > task in small increments. The existing order that garbagecollect
> processes
> > SSTables prevents it from being useful in any incremental fashion -- if
> you
> > stop it and later restart, it will first process the SSTables you just
> > garbage collected.
> >
> > The third ticket adds an option for* nodetool garbagecollect*,
> > *--oldest-fraction* that can select a fraction of the oldest table data
> in
> > bytes, and garbagecollect only the SSTables that 'cover' that percentage
> of
> > data. Operationally, this lends itself to easy automation -- for example
> > running this once a week on 10% of a table's data would imply that there
> is
> > no data on disk that has been overwritten within the last 10 weeks. This
> > caps data bloat in ways neither LCS nor STCS can currently achieve
> without
> > regular major compactions or full-pass garbagecollect.
>
> This is a less obvious thing to be added, and I personally lack the
> operational experience to comment on how much relief this would
> provide firsthand, so I'll leave that to others. But it does make
> sense to me and since it isn't heavily modifying anything my
> inclination is that this could be an acceptable addition as well.
>
> Kind Regards,
> Brandon
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@cassandra.apache.org
> For additional commands, e-mail: dev-help@cassandra.apache.org
>
>
Re: [DISCUSS] CASSANDRA-16767, CASSANDRA-16768, and CASSANDRA-16769
for 3.11.x
Posted by Brandon Williams <dr...@gmail.com>.
On Mon, Aug 9, 2021 at 2:05 PM Scott Carey <sc...@apache.org> wrote:
> Does anyone else have any comments or opinions? Can a decision be reached
> one way or another? It is my understanding that we'll need more than one
> +1 to move forward here.
Lazy consensus says 72 hours
(https://community.apache.org/committers/lazyConsensus.html) with
considerations for availability, but I think it has been more than
long enough at this point. If no one else objects after this
prodding, I think you should move forward.
Kind Regards,
Brandon
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@cassandra.apache.org
For additional commands, e-mail: dev-help@cassandra.apache.org
Re: [DISCUSS] CASSANDRA-16767, CASSANDRA-16768, and CASSANDRA-16769
for 3.11.x
Posted by Scott Carey <sc...@apache.org>.
Thank you Brandon, for answering my questions on slack, and providing early
feedback on these ideas more than a month before I created the patches and
replying here.
Does anyone else have any comments or opinions? Can a decision be reached
one way or another? It is my understanding that we'll need more than one
+1 to move forward here.
I understand that the 4.0 release was a busy time, and that many probably
saw this, thought about replying, but got too busy and never did.
However, in light of the recent discussions around attracting new
contributors, I would like to highlight that being left in limbo with no
resolution is worse than being told "no", especially for new contributors.
On Fri, Jul 2, 2021 at 1:23 PM Brandon Williams <dr...@gmail.com> wrote:
> On Tue, Jun 29, 2021 at 5:49 PM Scott Carey <sc...@apache.org> wrote:
> >
> > I'd like to discuss the inclusion of the above tickets for a 3.11.x
> > release. These are not a pure 'bug fix' so I'll need a waiver to get
> them
> > into 3.11.x (and implicitly, 4.0.x).
> >
> > The first two are straightforward oversights: neither *nodetool
> > garbagecollect *nor *nodetool scrub* currently accept a *--user-defined*
> > parameter list of SSTables in the same way that *nodetool compact* does.
> >
> > This is an operational problem for large tables.
> >
> > I often need to scrub just one file that is corrupted for some reason,
> and
> > not scrub an entire 1TB+ of data for a table on a node. This renders
> > 'nodetool scrub' operationally useless for large tables.
>
> I think that given not having user defined options for these
> compaction types is clearly an oversight, and that the alternative of
> deleting the large 1TB+ sstable and then repairing is a cure worse
> than the disease, this should be added to 3.11.x and 4.0.x. I am +1
> here.
>
> > For *garbagecollect* it is often operationally easy to identify which
> > tables are likely to be full of bloa- and operationally useful to do this
> > task in small increments. The existing order that garbagecollect
> processes
> > SSTables prevents it from being useful in any incremental fashion -- if
> you
> > stop it and later restart, it will first process the SSTables you just
> > garbage collected.
> >
> > The third ticket adds an option for* nodetool garbagecollect*,
> > *--oldest-fraction* that can select a fraction of the oldest table data
> in
> > bytes, and garbagecollect only the SSTables that 'cover' that percentage
> of
> > data. Operationally, this lends itself to easy automation -- for example
> > running this once a week on 10% of a table's data would imply that there
> is
> > no data on disk that has been overwritten within the last 10 weeks. This
> > caps data bloat in ways neither LCS nor STCS can currently achieve
> without
> > regular major compactions or full-pass garbagecollect.
>
> This is a less obvious thing to be added, and I personally lack the
> operational experience to comment on how much relief this would
> provide firsthand, so I'll leave that to others. But it does make
> sense to me and since it isn't heavily modifying anything my
> inclination is that this could be an acceptable addition as well.
>
> Kind Regards,
> Brandon
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@cassandra.apache.org
> For additional commands, e-mail: dev-help@cassandra.apache.org
>
>
Re: [DISCUSS] CASSANDRA-16767, CASSANDRA-16768, and CASSANDRA-16769
for 3.11.x
Posted by Brandon Williams <dr...@gmail.com>.
On Tue, Jun 29, 2021 at 5:49 PM Scott Carey <sc...@apache.org> wrote:
>
> I'd like to discuss the inclusion of the above tickets for a 3.11.x
> release. These are not a pure 'bug fix' so I'll need a waiver to get them
> into 3.11.x (and implicitly, 4.0.x).
>
> The first two are straightforward oversights: neither *nodetool
> garbagecollect *nor *nodetool scrub* currently accept a *--user-defined*
> parameter list of SSTables in the same way that *nodetool compact* does.
>
> This is an operational problem for large tables.
>
> I often need to scrub just one file that is corrupted for some reason, and
> not scrub an entire 1TB+ of data for a table on a node. This renders
> 'nodetool scrub' operationally useless for large tables.
I think that given not having user defined options for these
compaction types is clearly an oversight, and that the alternative of
deleting the large 1TB+ sstable and then repairing is a cure worse
than the disease, this should be added to 3.11.x and 4.0.x. I am +1
here.
> For *garbagecollect* it is often operationally easy to identify which
> tables are likely to be full of bloa- and operationally useful to do this
> task in small increments. The existing order that garbagecollect processes
> SSTables prevents it from being useful in any incremental fashion -- if you
> stop it and later restart, it will first process the SSTables you just
> garbage collected.
>
> The third ticket adds an option for* nodetool garbagecollect*,
> *--oldest-fraction* that can select a fraction of the oldest table data in
> bytes, and garbagecollect only the SSTables that 'cover' that percentage of
> data. Operationally, this lends itself to easy automation -- for example
> running this once a week on 10% of a table's data would imply that there is
> no data on disk that has been overwritten within the last 10 weeks. This
> caps data bloat in ways neither LCS nor STCS can currently achieve without
> regular major compactions or full-pass garbagecollect.
This is a less obvious thing to be added, and I personally lack the
operational experience to comment on how much relief this would
provide firsthand, so I'll leave that to others. But it does make
sense to me and since it isn't heavily modifying anything my
inclination is that this could be an acceptable addition as well.
Kind Regards,
Brandon
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@cassandra.apache.org
For additional commands, e-mail: dev-help@cassandra.apache.org