You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@cassandra.apache.org by Brandon Williams <dr...@gmail.com> on 2021/07/02 20:23:07 UTC
Re: [DISCUSS] CASSANDRA-16767, CASSANDRA-16768, and CASSANDRA-16769
for 3.11.x
On Tue, Jun 29, 2021 at 5:49 PM Scott Carey <sc...@apache.org> wrote:
>
> I'd like to discuss the inclusion of the above tickets for a 3.11.x
> release. These are not a pure 'bug fix' so I'll need a waiver to get them
> into 3.11.x (and implicitly, 4.0.x).
>
> The first two are straightforward oversights: neither *nodetool
> garbagecollect *nor *nodetool scrub* currently accept a *--user-defined*
> parameter list of SSTables in the same way that *nodetool compact* does.
>
> This is an operational problem for large tables.
>
> I often need to scrub just one file that is corrupted for some reason, and
> not scrub an entire 1TB+ of data for a table on a node. This renders
> 'nodetool scrub' operationally useless for large tables.
I think that given not having user defined options for these
compaction types is clearly an oversight, and that the alternative of
deleting the large 1TB+ sstable and then repairing is a cure worse
than the disease, this should be added to 3.11.x and 4.0.x. I am +1
here.
> For *garbagecollect* it is often operationally easy to identify which
> tables are likely to be full of bloa- and operationally useful to do this
> task in small increments. The existing order that garbagecollect processes
> SSTables prevents it from being useful in any incremental fashion -- if you
> stop it and later restart, it will first process the SSTables you just
> garbage collected.
>
> The third ticket adds an option for* nodetool garbagecollect*,
> *--oldest-fraction* that can select a fraction of the oldest table data in
> bytes, and garbagecollect only the SSTables that 'cover' that percentage of
> data. Operationally, this lends itself to easy automation -- for example
> running this once a week on 10% of a table's data would imply that there is
> no data on disk that has been overwritten within the last 10 weeks. This
> caps data bloat in ways neither LCS nor STCS can currently achieve without
> regular major compactions or full-pass garbagecollect.
This is a less obvious thing to be added, and I personally lack the
operational experience to comment on how much relief this would
provide firsthand, so I'll leave that to others. But it does make
sense to me and since it isn't heavily modifying anything my
inclination is that this could be an acceptable addition as well.
Kind Regards,
Brandon
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@cassandra.apache.org
For additional commands, e-mail: dev-help@cassandra.apache.org
Re: [DISCUSS] CASSANDRA-16767, CASSANDRA-16768, and CASSANDRA-16769
for 3.11.x
Posted by "benedict@apache.org" <be...@apache.org>.
Hi Scott,
I wonder if it’s possible that too few people who saw your email consider themselves sufficiently involved in this part of the codebase. People tend to keep quiet about stuff they don’t participate in deeply, which is why I haven’t responded – and I wonder if this might explain the tumbleweed. I wonder how we might generally track if areas of the codebase are adequately covered by active contributors.
To answer your question, I don’t personally believe it is problematic to add additional features to command line tools in a patch version – they’re not scary systems where new features introduce much risk of high impact bugs. Others have stricter interpretations of the rules, but if they haven’t spoken up yet I’d say you’re clear to post some patches – but you might want to first make sure there’s somebody able and willing to review them.
From: Scott Carey <sc...@apache.org>
Date: Monday, 9 August 2021 at 20:12
To: dev@cassandra.apache.org <de...@cassandra.apache.org>
Subject: Re: [DISCUSS] CASSANDRA-16767, CASSANDRA-16768, and CASSANDRA-16769 for 3.11.x
Thank you Brandon, for answering my questions on slack, and providing early
feedback on these ideas more than a month before I created the patches and
replying here.
Does anyone else have any comments or opinions? Can a decision be reached
one way or another? It is my understanding that we'll need more than one
+1 to move forward here.
I understand that the 4.0 release was a busy time, and that many probably
saw this, thought about replying, but got too busy and never did.
However, in light of the recent discussions around attracting new
contributors, I would like to highlight that being left in limbo with no
resolution is worse than being told "no", especially for new contributors.
On Fri, Jul 2, 2021 at 1:23 PM Brandon Williams <dr...@gmail.com> wrote:
> On Tue, Jun 29, 2021 at 5:49 PM Scott Carey <sc...@apache.org> wrote:
> >
> > I'd like to discuss the inclusion of the above tickets for a 3.11.x
> > release. These are not a pure 'bug fix' so I'll need a waiver to get
> them
> > into 3.11.x (and implicitly, 4.0.x).
> >
> > The first two are straightforward oversights: neither *nodetool
> > garbagecollect *nor *nodetool scrub* currently accept a *--user-defined*
> > parameter list of SSTables in the same way that *nodetool compact* does.
> >
> > This is an operational problem for large tables.
> >
> > I often need to scrub just one file that is corrupted for some reason,
> and
> > not scrub an entire 1TB+ of data for a table on a node. This renders
> > 'nodetool scrub' operationally useless for large tables.
>
> I think that given not having user defined options for these
> compaction types is clearly an oversight, and that the alternative of
> deleting the large 1TB+ sstable and then repairing is a cure worse
> than the disease, this should be added to 3.11.x and 4.0.x. I am +1
> here.
>
> > For *garbagecollect* it is often operationally easy to identify which
> > tables are likely to be full of bloa- and operationally useful to do this
> > task in small increments. The existing order that garbagecollect
> processes
> > SSTables prevents it from being useful in any incremental fashion -- if
> you
> > stop it and later restart, it will first process the SSTables you just
> > garbage collected.
> >
> > The third ticket adds an option for* nodetool garbagecollect*,
> > *--oldest-fraction* that can select a fraction of the oldest table data
> in
> > bytes, and garbagecollect only the SSTables that 'cover' that percentage
> of
> > data. Operationally, this lends itself to easy automation -- for example
> > running this once a week on 10% of a table's data would imply that there
> is
> > no data on disk that has been overwritten within the last 10 weeks. This
> > caps data bloat in ways neither LCS nor STCS can currently achieve
> without
> > regular major compactions or full-pass garbagecollect.
>
> This is a less obvious thing to be added, and I personally lack the
> operational experience to comment on how much relief this would
> provide firsthand, so I'll leave that to others. But it does make
> sense to me and since it isn't heavily modifying anything my
> inclination is that this could be an acceptable addition as well.
>
> Kind Regards,
> Brandon
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@cassandra.apache.org
> For additional commands, e-mail: dev-help@cassandra.apache.org
>
>
Re: [DISCUSS] CASSANDRA-16767, CASSANDRA-16768, and CASSANDRA-16769
for 3.11.x
Posted by Brandon Williams <dr...@gmail.com>.
On Mon, Aug 9, 2021 at 2:05 PM Scott Carey <sc...@apache.org> wrote:
> Does anyone else have any comments or opinions? Can a decision be reached
> one way or another? It is my understanding that we'll need more than one
> +1 to move forward here.
Lazy consensus says 72 hours
(https://community.apache.org/committers/lazyConsensus.html) with
considerations for availability, but I think it has been more than
long enough at this point. If no one else objects after this
prodding, I think you should move forward.
Kind Regards,
Brandon
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@cassandra.apache.org
For additional commands, e-mail: dev-help@cassandra.apache.org
Re: [DISCUSS] CASSANDRA-16767, CASSANDRA-16768, and CASSANDRA-16769
for 3.11.x
Posted by Scott Carey <sc...@apache.org>.
Thank you Brandon, for answering my questions on slack, and providing early
feedback on these ideas more than a month before I created the patches and
replying here.
Does anyone else have any comments or opinions? Can a decision be reached
one way or another? It is my understanding that we'll need more than one
+1 to move forward here.
I understand that the 4.0 release was a busy time, and that many probably
saw this, thought about replying, but got too busy and never did.
However, in light of the recent discussions around attracting new
contributors, I would like to highlight that being left in limbo with no
resolution is worse than being told "no", especially for new contributors.
On Fri, Jul 2, 2021 at 1:23 PM Brandon Williams <dr...@gmail.com> wrote:
> On Tue, Jun 29, 2021 at 5:49 PM Scott Carey <sc...@apache.org> wrote:
> >
> > I'd like to discuss the inclusion of the above tickets for a 3.11.x
> > release. These are not a pure 'bug fix' so I'll need a waiver to get
> them
> > into 3.11.x (and implicitly, 4.0.x).
> >
> > The first two are straightforward oversights: neither *nodetool
> > garbagecollect *nor *nodetool scrub* currently accept a *--user-defined*
> > parameter list of SSTables in the same way that *nodetool compact* does.
> >
> > This is an operational problem for large tables.
> >
> > I often need to scrub just one file that is corrupted for some reason,
> and
> > not scrub an entire 1TB+ of data for a table on a node. This renders
> > 'nodetool scrub' operationally useless for large tables.
>
> I think that given not having user defined options for these
> compaction types is clearly an oversight, and that the alternative of
> deleting the large 1TB+ sstable and then repairing is a cure worse
> than the disease, this should be added to 3.11.x and 4.0.x. I am +1
> here.
>
> > For *garbagecollect* it is often operationally easy to identify which
> > tables are likely to be full of bloa- and operationally useful to do this
> > task in small increments. The existing order that garbagecollect
> processes
> > SSTables prevents it from being useful in any incremental fashion -- if
> you
> > stop it and later restart, it will first process the SSTables you just
> > garbage collected.
> >
> > The third ticket adds an option for* nodetool garbagecollect*,
> > *--oldest-fraction* that can select a fraction of the oldest table data
> in
> > bytes, and garbagecollect only the SSTables that 'cover' that percentage
> of
> > data. Operationally, this lends itself to easy automation -- for example
> > running this once a week on 10% of a table's data would imply that there
> is
> > no data on disk that has been overwritten within the last 10 weeks. This
> > caps data bloat in ways neither LCS nor STCS can currently achieve
> without
> > regular major compactions or full-pass garbagecollect.
>
> This is a less obvious thing to be added, and I personally lack the
> operational experience to comment on how much relief this would
> provide firsthand, so I'll leave that to others. But it does make
> sense to me and since it isn't heavily modifying anything my
> inclination is that this could be an acceptable addition as well.
>
> Kind Regards,
> Brandon
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@cassandra.apache.org
> For additional commands, e-mail: dev-help@cassandra.apache.org
>
>