You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@cassandra.apache.org by Sumanth Pasupuleti <su...@gmail.com> on 2021/08/16 18:14:23 UTC
[DISCUSS] CEP-13: Denylisting partitions
Starting a discussion thread for CEP-13
https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-13%3A+Denylisting+partitions
This CEP proposes adding a new feature to Cassandra to be able to denylist
partitions.
Looking forward to any feedback/ thoughts.
Thanks,
Sumanth
Re: [DISCUSS] CEP-13: Denylisting partitions
Posted by Sumanth Pasupuleti <su...@gmail.com>.
Resolved comment discussions in the design document that are closed.
If there is no further feedback, I will start a voting thread tomorrow.
Thanks,
Sumanth
On Thu, Sep 2, 2021 at 6:54 AM Joshua McKenzie <jm...@apache.org> wrote:
> I'm +1 on where it currently stands after the revisions. Consider resolving
> out comment threads on the design doc that are closed so we can see if
> there's any outstanding discussions from a high level?
>
> ~Josh
>
> On Mon, Aug 30, 2021 at 1:14 AM Sumanth Pasupuleti <
> sumanth.pasupuleti.is@gmail.com> wrote:
>
> > +1. Made changes to the design document linked against the CEP to reflect
> > this feedback. Specifically, the following sections have been updated
> > * Operations to blacklist
> > * Blacklist information store
> >
> > Thanks,
> > Sumanth
> >
> >
> > On Fri, Aug 27, 2021 at 7:57 AM Joshua McKenzie <jm...@apache.org>
> > wrote:
> >
> > > I can see the case for all three:
> > > * Deny both reads and writes to a partition (wide, heavily tombstones,
> > too
> > > many stables, etc) causing disruption to a replica set; don't want
> > further
> > > growth nor reads until operator intervention
> > > * Deny reads but allow writing to rectify problems on a partition
> > > (intervention window; see above)
> > > * Deny writes to a partition but allow reads (prevent partitions
> growing
> > > unbounded, or potentially evolving into a future feature creating a
> > ceiling
> > > on partition sizes that kicks in and demands application intervention
> to
> > > reduce partition size at a guardrail limit)
> > >
> > > So yeah, at least to me at face value it seems like it'd be worth it
> not
> > > only to allow denylisting both reads and writes, but to be able to
> choose
> > > from the set of reads|writes|both on a per-partition basis.
> > >
> > > ~Josh
> > >
> > >
> > > On Thu, Aug 26, 2021 at 2:16 PM Sumanth Pasupuleti <
> > > sumanth.pasupuleti.is@gmail.com> wrote:
> > >
> > > > Thank you, Josh for the elaborate explanation of a potential scenario
> > > where
> > > > denylisting writes would make sense.
> > > > I, 100% agree that could benefit in a situation where we would want
> to
> > > deny
> > > > writes to a partition that we do not have much control on (which is
> > true
> > > in
> > > > most situations) and such behavior can eventually lead to
> > unavailability
> > > of
> > > > other partitions too, as you indicate.
> > > >
> > > > Do you think it makes sense to make it configurable per partition
> > though?
> > > > As in, maybe by default, we would want to deny both reads and writes
> > to a
> > > > partition, but for certain partitions, we may still want to allow
> > writes
> > > > just so we can issue a delete against that partition as an example.
> > > > Ofcourse this would make the feature and the interface more heavy,
> and
> > we
> > > > need to think through if its worth it. I personally feel it could be
> > > worth
> > > > it, especially if we agree on the default behavior that makes the
> > > interface
> > > > simple in most cases. Thoughts?
> > > >
> > > > And yes, so good to see CEP process reaping benefits in multiple
> ways -
> > > > especially around collaboration and documentation.
> > > >
> > > >
> > > > On Thu, Aug 26, 2021 at 8:31 AM Joshua McKenzie <
> jmckenzie@apache.org>
> > > > wrote:
> > > >
> > > > > The design doc and CEP currently pass on blocklisting / denylisting
> > > > writes
> > > > > at this time. In the proposed new patch it states:
> > > > > "Note: We do not want to blacklist writes since it is the reads
> that
> > > > > primarily impact the performance when reading a bad partition, and
> we
> > > may
> > > > > want writes to be allowed to “fix” a bad partition. We could
> revisit
> > > this
> > > > > in the future"
> > > > >
> > > > > In situations where you have an air gap between database ops and
> > > > > application access (ops <> application teams, or more autonomous
> > > > > application access patterns, self-service, etc), you can easily get
> > > into
> > > > a
> > > > > situation where you have either a pathological client hammering
> > writes
> > > > to a
> > > > > specific partition causing impact to other clients or in the worst
> > > case,
> > > > > the replica set, or unbounded partition growth that again leads to
> > > > > performance degradation or replica set unavailability. The tradeoff
> > > there
> > > > > becomes "do we interrupt the application's ability to write to this
> > > > > partition now, or do we instead defer and risk losing access to
> *all*
> > > > > partitions on this replica set and still interrupt their access
> > > > eventually
> > > > > anyway?"
> > > > >
> > > > > Given this, I strongly advocate for support of denylisting both
> reads
> > > > *and*
> > > > > writes on these grounds; operators need another tool in their
> toolbox
> > > to
> > > > > deal with situations where specific partition writing has wider
> > > negative
> > > > > impacts on the replicas.
> > > > >
> > > > > Acknowledging of course that there was extensive discussion on this
> > > back
> > > > in
> > > > > 2018, and that would have been a *great* time to engage in the
> > > > discussion.
> > > > > =/ Good thing we have this new CEP process! :)
> > > > >
> > > > > Curious what you think about this perspective Sumanth.
> > > > >
> > > > > ~Josh
> > > > >
> > > > >
> > > > > On Tue, Aug 17, 2021 at 2:04 PM Joshua McKenzie <
> > jmckenzie@apache.org>
> > > > > wrote:
> > > > >
> > > > > > Certainly. I'll take on distilling a high level view of the
> feature
> > > > from
> > > > > > what Jon and I are working on to bring to this discussion.
> > > > > >
> > > > > >
> > > > > > On Tue, Aug 17, 2021 at 1:40 PM Sumanth Pasupuleti <
> > > > > > sumanth.pasupuleti.is@gmail.com> wrote:
> > > > > >
> > > > > >> +1 to collaborating on the patch, Josh. My 2 cents would be to
> > > > continue
> > > > > to
> > > > > >> pursue this CEP in the community through Discuss and Vote phases
> > and
> > > > > then
> > > > > >> invest further on the patch (based on the Vote phase outcome),
> so
> > we
> > > > can
> > > > > >> reflect any additional feedback we may gather from the community
> > > > through
> > > > > >> this CEP.
> > > > > >>
> > > > > >> Thanks,
> > > > > >> Sumanth
> > > > > >>
> > > > > >> On Tue, Aug 17, 2021 at 10:13 AM Joshua McKenzie <
> > > > jmckenzie@apache.org>
> > > > > >> wrote:
> > > > > >>
> > > > > >> > Sumanth,
> > > > > >> >
> > > > > >> > Jon Meredith and I are recently working on an OSS patch of one
> > of
> > > > > those
> > > > > >> "ad
> > > > > >> > hoc" implementations of this feature that's been running at
> > scale
> > > > for
> > > > > >> > awhile like Jirsa mentioned; sorry for not catching the
> > discussion
> > > > on
> > > > > >> > https://issues.apache.org/jira/browse/CASSANDRA-12106 and
> > > engaging
> > > > > >> > earlier!
> > > > > >> >
> > > > > >> > I believe I can have a tidied up patch on 4.0 of this by early
> > > next
> > > > > >> week -
> > > > > >> > then we can look at the two implementations and take best of
> > both.
> > > > > What
> > > > > >> do
> > > > > >> > you think?
> > > > > >> >
> > > > > >> > ~Josh
> > > > > >> >
> > > > > >> > On Mon, Aug 16, 2021 at 2:14 PM Sumanth Pasupuleti <
> > > > > >> > sumanth.pasupuleti.is@gmail.com> wrote:
> > > > > >> >
> > > > > >> > > Starting a discussion thread for CEP-13
> > > > > >> > >
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-13%3A+Denylisting+partitions
> > > > > >> > >
> > > > > >> > > This CEP proposes adding a new feature to Cassandra to be
> able
> > > to
> > > > > >> > denylist
> > > > > >> > > partitions.
> > > > > >> > >
> > > > > >> > > Looking forward to any feedback/ thoughts.
> > > > > >> > >
> > > > > >> > > Thanks,
> > > > > >> > > Sumanth
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: [DISCUSS] CEP-13: Denylisting partitions
Posted by Joshua McKenzie <jm...@apache.org>.
I'm +1 on where it currently stands after the revisions. Consider resolving
out comment threads on the design doc that are closed so we can see if
there's any outstanding discussions from a high level?
~Josh
On Mon, Aug 30, 2021 at 1:14 AM Sumanth Pasupuleti <
sumanth.pasupuleti.is@gmail.com> wrote:
> +1. Made changes to the design document linked against the CEP to reflect
> this feedback. Specifically, the following sections have been updated
> * Operations to blacklist
> * Blacklist information store
>
> Thanks,
> Sumanth
>
>
> On Fri, Aug 27, 2021 at 7:57 AM Joshua McKenzie <jm...@apache.org>
> wrote:
>
> > I can see the case for all three:
> > * Deny both reads and writes to a partition (wide, heavily tombstones,
> too
> > many stables, etc) causing disruption to a replica set; don't want
> further
> > growth nor reads until operator intervention
> > * Deny reads but allow writing to rectify problems on a partition
> > (intervention window; see above)
> > * Deny writes to a partition but allow reads (prevent partitions growing
> > unbounded, or potentially evolving into a future feature creating a
> ceiling
> > on partition sizes that kicks in and demands application intervention to
> > reduce partition size at a guardrail limit)
> >
> > So yeah, at least to me at face value it seems like it'd be worth it not
> > only to allow denylisting both reads and writes, but to be able to choose
> > from the set of reads|writes|both on a per-partition basis.
> >
> > ~Josh
> >
> >
> > On Thu, Aug 26, 2021 at 2:16 PM Sumanth Pasupuleti <
> > sumanth.pasupuleti.is@gmail.com> wrote:
> >
> > > Thank you, Josh for the elaborate explanation of a potential scenario
> > where
> > > denylisting writes would make sense.
> > > I, 100% agree that could benefit in a situation where we would want to
> > deny
> > > writes to a partition that we do not have much control on (which is
> true
> > in
> > > most situations) and such behavior can eventually lead to
> unavailability
> > of
> > > other partitions too, as you indicate.
> > >
> > > Do you think it makes sense to make it configurable per partition
> though?
> > > As in, maybe by default, we would want to deny both reads and writes
> to a
> > > partition, but for certain partitions, we may still want to allow
> writes
> > > just so we can issue a delete against that partition as an example.
> > > Ofcourse this would make the feature and the interface more heavy, and
> we
> > > need to think through if its worth it. I personally feel it could be
> > worth
> > > it, especially if we agree on the default behavior that makes the
> > interface
> > > simple in most cases. Thoughts?
> > >
> > > And yes, so good to see CEP process reaping benefits in multiple ways -
> > > especially around collaboration and documentation.
> > >
> > >
> > > On Thu, Aug 26, 2021 at 8:31 AM Joshua McKenzie <jm...@apache.org>
> > > wrote:
> > >
> > > > The design doc and CEP currently pass on blocklisting / denylisting
> > > writes
> > > > at this time. In the proposed new patch it states:
> > > > "Note: We do not want to blacklist writes since it is the reads that
> > > > primarily impact the performance when reading a bad partition, and we
> > may
> > > > want writes to be allowed to “fix” a bad partition. We could revisit
> > this
> > > > in the future"
> > > >
> > > > In situations where you have an air gap between database ops and
> > > > application access (ops <> application teams, or more autonomous
> > > > application access patterns, self-service, etc), you can easily get
> > into
> > > a
> > > > situation where you have either a pathological client hammering
> writes
> > > to a
> > > > specific partition causing impact to other clients or in the worst
> > case,
> > > > the replica set, or unbounded partition growth that again leads to
> > > > performance degradation or replica set unavailability. The tradeoff
> > there
> > > > becomes "do we interrupt the application's ability to write to this
> > > > partition now, or do we instead defer and risk losing access to *all*
> > > > partitions on this replica set and still interrupt their access
> > > eventually
> > > > anyway?"
> > > >
> > > > Given this, I strongly advocate for support of denylisting both reads
> > > *and*
> > > > writes on these grounds; operators need another tool in their toolbox
> > to
> > > > deal with situations where specific partition writing has wider
> > negative
> > > > impacts on the replicas.
> > > >
> > > > Acknowledging of course that there was extensive discussion on this
> > back
> > > in
> > > > 2018, and that would have been a *great* time to engage in the
> > > discussion.
> > > > =/ Good thing we have this new CEP process! :)
> > > >
> > > > Curious what you think about this perspective Sumanth.
> > > >
> > > > ~Josh
> > > >
> > > >
> > > > On Tue, Aug 17, 2021 at 2:04 PM Joshua McKenzie <
> jmckenzie@apache.org>
> > > > wrote:
> > > >
> > > > > Certainly. I'll take on distilling a high level view of the feature
> > > from
> > > > > what Jon and I are working on to bring to this discussion.
> > > > >
> > > > >
> > > > > On Tue, Aug 17, 2021 at 1:40 PM Sumanth Pasupuleti <
> > > > > sumanth.pasupuleti.is@gmail.com> wrote:
> > > > >
> > > > >> +1 to collaborating on the patch, Josh. My 2 cents would be to
> > > continue
> > > > to
> > > > >> pursue this CEP in the community through Discuss and Vote phases
> and
> > > > then
> > > > >> invest further on the patch (based on the Vote phase outcome), so
> we
> > > can
> > > > >> reflect any additional feedback we may gather from the community
> > > through
> > > > >> this CEP.
> > > > >>
> > > > >> Thanks,
> > > > >> Sumanth
> > > > >>
> > > > >> On Tue, Aug 17, 2021 at 10:13 AM Joshua McKenzie <
> > > jmckenzie@apache.org>
> > > > >> wrote:
> > > > >>
> > > > >> > Sumanth,
> > > > >> >
> > > > >> > Jon Meredith and I are recently working on an OSS patch of one
> of
> > > > those
> > > > >> "ad
> > > > >> > hoc" implementations of this feature that's been running at
> scale
> > > for
> > > > >> > awhile like Jirsa mentioned; sorry for not catching the
> discussion
> > > on
> > > > >> > https://issues.apache.org/jira/browse/CASSANDRA-12106 and
> > engaging
> > > > >> > earlier!
> > > > >> >
> > > > >> > I believe I can have a tidied up patch on 4.0 of this by early
> > next
> > > > >> week -
> > > > >> > then we can look at the two implementations and take best of
> both.
> > > > What
> > > > >> do
> > > > >> > you think?
> > > > >> >
> > > > >> > ~Josh
> > > > >> >
> > > > >> > On Mon, Aug 16, 2021 at 2:14 PM Sumanth Pasupuleti <
> > > > >> > sumanth.pasupuleti.is@gmail.com> wrote:
> > > > >> >
> > > > >> > > Starting a discussion thread for CEP-13
> > > > >> > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-13%3A+Denylisting+partitions
> > > > >> > >
> > > > >> > > This CEP proposes adding a new feature to Cassandra to be able
> > to
> > > > >> > denylist
> > > > >> > > partitions.
> > > > >> > >
> > > > >> > > Looking forward to any feedback/ thoughts.
> > > > >> > >
> > > > >> > > Thanks,
> > > > >> > > Sumanth
> > > > >> > >
> > > > >> >
> > > > >>
> > > > >
> > > >
> > >
> >
>
Re: [DISCUSS] CEP-13: Denylisting partitions
Posted by Sumanth Pasupuleti <su...@gmail.com>.
+1. Made changes to the design document linked against the CEP to reflect
this feedback. Specifically, the following sections have been updated
* Operations to blacklist
* Blacklist information store
Thanks,
Sumanth
On Fri, Aug 27, 2021 at 7:57 AM Joshua McKenzie <jm...@apache.org>
wrote:
> I can see the case for all three:
> * Deny both reads and writes to a partition (wide, heavily tombstones, too
> many stables, etc) causing disruption to a replica set; don't want further
> growth nor reads until operator intervention
> * Deny reads but allow writing to rectify problems on a partition
> (intervention window; see above)
> * Deny writes to a partition but allow reads (prevent partitions growing
> unbounded, or potentially evolving into a future feature creating a ceiling
> on partition sizes that kicks in and demands application intervention to
> reduce partition size at a guardrail limit)
>
> So yeah, at least to me at face value it seems like it'd be worth it not
> only to allow denylisting both reads and writes, but to be able to choose
> from the set of reads|writes|both on a per-partition basis.
>
> ~Josh
>
>
> On Thu, Aug 26, 2021 at 2:16 PM Sumanth Pasupuleti <
> sumanth.pasupuleti.is@gmail.com> wrote:
>
> > Thank you, Josh for the elaborate explanation of a potential scenario
> where
> > denylisting writes would make sense.
> > I, 100% agree that could benefit in a situation where we would want to
> deny
> > writes to a partition that we do not have much control on (which is true
> in
> > most situations) and such behavior can eventually lead to unavailability
> of
> > other partitions too, as you indicate.
> >
> > Do you think it makes sense to make it configurable per partition though?
> > As in, maybe by default, we would want to deny both reads and writes to a
> > partition, but for certain partitions, we may still want to allow writes
> > just so we can issue a delete against that partition as an example.
> > Ofcourse this would make the feature and the interface more heavy, and we
> > need to think through if its worth it. I personally feel it could be
> worth
> > it, especially if we agree on the default behavior that makes the
> interface
> > simple in most cases. Thoughts?
> >
> > And yes, so good to see CEP process reaping benefits in multiple ways -
> > especially around collaboration and documentation.
> >
> >
> > On Thu, Aug 26, 2021 at 8:31 AM Joshua McKenzie <jm...@apache.org>
> > wrote:
> >
> > > The design doc and CEP currently pass on blocklisting / denylisting
> > writes
> > > at this time. In the proposed new patch it states:
> > > "Note: We do not want to blacklist writes since it is the reads that
> > > primarily impact the performance when reading a bad partition, and we
> may
> > > want writes to be allowed to “fix” a bad partition. We could revisit
> this
> > > in the future"
> > >
> > > In situations where you have an air gap between database ops and
> > > application access (ops <> application teams, or more autonomous
> > > application access patterns, self-service, etc), you can easily get
> into
> > a
> > > situation where you have either a pathological client hammering writes
> > to a
> > > specific partition causing impact to other clients or in the worst
> case,
> > > the replica set, or unbounded partition growth that again leads to
> > > performance degradation or replica set unavailability. The tradeoff
> there
> > > becomes "do we interrupt the application's ability to write to this
> > > partition now, or do we instead defer and risk losing access to *all*
> > > partitions on this replica set and still interrupt their access
> > eventually
> > > anyway?"
> > >
> > > Given this, I strongly advocate for support of denylisting both reads
> > *and*
> > > writes on these grounds; operators need another tool in their toolbox
> to
> > > deal with situations where specific partition writing has wider
> negative
> > > impacts on the replicas.
> > >
> > > Acknowledging of course that there was extensive discussion on this
> back
> > in
> > > 2018, and that would have been a *great* time to engage in the
> > discussion.
> > > =/ Good thing we have this new CEP process! :)
> > >
> > > Curious what you think about this perspective Sumanth.
> > >
> > > ~Josh
> > >
> > >
> > > On Tue, Aug 17, 2021 at 2:04 PM Joshua McKenzie <jm...@apache.org>
> > > wrote:
> > >
> > > > Certainly. I'll take on distilling a high level view of the feature
> > from
> > > > what Jon and I are working on to bring to this discussion.
> > > >
> > > >
> > > > On Tue, Aug 17, 2021 at 1:40 PM Sumanth Pasupuleti <
> > > > sumanth.pasupuleti.is@gmail.com> wrote:
> > > >
> > > >> +1 to collaborating on the patch, Josh. My 2 cents would be to
> > continue
> > > to
> > > >> pursue this CEP in the community through Discuss and Vote phases and
> > > then
> > > >> invest further on the patch (based on the Vote phase outcome), so we
> > can
> > > >> reflect any additional feedback we may gather from the community
> > through
> > > >> this CEP.
> > > >>
> > > >> Thanks,
> > > >> Sumanth
> > > >>
> > > >> On Tue, Aug 17, 2021 at 10:13 AM Joshua McKenzie <
> > jmckenzie@apache.org>
> > > >> wrote:
> > > >>
> > > >> > Sumanth,
> > > >> >
> > > >> > Jon Meredith and I are recently working on an OSS patch of one of
> > > those
> > > >> "ad
> > > >> > hoc" implementations of this feature that's been running at scale
> > for
> > > >> > awhile like Jirsa mentioned; sorry for not catching the discussion
> > on
> > > >> > https://issues.apache.org/jira/browse/CASSANDRA-12106 and
> engaging
> > > >> > earlier!
> > > >> >
> > > >> > I believe I can have a tidied up patch on 4.0 of this by early
> next
> > > >> week -
> > > >> > then we can look at the two implementations and take best of both.
> > > What
> > > >> do
> > > >> > you think?
> > > >> >
> > > >> > ~Josh
> > > >> >
> > > >> > On Mon, Aug 16, 2021 at 2:14 PM Sumanth Pasupuleti <
> > > >> > sumanth.pasupuleti.is@gmail.com> wrote:
> > > >> >
> > > >> > > Starting a discussion thread for CEP-13
> > > >> > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
> https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-13%3A+Denylisting+partitions
> > > >> > >
> > > >> > > This CEP proposes adding a new feature to Cassandra to be able
> to
> > > >> > denylist
> > > >> > > partitions.
> > > >> > >
> > > >> > > Looking forward to any feedback/ thoughts.
> > > >> > >
> > > >> > > Thanks,
> > > >> > > Sumanth
> > > >> > >
> > > >> >
> > > >>
> > > >
> > >
> >
>
Re: [DISCUSS] CEP-13: Denylisting partitions
Posted by Joshua McKenzie <jm...@apache.org>.
I can see the case for all three:
* Deny both reads and writes to a partition (wide, heavily tombstones, too
many stables, etc) causing disruption to a replica set; don't want further
growth nor reads until operator intervention
* Deny reads but allow writing to rectify problems on a partition
(intervention window; see above)
* Deny writes to a partition but allow reads (prevent partitions growing
unbounded, or potentially evolving into a future feature creating a ceiling
on partition sizes that kicks in and demands application intervention to
reduce partition size at a guardrail limit)
So yeah, at least to me at face value it seems like it'd be worth it not
only to allow denylisting both reads and writes, but to be able to choose
from the set of reads|writes|both on a per-partition basis.
~Josh
On Thu, Aug 26, 2021 at 2:16 PM Sumanth Pasupuleti <
sumanth.pasupuleti.is@gmail.com> wrote:
> Thank you, Josh for the elaborate explanation of a potential scenario where
> denylisting writes would make sense.
> I, 100% agree that could benefit in a situation where we would want to deny
> writes to a partition that we do not have much control on (which is true in
> most situations) and such behavior can eventually lead to unavailability of
> other partitions too, as you indicate.
>
> Do you think it makes sense to make it configurable per partition though?
> As in, maybe by default, we would want to deny both reads and writes to a
> partition, but for certain partitions, we may still want to allow writes
> just so we can issue a delete against that partition as an example.
> Ofcourse this would make the feature and the interface more heavy, and we
> need to think through if its worth it. I personally feel it could be worth
> it, especially if we agree on the default behavior that makes the interface
> simple in most cases. Thoughts?
>
> And yes, so good to see CEP process reaping benefits in multiple ways -
> especially around collaboration and documentation.
>
>
> On Thu, Aug 26, 2021 at 8:31 AM Joshua McKenzie <jm...@apache.org>
> wrote:
>
> > The design doc and CEP currently pass on blocklisting / denylisting
> writes
> > at this time. In the proposed new patch it states:
> > "Note: We do not want to blacklist writes since it is the reads that
> > primarily impact the performance when reading a bad partition, and we may
> > want writes to be allowed to “fix” a bad partition. We could revisit this
> > in the future"
> >
> > In situations where you have an air gap between database ops and
> > application access (ops <> application teams, or more autonomous
> > application access patterns, self-service, etc), you can easily get into
> a
> > situation where you have either a pathological client hammering writes
> to a
> > specific partition causing impact to other clients or in the worst case,
> > the replica set, or unbounded partition growth that again leads to
> > performance degradation or replica set unavailability. The tradeoff there
> > becomes "do we interrupt the application's ability to write to this
> > partition now, or do we instead defer and risk losing access to *all*
> > partitions on this replica set and still interrupt their access
> eventually
> > anyway?"
> >
> > Given this, I strongly advocate for support of denylisting both reads
> *and*
> > writes on these grounds; operators need another tool in their toolbox to
> > deal with situations where specific partition writing has wider negative
> > impacts on the replicas.
> >
> > Acknowledging of course that there was extensive discussion on this back
> in
> > 2018, and that would have been a *great* time to engage in the
> discussion.
> > =/ Good thing we have this new CEP process! :)
> >
> > Curious what you think about this perspective Sumanth.
> >
> > ~Josh
> >
> >
> > On Tue, Aug 17, 2021 at 2:04 PM Joshua McKenzie <jm...@apache.org>
> > wrote:
> >
> > > Certainly. I'll take on distilling a high level view of the feature
> from
> > > what Jon and I are working on to bring to this discussion.
> > >
> > >
> > > On Tue, Aug 17, 2021 at 1:40 PM Sumanth Pasupuleti <
> > > sumanth.pasupuleti.is@gmail.com> wrote:
> > >
> > >> +1 to collaborating on the patch, Josh. My 2 cents would be to
> continue
> > to
> > >> pursue this CEP in the community through Discuss and Vote phases and
> > then
> > >> invest further on the patch (based on the Vote phase outcome), so we
> can
> > >> reflect any additional feedback we may gather from the community
> through
> > >> this CEP.
> > >>
> > >> Thanks,
> > >> Sumanth
> > >>
> > >> On Tue, Aug 17, 2021 at 10:13 AM Joshua McKenzie <
> jmckenzie@apache.org>
> > >> wrote:
> > >>
> > >> > Sumanth,
> > >> >
> > >> > Jon Meredith and I are recently working on an OSS patch of one of
> > those
> > >> "ad
> > >> > hoc" implementations of this feature that's been running at scale
> for
> > >> > awhile like Jirsa mentioned; sorry for not catching the discussion
> on
> > >> > https://issues.apache.org/jira/browse/CASSANDRA-12106 and engaging
> > >> > earlier!
> > >> >
> > >> > I believe I can have a tidied up patch on 4.0 of this by early next
> > >> week -
> > >> > then we can look at the two implementations and take best of both.
> > What
> > >> do
> > >> > you think?
> > >> >
> > >> > ~Josh
> > >> >
> > >> > On Mon, Aug 16, 2021 at 2:14 PM Sumanth Pasupuleti <
> > >> > sumanth.pasupuleti.is@gmail.com> wrote:
> > >> >
> > >> > > Starting a discussion thread for CEP-13
> > >> > >
> > >> > >
> > >> >
> > >>
> >
> https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-13%3A+Denylisting+partitions
> > >> > >
> > >> > > This CEP proposes adding a new feature to Cassandra to be able to
> > >> > denylist
> > >> > > partitions.
> > >> > >
> > >> > > Looking forward to any feedback/ thoughts.
> > >> > >
> > >> > > Thanks,
> > >> > > Sumanth
> > >> > >
> > >> >
> > >>
> > >
> >
>
Re: [DISCUSS] CEP-13: Denylisting partitions
Posted by Sumanth Pasupuleti <su...@gmail.com>.
Thank you, Josh for the elaborate explanation of a potential scenario where
denylisting writes would make sense.
I, 100% agree that could benefit in a situation where we would want to deny
writes to a partition that we do not have much control on (which is true in
most situations) and such behavior can eventually lead to unavailability of
other partitions too, as you indicate.
Do you think it makes sense to make it configurable per partition though?
As in, maybe by default, we would want to deny both reads and writes to a
partition, but for certain partitions, we may still want to allow writes
just so we can issue a delete against that partition as an example.
Ofcourse this would make the feature and the interface more heavy, and we
need to think through if its worth it. I personally feel it could be worth
it, especially if we agree on the default behavior that makes the interface
simple in most cases. Thoughts?
And yes, so good to see CEP process reaping benefits in multiple ways -
especially around collaboration and documentation.
On Thu, Aug 26, 2021 at 8:31 AM Joshua McKenzie <jm...@apache.org>
wrote:
> The design doc and CEP currently pass on blocklisting / denylisting writes
> at this time. In the proposed new patch it states:
> "Note: We do not want to blacklist writes since it is the reads that
> primarily impact the performance when reading a bad partition, and we may
> want writes to be allowed to “fix” a bad partition. We could revisit this
> in the future"
>
> In situations where you have an air gap between database ops and
> application access (ops <> application teams, or more autonomous
> application access patterns, self-service, etc), you can easily get into a
> situation where you have either a pathological client hammering writes to a
> specific partition causing impact to other clients or in the worst case,
> the replica set, or unbounded partition growth that again leads to
> performance degradation or replica set unavailability. The tradeoff there
> becomes "do we interrupt the application's ability to write to this
> partition now, or do we instead defer and risk losing access to *all*
> partitions on this replica set and still interrupt their access eventually
> anyway?"
>
> Given this, I strongly advocate for support of denylisting both reads *and*
> writes on these grounds; operators need another tool in their toolbox to
> deal with situations where specific partition writing has wider negative
> impacts on the replicas.
>
> Acknowledging of course that there was extensive discussion on this back in
> 2018, and that would have been a *great* time to engage in the discussion.
> =/ Good thing we have this new CEP process! :)
>
> Curious what you think about this perspective Sumanth.
>
> ~Josh
>
>
> On Tue, Aug 17, 2021 at 2:04 PM Joshua McKenzie <jm...@apache.org>
> wrote:
>
> > Certainly. I'll take on distilling a high level view of the feature from
> > what Jon and I are working on to bring to this discussion.
> >
> >
> > On Tue, Aug 17, 2021 at 1:40 PM Sumanth Pasupuleti <
> > sumanth.pasupuleti.is@gmail.com> wrote:
> >
> >> +1 to collaborating on the patch, Josh. My 2 cents would be to continue
> to
> >> pursue this CEP in the community through Discuss and Vote phases and
> then
> >> invest further on the patch (based on the Vote phase outcome), so we can
> >> reflect any additional feedback we may gather from the community through
> >> this CEP.
> >>
> >> Thanks,
> >> Sumanth
> >>
> >> On Tue, Aug 17, 2021 at 10:13 AM Joshua McKenzie <jm...@apache.org>
> >> wrote:
> >>
> >> > Sumanth,
> >> >
> >> > Jon Meredith and I are recently working on an OSS patch of one of
> those
> >> "ad
> >> > hoc" implementations of this feature that's been running at scale for
> >> > awhile like Jirsa mentioned; sorry for not catching the discussion on
> >> > https://issues.apache.org/jira/browse/CASSANDRA-12106 and engaging
> >> > earlier!
> >> >
> >> > I believe I can have a tidied up patch on 4.0 of this by early next
> >> week -
> >> > then we can look at the two implementations and take best of both.
> What
> >> do
> >> > you think?
> >> >
> >> > ~Josh
> >> >
> >> > On Mon, Aug 16, 2021 at 2:14 PM Sumanth Pasupuleti <
> >> > sumanth.pasupuleti.is@gmail.com> wrote:
> >> >
> >> > > Starting a discussion thread for CEP-13
> >> > >
> >> > >
> >> >
> >>
> https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-13%3A+Denylisting+partitions
> >> > >
> >> > > This CEP proposes adding a new feature to Cassandra to be able to
> >> > denylist
> >> > > partitions.
> >> > >
> >> > > Looking forward to any feedback/ thoughts.
> >> > >
> >> > > Thanks,
> >> > > Sumanth
> >> > >
> >> >
> >>
> >
>
Re: [DISCUSS] CEP-13: Denylisting partitions
Posted by Joshua McKenzie <jm...@apache.org>.
The design doc and CEP currently pass on blocklisting / denylisting writes
at this time. In the proposed new patch it states:
"Note: We do not want to blacklist writes since it is the reads that
primarily impact the performance when reading a bad partition, and we may
want writes to be allowed to “fix” a bad partition. We could revisit this
in the future"
In situations where you have an air gap between database ops and
application access (ops <> application teams, or more autonomous
application access patterns, self-service, etc), you can easily get into a
situation where you have either a pathological client hammering writes to a
specific partition causing impact to other clients or in the worst case,
the replica set, or unbounded partition growth that again leads to
performance degradation or replica set unavailability. The tradeoff there
becomes "do we interrupt the application's ability to write to this
partition now, or do we instead defer and risk losing access to *all*
partitions on this replica set and still interrupt their access eventually
anyway?"
Given this, I strongly advocate for support of denylisting both reads *and*
writes on these grounds; operators need another tool in their toolbox to
deal with situations where specific partition writing has wider negative
impacts on the replicas.
Acknowledging of course that there was extensive discussion on this back in
2018, and that would have been a *great* time to engage in the discussion.
=/ Good thing we have this new CEP process! :)
Curious what you think about this perspective Sumanth.
~Josh
On Tue, Aug 17, 2021 at 2:04 PM Joshua McKenzie <jm...@apache.org>
wrote:
> Certainly. I'll take on distilling a high level view of the feature from
> what Jon and I are working on to bring to this discussion.
>
>
> On Tue, Aug 17, 2021 at 1:40 PM Sumanth Pasupuleti <
> sumanth.pasupuleti.is@gmail.com> wrote:
>
>> +1 to collaborating on the patch, Josh. My 2 cents would be to continue to
>> pursue this CEP in the community through Discuss and Vote phases and then
>> invest further on the patch (based on the Vote phase outcome), so we can
>> reflect any additional feedback we may gather from the community through
>> this CEP.
>>
>> Thanks,
>> Sumanth
>>
>> On Tue, Aug 17, 2021 at 10:13 AM Joshua McKenzie <jm...@apache.org>
>> wrote:
>>
>> > Sumanth,
>> >
>> > Jon Meredith and I are recently working on an OSS patch of one of those
>> "ad
>> > hoc" implementations of this feature that's been running at scale for
>> > awhile like Jirsa mentioned; sorry for not catching the discussion on
>> > https://issues.apache.org/jira/browse/CASSANDRA-12106 and engaging
>> > earlier!
>> >
>> > I believe I can have a tidied up patch on 4.0 of this by early next
>> week -
>> > then we can look at the two implementations and take best of both. What
>> do
>> > you think?
>> >
>> > ~Josh
>> >
>> > On Mon, Aug 16, 2021 at 2:14 PM Sumanth Pasupuleti <
>> > sumanth.pasupuleti.is@gmail.com> wrote:
>> >
>> > > Starting a discussion thread for CEP-13
>> > >
>> > >
>> >
>> https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-13%3A+Denylisting+partitions
>> > >
>> > > This CEP proposes adding a new feature to Cassandra to be able to
>> > denylist
>> > > partitions.
>> > >
>> > > Looking forward to any feedback/ thoughts.
>> > >
>> > > Thanks,
>> > > Sumanth
>> > >
>> >
>>
>
Re: [DISCUSS] CEP-13: Denylisting partitions
Posted by Joshua McKenzie <jm...@apache.org>.
Certainly. I'll take on distilling a high level view of the feature from
what Jon and I are working on to bring to this discussion.
On Tue, Aug 17, 2021 at 1:40 PM Sumanth Pasupuleti <
sumanth.pasupuleti.is@gmail.com> wrote:
> +1 to collaborating on the patch, Josh. My 2 cents would be to continue to
> pursue this CEP in the community through Discuss and Vote phases and then
> invest further on the patch (based on the Vote phase outcome), so we can
> reflect any additional feedback we may gather from the community through
> this CEP.
>
> Thanks,
> Sumanth
>
> On Tue, Aug 17, 2021 at 10:13 AM Joshua McKenzie <jm...@apache.org>
> wrote:
>
> > Sumanth,
> >
> > Jon Meredith and I are recently working on an OSS patch of one of those
> "ad
> > hoc" implementations of this feature that's been running at scale for
> > awhile like Jirsa mentioned; sorry for not catching the discussion on
> > https://issues.apache.org/jira/browse/CASSANDRA-12106 and engaging
> > earlier!
> >
> > I believe I can have a tidied up patch on 4.0 of this by early next week
> -
> > then we can look at the two implementations and take best of both. What
> do
> > you think?
> >
> > ~Josh
> >
> > On Mon, Aug 16, 2021 at 2:14 PM Sumanth Pasupuleti <
> > sumanth.pasupuleti.is@gmail.com> wrote:
> >
> > > Starting a discussion thread for CEP-13
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-13%3A+Denylisting+partitions
> > >
> > > This CEP proposes adding a new feature to Cassandra to be able to
> > denylist
> > > partitions.
> > >
> > > Looking forward to any feedback/ thoughts.
> > >
> > > Thanks,
> > > Sumanth
> > >
> >
>
Re: [DISCUSS] CEP-13: Denylisting partitions
Posted by Sumanth Pasupuleti <su...@gmail.com>.
+1 to collaborating on the patch, Josh. My 2 cents would be to continue to
pursue this CEP in the community through Discuss and Vote phases and then
invest further on the patch (based on the Vote phase outcome), so we can
reflect any additional feedback we may gather from the community through
this CEP.
Thanks,
Sumanth
On Tue, Aug 17, 2021 at 10:13 AM Joshua McKenzie <jm...@apache.org>
wrote:
> Sumanth,
>
> Jon Meredith and I are recently working on an OSS patch of one of those "ad
> hoc" implementations of this feature that's been running at scale for
> awhile like Jirsa mentioned; sorry for not catching the discussion on
> https://issues.apache.org/jira/browse/CASSANDRA-12106 and engaging
> earlier!
>
> I believe I can have a tidied up patch on 4.0 of this by early next week -
> then we can look at the two implementations and take best of both. What do
> you think?
>
> ~Josh
>
> On Mon, Aug 16, 2021 at 2:14 PM Sumanth Pasupuleti <
> sumanth.pasupuleti.is@gmail.com> wrote:
>
> > Starting a discussion thread for CEP-13
> >
> >
> https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-13%3A+Denylisting+partitions
> >
> > This CEP proposes adding a new feature to Cassandra to be able to
> denylist
> > partitions.
> >
> > Looking forward to any feedback/ thoughts.
> >
> > Thanks,
> > Sumanth
> >
>
Re: [DISCUSS] CEP-13: Denylisting partitions
Posted by Joshua McKenzie <jm...@apache.org>.
Sumanth,
Jon Meredith and I are recently working on an OSS patch of one of those "ad
hoc" implementations of this feature that's been running at scale for
awhile like Jirsa mentioned; sorry for not catching the discussion on
https://issues.apache.org/jira/browse/CASSANDRA-12106 and engaging earlier!
I believe I can have a tidied up patch on 4.0 of this by early next week -
then we can look at the two implementations and take best of both. What do
you think?
~Josh
On Mon, Aug 16, 2021 at 2:14 PM Sumanth Pasupuleti <
sumanth.pasupuleti.is@gmail.com> wrote:
> Starting a discussion thread for CEP-13
>
> https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-13%3A+Denylisting+partitions
>
> This CEP proposes adding a new feature to Cassandra to be able to denylist
> partitions.
>
> Looking forward to any feedback/ thoughts.
>
> Thanks,
> Sumanth
>