Re: [DISCUSS] CEP-13: Denylisting partitions

[Joshua McKenzie <jm...@apache.org>]

I'm +1 on where it currently stands after the revisions. Consider resolving
out comment threads on the design doc that are closed so we can see if
there's any outstanding discussions from a high level?

~Josh

On Mon, Aug 30, 2021 at 1:14 AM Sumanth Pasupuleti <
sumanth.pasupuleti.is@gmail.com> wrote:

> +1. Made changes to the design document linked against the CEP to reflect
> this feedback. Specifically, the following sections have been updated
> * Operations to blacklist
> * Blacklist information store
>
> Thanks,
> Sumanth
>
>
> On Fri, Aug 27, 2021 at 7:57 AM Joshua McKenzie <jm...@apache.org>
> wrote:
>
> > I can see the case for all three:
> > * Deny both reads and writes to a partition (wide, heavily tombstones,
> too
> > many stables, etc) causing disruption to a replica set; don't want
> further
> > growth nor reads until operator intervention
> > * Deny reads but allow writing to rectify problems on a partition
> > (intervention window; see above)
> > * Deny writes to a partition but allow reads (prevent partitions growing
> > unbounded, or potentially evolving into a future feature creating a
> ceiling
> > on partition sizes that kicks in and demands application intervention to
> > reduce partition size at a guardrail limit)
> >
> > So yeah, at least to me at face value it seems like it'd be worth it not
> > only to allow denylisting both reads and writes, but to be able to choose
> > from the set of reads|writes|both on a per-partition basis.
> >
> > ~Josh
> >
> >
> > On Thu, Aug 26, 2021 at 2:16 PM Sumanth Pasupuleti <
> > sumanth.pasupuleti.is@gmail.com> wrote:
> >
> > > Thank you, Josh for the elaborate explanation of a potential scenario
> > where
> > > denylisting writes would make sense.
> > > I, 100% agree that could benefit in a situation where we would want to
> > deny
> > > writes to a partition that we do not have much control on (which is
> true
> > in
> > > most situations) and such behavior can eventually lead to
> unavailability
> > of
> > > other partitions too, as you indicate.
> > >
> > > Do you think it makes sense to make it configurable per partition
> though?
> > > As in, maybe by default, we would want to deny both reads and writes
> to a
> > > partition, but for certain partitions, we may still want to allow
> writes
> > > just so we can issue a delete against that partition as an example.
> > > Ofcourse this would make the feature and the interface more heavy, and
> we
> > > need to think through if its worth it. I personally feel it could be
> > worth
> > > it, especially if we agree on the default behavior that makes the
> > interface
> > > simple in most cases. Thoughts?
> > >
> > > And yes, so good to see CEP process reaping benefits in multiple ways -
> > > especially around collaboration and documentation.
> > >
> > >
> > > On Thu, Aug 26, 2021 at 8:31 AM Joshua McKenzie <jm...@apache.org>
> > > wrote:
> > >
> > > > The design doc and CEP currently pass on blocklisting / denylisting
> > > writes
> > > > at this time. In the proposed new patch it states:
> > > > "Note: We do not want to blacklist writes since it is the reads that
> > > > primarily impact the performance when reading a bad partition, and we
> > may
> > > > want writes to be allowed to “fix” a bad partition. We could revisit
> > this
> > > > in the future"
> > > >
> > > > In situations where you have an air gap between database ops and
> > > > application access (ops <> application teams, or more autonomous
> > > > application access patterns, self-service, etc), you can easily get
> > into
> > > a
> > > > situation where you have either a pathological client hammering
> writes
> > > to a
> > > > specific partition causing impact to other clients or in the worst
> > case,
> > > > the replica set, or unbounded partition growth that again leads to
> > > > performance degradation or replica set unavailability. The tradeoff
> > there
> > > > becomes "do we interrupt the application's ability to write to this
> > > > partition now, or do we instead defer and risk losing access to *all*
> > > > partitions on this replica set and still interrupt their access
> > > eventually
> > > > anyway?"
> > > >
> > > > Given this, I strongly advocate for support of denylisting both reads
> > > *and*
> > > > writes on these grounds; operators need another tool in their toolbox
> > to
> > > > deal with situations where specific partition writing has wider
> > negative
> > > > impacts on the replicas.
> > > >
> > > > Acknowledging of course that there was extensive discussion on this
> > back
> > > in
> > > > 2018, and that would have been a *great* time to engage in the
> > > discussion.
> > > > =/ Good thing we have this new CEP process! :)
> > > >
> > > > Curious what you think about this perspective Sumanth.
> > > >
> > > > ~Josh
> > > >
> > > >
> > > > On Tue, Aug 17, 2021 at 2:04 PM Joshua McKenzie <
> jmckenzie@apache.org>
> > > > wrote:
> > > >
> > > > > Certainly. I'll take on distilling a high level view of the feature
> > > from
> > > > > what Jon and I are working on to bring to this discussion.
> > > > >
> > > > >
> > > > > On Tue, Aug 17, 2021 at 1:40 PM Sumanth Pasupuleti <
> > > > > sumanth.pasupuleti.is@gmail.com> wrote:
> > > > >
> > > > >> +1 to collaborating on the patch, Josh. My 2 cents would be to
> > > continue
> > > > to
> > > > >> pursue this CEP in the community through Discuss and Vote phases
> and
> > > > then
> > > > >> invest further on the patch (based on the Vote phase outcome), so
> we
> > > can
> > > > >> reflect any additional feedback we may gather from the community
> > > through
> > > > >> this CEP.
> > > > >>
> > > > >> Thanks,
> > > > >> Sumanth
> > > > >>
> > > > >> On Tue, Aug 17, 2021 at 10:13 AM Joshua McKenzie <
> > > jmckenzie@apache.org>
> > > > >> wrote:
> > > > >>
> > > > >> > Sumanth,
> > > > >> >
> > > > >> > Jon Meredith and I are recently working on an OSS patch of one
> of
> > > > those
> > > > >> "ad
> > > > >> > hoc" implementations of this feature that's been running at
> scale
> > > for
> > > > >> > awhile like Jirsa mentioned; sorry for not catching the
> discussion
> > > on
> > > > >> > https://issues.apache.org/jira/browse/CASSANDRA-12106 and
> > engaging
> > > > >> > earlier!
> > > > >> >
> > > > >> > I believe I can have a tidied up patch on 4.0 of this by early
> > next
> > > > >> week -
> > > > >> > then we can look at the two implementations and take best of
> both.
> > > > What
> > > > >> do
> > > > >> > you think?
> > > > >> >
> > > > >> > ~Josh
> > > > >> >
> > > > >> > On Mon, Aug 16, 2021 at 2:14 PM Sumanth Pasupuleti <
> > > > >> > sumanth.pasupuleti.is@gmail.com> wrote:
> > > > >> >
> > > > >> > > Starting a discussion thread for CEP-13
> > > > >> > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-13%3A+Denylisting+partitions
> > > > >> > >
> > > > >> > > This CEP proposes adding a new feature to Cassandra to be able
> > to
> > > > >> > denylist
> > > > >> > > partitions.
> > > > >> > >
> > > > >> > > Looking forward to any feedback/ thoughts.
> > > > >> > >
> > > > >> > > Thanks,
> > > > >> > > Sumanth
> > > > >> > >
> > > > >> >
> > > > >>
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] CEP-13: Denylisting partitions

[Sumanth Pasupuleti <su...@gmail.com>]

Resolved comment discussions in the design document that are closed.
If there is no further feedback, I will start a voting thread tomorrow.

Thanks,
Sumanth

On Thu, Sep 2, 2021 at 6:54 AM Joshua McKenzie <jm...@apache.org> wrote:

> I'm +1 on where it currently stands after the revisions. Consider resolving
> out comment threads on the design doc that are closed so we can see if
> there's any outstanding discussions from a high level?
>
> ~Josh
>
> On Mon, Aug 30, 2021 at 1:14 AM Sumanth Pasupuleti <
> sumanth.pasupuleti.is@gmail.com> wrote:
>
> > +1. Made changes to the design document linked against the CEP to reflect
> > this feedback. Specifically, the following sections have been updated
> > * Operations to blacklist
> > * Blacklist information store
> >
> > Thanks,
> > Sumanth
> >
> >
> > On Fri, Aug 27, 2021 at 7:57 AM Joshua McKenzie <jm...@apache.org>
> > wrote:
> >
> > > I can see the case for all three:
> > > * Deny both reads and writes to a partition (wide, heavily tombstones,
> > too
> > > many stables, etc) causing disruption to a replica set; don't want
> > further
> > > growth nor reads until operator intervention
> > > * Deny reads but allow writing to rectify problems on a partition
> > > (intervention window; see above)
> > > * Deny writes to a partition but allow reads (prevent partitions
> growing
> > > unbounded, or potentially evolving into a future feature creating a
> > ceiling
> > > on partition sizes that kicks in and demands application intervention
> to
> > > reduce partition size at a guardrail limit)
> > >
> > > So yeah, at least to me at face value it seems like it'd be worth it
> not
> > > only to allow denylisting both reads and writes, but to be able to
> choose
> > > from the set of reads|writes|both on a per-partition basis.
> > >
> > > ~Josh
> > >
> > >
> > > On Thu, Aug 26, 2021 at 2:16 PM Sumanth Pasupuleti <
> > > sumanth.pasupuleti.is@gmail.com> wrote:
> > >
> > > > Thank you, Josh for the elaborate explanation of a potential scenario
> > > where
> > > > denylisting writes would make sense.
> > > > I, 100% agree that could benefit in a situation where we would want
> to
> > > deny
> > > > writes to a partition that we do not have much control on (which is
> > true
> > > in
> > > > most situations) and such behavior can eventually lead to
> > unavailability
> > > of
> > > > other partitions too, as you indicate.
> > > >
> > > > Do you think it makes sense to make it configurable per partition
> > though?
> > > > As in, maybe by default, we would want to deny both reads and writes
> > to a
> > > > partition, but for certain partitions, we may still want to allow
> > writes
> > > > just so we can issue a delete against that partition as an example.
> > > > Ofcourse this would make the feature and the interface more heavy,
> and
> > we
> > > > need to think through if its worth it. I personally feel it could be
> > > worth
> > > > it, especially if we agree on the default behavior that makes the
> > > interface
> > > > simple in most cases. Thoughts?
> > > >
> > > > And yes, so good to see CEP process reaping benefits in multiple
> ways -
> > > > especially around collaboration and documentation.
> > > >
> > > >
> > > > On Thu, Aug 26, 2021 at 8:31 AM Joshua McKenzie <
> jmckenzie@apache.org>
> > > > wrote:
> > > >
> > > > > The design doc and CEP currently pass on blocklisting / denylisting
> > > > writes
> > > > > at this time. In the proposed new patch it states:
> > > > > "Note: We do not want to blacklist writes since it is the reads
> that
> > > > > primarily impact the performance when reading a bad partition, and
> we
> > > may
> > > > > want writes to be allowed to “fix” a bad partition. We could
> revisit
> > > this
> > > > > in the future"
> > > > >
> > > > > In situations where you have an air gap between database ops and
> > > > > application access (ops <> application teams, or more autonomous
> > > > > application access patterns, self-service, etc), you can easily get
> > > into
> > > > a
> > > > > situation where you have either a pathological client hammering
> > writes
> > > > to a
> > > > > specific partition causing impact to other clients or in the worst
> > > case,
> > > > > the replica set, or unbounded partition growth that again leads to
> > > > > performance degradation or replica set unavailability. The tradeoff
> > > there
> > > > > becomes "do we interrupt the application's ability to write to this
> > > > > partition now, or do we instead defer and risk losing access to
> *all*
> > > > > partitions on this replica set and still interrupt their access
> > > > eventually
> > > > > anyway?"
> > > > >
> > > > > Given this, I strongly advocate for support of denylisting both
> reads
> > > > *and*
> > > > > writes on these grounds; operators need another tool in their
> toolbox
> > > to
> > > > > deal with situations where specific partition writing has wider
> > > negative
> > > > > impacts on the replicas.
> > > > >
> > > > > Acknowledging of course that there was extensive discussion on this
> > > back
> > > > in
> > > > > 2018, and that would have been a *great* time to engage in the
> > > > discussion.
> > > > > =/ Good thing we have this new CEP process! :)
> > > > >
> > > > > Curious what you think about this perspective Sumanth.
> > > > >
> > > > > ~Josh
> > > > >
> > > > >
> > > > > On Tue, Aug 17, 2021 at 2:04 PM Joshua McKenzie <
> > jmckenzie@apache.org>
> > > > > wrote:
> > > > >
> > > > > > Certainly. I'll take on distilling a high level view of the
> feature
> > > > from
> > > > > > what Jon and I are working on to bring to this discussion.
> > > > > >
> > > > > >
> > > > > > On Tue, Aug 17, 2021 at 1:40 PM Sumanth Pasupuleti <
> > > > > > sumanth.pasupuleti.is@gmail.com> wrote:
> > > > > >
> > > > > >> +1 to collaborating on the patch, Josh. My 2 cents would be to
> > > > continue
> > > > > to
> > > > > >> pursue this CEP in the community through Discuss and Vote phases
> > and
> > > > > then
> > > > > >> invest further on the patch (based on the Vote phase outcome),
> so
> > we
> > > > can
> > > > > >> reflect any additional feedback we may gather from the community
> > > > through
> > > > > >> this CEP.
> > > > > >>
> > > > > >> Thanks,
> > > > > >> Sumanth
> > > > > >>
> > > > > >> On Tue, Aug 17, 2021 at 10:13 AM Joshua McKenzie <
> > > > jmckenzie@apache.org>
> > > > > >> wrote:
> > > > > >>
> > > > > >> > Sumanth,
> > > > > >> >
> > > > > >> > Jon Meredith and I are recently working on an OSS patch of one
> > of
> > > > > those
> > > > > >> "ad
> > > > > >> > hoc" implementations of this feature that's been running at
> > scale
> > > > for
> > > > > >> > awhile like Jirsa mentioned; sorry for not catching the
> > discussion
> > > > on
> > > > > >> > https://issues.apache.org/jira/browse/CASSANDRA-12106 and
> > > engaging
> > > > > >> > earlier!
> > > > > >> >
> > > > > >> > I believe I can have a tidied up patch on 4.0 of this by early
> > > next
> > > > > >> week -
> > > > > >> > then we can look at the two implementations and take best of
> > both.
> > > > > What
> > > > > >> do
> > > > > >> > you think?
> > > > > >> >
> > > > > >> > ~Josh
> > > > > >> >
> > > > > >> > On Mon, Aug 16, 2021 at 2:14 PM Sumanth Pasupuleti <
> > > > > >> > sumanth.pasupuleti.is@gmail.com> wrote:
> > > > > >> >
> > > > > >> > > Starting a discussion thread for CEP-13
> > > > > >> > >
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-13%3A+Denylisting+partitions
> > > > > >> > >
> > > > > >> > > This CEP proposes adding a new feature to Cassandra to be
> able
> > > to
> > > > > >> > denylist
> > > > > >> > > partitions.
> > > > > >> > >
> > > > > >> > > Looking forward to any feedback/ thoughts.
> > > > > >> > >
> > > > > >> > > Thanks,
> > > > > >> > > Sumanth
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
>