You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@cassandra.apache.org by Andrés de la Peña <ad...@apache.org> on 2021/11/01 16:31:32 UTC
[DISCUSS] CEP-3: Guardrails
Hi everyone,
I'd like to start a discussion about Guardrails proposal:
https://cwiki.apache.org/confluence/display/CASSANDRA/%28DRAFT%29+-+CEP-3%3A+Guardrails
Guardrails are an easy way to enforce system-wide soft and hard limits to
prevent anti-patterns of bad usage and in the long run make it not possible
to severely degrade the performance of a node/cluster through user actions
such as having too many secondary indexes, too large partitions, almost
full disks, etc.
Thanks,
Re: [DISCUSS] CEP-3: Guardrails
Posted by David Capwell <dc...@apple.com.INVALID>.
Under "Migrating existing cassandra.yaml warn/fail thresholds”, I recently added a few things which are basically guardrails, so should be included in this set; they are configured by track_warnings (coordinator_read_size, local_read_size, and row_index_size). With track_warnings I setup the plumbing to have read queries trigger warnings (or abort the query) to the client exists (under "Event logging" you mention "and also to the client connection when applicable”) and isn’t limited to the coordinator participating in the query (previous limitation for tombstone warnings). One thing I found which was problematic for track_warnings was that altering clients is annoying as java and python both ignore the error message we send (see https://github.com/datastax/java-driver/blob/3.11.0/driver-core/src/main/java/com/datastax/driver/core/Responses.java#L73-L131). We log client warnings (if enabled) but ignore any detailed error message received from the server; it would be good to talk about client integrations and how users are informed of issues in more detail.
> On Nov 1, 2021, at 9:46 AM, Jeff Jirsa <jj...@gmail.com> wrote:
>
> Without bike-shedding too much, guardrails would be great, building them
> into a more general purpose framework that limits various dangerous things
> would be fantastic. The CEP says that the guardrails should be distinct
> from the capability restrictions (
> https://issues.apache.org/jira/browse/CASSANDRA-8303 ), but I don't see why
> that needs to be the case. A system-level guardrail and a personal-level
> guardrail are both restrictions, they just have different scopes, so
> implement the restriction framework first, and allow the scopes to be
> expanded as needed?
>
> Naming wise, I don't know that I'd actually surface these as "guardrails",
> but more as general "limits", and having them only configured via yaml
> seems like a bad outcome
>
>
>
> https://issues.apache.org/jira/browse/CASSANDRA-8303
>
>
>
> On Mon, Nov 1, 2021 at 9:31 AM Andrés de la Peña <ad...@apache.org>
> wrote:
>
>> Hi everyone,
>>
>> I'd like to start a discussion about Guardrails proposal:
>>
>> https://cwiki.apache.org/confluence/display/CASSANDRA/%28DRAFT%29+-+CEP-3%3A+Guardrails
>>
>> Guardrails are an easy way to enforce system-wide soft and hard limits to
>> prevent anti-patterns of bad usage and in the long run make it not possible
>> to severely degrade the performance of a node/cluster through user actions
>> such as having too many secondary indexes, too large partitions, almost
>> full disks, etc.
>>
>> Thanks,
>>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@cassandra.apache.org
For additional commands, e-mail: dev-help@cassandra.apache.org
Re: [DISCUSS] CEP-3: Guardrails
Posted by David Capwell <dc...@apple.com.INVALID>.
If anyone wants to bite off making https://github.com/apache/cassandra/blob/ab920c30310a8c095ba76b363142b8e74cbf0a0a/src/java/org/apache/cassandra/db/virtual/SettingsTable.java <https://github.com/apache/cassandra/blob/ab920c30310a8c095ba76b363142b8e74cbf0a0a/src/java/org/apache/cassandra/db/virtual/SettingsTable.java> support mutability then we get vtable support. I am cool with JMX and/or vtable, to me its just more important to allow dynamic setting of these configs.
> On Nov 1, 2021, at 10:36 AM, benedict@apache.org wrote:
>
>> having them only configured via yaml seems like a bad outcome
>
> +1
>
> I would like to see us move towards configuration being driven through virtual tables where possible, so that the whole cluster can be managed from a single interface. Not sure if this is the right place to bite this off, but perhaps?
>
> From: Jeff Jirsa <jj...@gmail.com>
> Date: Monday, 1 November 2021 at 16:47
> To: Cassandra DEV <de...@cassandra.apache.org>
> Subject: Re: [DISCUSS] CEP-3: Guardrails
> Without bike-shedding too much, guardrails would be great, building them
> into a more general purpose framework that limits various dangerous things
> would be fantastic. The CEP says that the guardrails should be distinct
> from the capability restrictions (
> https://issues.apache.org/jira/browse/CASSANDRA-8303 ), but I don't see why
> that needs to be the case. A system-level guardrail and a personal-level
> guardrail are both restrictions, they just have different scopes, so
> implement the restriction framework first, and allow the scopes to be
> expanded as needed?
>
> Naming wise, I don't know that I'd actually surface these as "guardrails",
> but more as general "limits", and having them only configured via yaml
> seems like a bad outcome
>
>
>
> https://issues.apache.org/jira/browse/CASSANDRA-8303
>
>
>
> On Mon, Nov 1, 2021 at 9:31 AM Andrés de la Peña <ad...@apache.org>
> wrote:
>
>> Hi everyone,
>>
>> I'd like to start a discussion about Guardrails proposal:
>>
>> https://cwiki.apache.org/confluence/display/CASSANDRA/%28DRAFT%29+-+CEP-3%3A+Guardrails
>>
>> Guardrails are an easy way to enforce system-wide soft and hard limits to
>> prevent anti-patterns of bad usage and in the long run make it not possible
>> to severely degrade the performance of a node/cluster through user actions
>> such as having too many secondary indexes, too large partitions, almost
>> full disks, etc.
>>
>> Thanks,
>>
Re: [DISCUSS] CEP-3: Guardrails
Posted by "benedict@apache.org" <be...@apache.org>.
> having them only configured via yaml seems like a bad outcome
+1
I would like to see us move towards configuration being driven through virtual tables where possible, so that the whole cluster can be managed from a single interface. Not sure if this is the right place to bite this off, but perhaps?
From: Jeff Jirsa <jj...@gmail.com>
Date: Monday, 1 November 2021 at 16:47
To: Cassandra DEV <de...@cassandra.apache.org>
Subject: Re: [DISCUSS] CEP-3: Guardrails
Without bike-shedding too much, guardrails would be great, building them
into a more general purpose framework that limits various dangerous things
would be fantastic. The CEP says that the guardrails should be distinct
from the capability restrictions (
https://issues.apache.org/jira/browse/CASSANDRA-8303 ), but I don't see why
that needs to be the case. A system-level guardrail and a personal-level
guardrail are both restrictions, they just have different scopes, so
implement the restriction framework first, and allow the scopes to be
expanded as needed?
Naming wise, I don't know that I'd actually surface these as "guardrails",
but more as general "limits", and having them only configured via yaml
seems like a bad outcome
https://issues.apache.org/jira/browse/CASSANDRA-8303
On Mon, Nov 1, 2021 at 9:31 AM Andrés de la Peña <ad...@apache.org>
wrote:
> Hi everyone,
>
> I'd like to start a discussion about Guardrails proposal:
>
> https://cwiki.apache.org/confluence/display/CASSANDRA/%28DRAFT%29+-+CEP-3%3A+Guardrails
>
> Guardrails are an easy way to enforce system-wide soft and hard limits to
> prevent anti-patterns of bad usage and in the long run make it not possible
> to severely degrade the performance of a node/cluster through user actions
> such as having too many secondary indexes, too large partitions, almost
> full disks, etc.
>
> Thanks,
>
Re: [DISCUSS] CEP-3: Guardrails
Posted by Jeff Jirsa <jj...@gmail.com>.
Without bike-shedding too much, guardrails would be great, building them
into a more general purpose framework that limits various dangerous things
would be fantastic. The CEP says that the guardrails should be distinct
from the capability restrictions (
https://issues.apache.org/jira/browse/CASSANDRA-8303 ), but I don't see why
that needs to be the case. A system-level guardrail and a personal-level
guardrail are both restrictions, they just have different scopes, so
implement the restriction framework first, and allow the scopes to be
expanded as needed?
Naming wise, I don't know that I'd actually surface these as "guardrails",
but more as general "limits", and having them only configured via yaml
seems like a bad outcome
https://issues.apache.org/jira/browse/CASSANDRA-8303
On Mon, Nov 1, 2021 at 9:31 AM Andrés de la Peña <ad...@apache.org>
wrote:
> Hi everyone,
>
> I'd like to start a discussion about Guardrails proposal:
>
> https://cwiki.apache.org/confluence/display/CASSANDRA/%28DRAFT%29+-+CEP-3%3A+Guardrails
>
> Guardrails are an easy way to enforce system-wide soft and hard limits to
> prevent anti-patterns of bad usage and in the long run make it not possible
> to severely degrade the performance of a node/cluster through user actions
> such as having too many secondary indexes, too large partitions, almost
> full disks, etc.
>
> Thanks,
>