Re: [EXTERNAL] Re: [Discuss] Generic Purpose Rate Limiter in Cassandra

[German Eichberger via dev <de...@cassandra.apache.org>]

Jaydeep,

I concur with Stefan that extensibility of this  should be a design goal:

  *   It should be easy to add additional metrics (e.g. write queue depth) and decision logic
  *   There should be a way to interact with other systems to signal a resource need  which then could kick off things like scaling

Super interested in this and we have been thinking about siimilar things internally 😉

Thanks,
German
________________________________
From: Jaydeep Chovatia <ch...@gmail.com>
Sent: Tuesday, January 16, 2024 1:16 PM
To: dev@cassandra.apache.org <de...@cassandra.apache.org>
Subject: [EXTERNAL] Re: [Discuss] Generic Purpose Rate Limiter in Cassandra

You don't often get email from chovatia.jaydeep@gmail.com. Learn why this is important<https://aka.ms/LearnAboutSenderIdentification>
Hi Stefan,

Please find my response below:
1) Currently, I am keeping the signals as interface, so one can override with a different implementation, but a point noted that even the interface APIs could be also made dynamic so one can define APIs and its implementation, if they wish to override.
2) I've not looked into that yet, but I will look into it and see if it can be easily integrated into the Guardrails framework.
3) On the server side, when the framework detects that a node is overloaded, then it will throw OverloadedException back to the client. Because if the node while busy continues to serve additional requests, then it will slow down other peer nodes due to dependencies on meeting the QUORUM, etc. In this, we are at least preventing server nodes from melting down, and giving the control to the client via OverloadedException. Now, it will be up to the client policy, if client wishes to retry immediately on a different server node then eventually that server node might be impacted, but if client wishes to do exponential back off or throw exception back to the application then that server node will not be impacted.


Jaydeep

On Tue, Jan 16, 2024 at 10:03 AM Štefan Miklošovič <st...@gmail.com>> wrote:
Hi Jaydeep,

That seems quite interesting. Couple points though:

1) It would be nice if there is a way to "subscribe" to decisions your detection framework comes up with. Integration with e.g. diagnostics subsystem would be beneficial. This should be pluggable - just coding up an interface to dump / react on the decisions how I want. This might also act as a notifier to other systems, e-mail, slack channels ...

2) Have you tried to incorporate this with the Guardrails framework? I think that if something is detected to be throttled or rejected (e.g writing to a table), there might be a guardrail which would be triggered dynamically in runtime. Guardrails are useful as such but here we might reuse them so we do not need to code it twice.

3) I am curious how complex this detection framework would be, it can be complicated pretty fast I guess. What would be desirable is to act on it in such a way that you will not put that node under even more pressure. In other words, your detection system should work in such a way that there will not be any "doom loop" whereby mere throttling of various parts of Cassandra you make it even worse for other nodes in the cluster. For example, if a particular node starts to be overwhelmed and you detect this and requests start to be rejected, is it not possible that Java driver would start to see this node as "erroneous" with delayed response time etc and it would start to prefer other nodes in the cluster when deciding what node to contact for query coordination? So you would put more load on other nodes, making them more susceptible to be throttled as well ...

Regards

Stefan Miklosovic

On Tue, Jan 16, 2024 at 6:41 PM Jaydeep Chovatia <ch...@gmail.com>> wrote:
Hi,

Happy New Year!

I would like to discuss the following idea:

Open-source Cassandra (CASSANDRA-15013<https://issues.apache.org/jira/browse/CASSANDRA-15013>) has an elementary built-in memory rate limiter based on the incoming payload from user requests. This rate limiter activates if any incoming user request’s payload exceeds certain thresholds. However, the existing rate limiter only solves limited-scope issues. Cassandra's server-side meltdown due to overload is a known problem. Often we see that a couple of busy nodes take down the entire Cassandra ring due to the ripple effect. The following document proposes a generic purpose comprehensive rate limiter that works considering system signals, such as CPU, and internal signals, such as thread pools. The rate limiter will have knobs to filter out internal traffic, system traffic, replication traffic, and furthermore based on the types of queries.

More design details to this doc: [OSS] Cassandra Generic Purpose Rate Limiter - Google Docs<https://docs.google.com/document/d/1w-A3fnoeBS6tS1ffBda_R0QR90olzFoMqLE7znFEUrQ/edit>

Please let me know your thoughts.

Jaydeep

Re: [EXTERNAL] [Discuss] Generic Purpose Rate Limiter in Cassandra

[Jon Haddad <jo...@rustyrazorblade.com>]

Oh, one last thing.  If the client drivers were to implement a rate limiter
based on each node's error rate, and had the ability to back off, paired
with CASSANDRA-19534 <https://issues.apache.org/jira/browse/CASSANDRA-19534>,
I think the majority of severe cluster outages that people experience would
simply disappear, making fast rejection of requests at the server side more
of an optimization than a necessity.

There was a nice Netflix post about it here [1], and was the inspiration
behind my latency target mode that's in easy-cass-stress.

https://netflixtechblog.medium.com/performance-under-load-3e6fa9a60581

Jon

On Sat, Sep 21, 2024 at 12:38 PM Jon Haddad <jo...@rustyrazorblade.com> wrote:

> Can you elaborate what “the bad” is here? Maybe a scenario would help. I’m
> trying to visualize what kind of workload would be running where you
> wouldn’t have timeouts or a deep queue yet a node is overloaded.  What is
> “the bad” if requests aren’t timing out? How is a node overloaded if there
> isn’t work queued up?
>
> The levers we have under our control are concurrent work and the queue of
> work. Applying a rate limiter to the queue is a roundabout way of limiting
> the queue depth, so i view it as unnecessary in a world where the queue has
> a limit.
>
> Dynamically sizing the depth of the queue based on timeouts allows us to
> adjust it based on the heuristic of the amount of work that was able to be
> completed successfully in the past.  If you know that the queue has seen
> timeouts when it gets deeper than 100k requests, you stop accepting
> requests after 100k is reached.  The rate of rejection will be roughly
> correlated to the number of requests that would otherwise timeout.
> Basically we say, we can’t serve you, so don’t even get in line, and we use
> the length of the line as our throttle.
>
> Fwiw, I fully acknowledge that I didn’t invent any of this, it’s how
> hardware works already. I’m just applying the same concepts to our software
> queues that cpu, disks, and network cards already successfully use.
>
> Jon
>
> On Sat, Sep 21, 2024 at 12:05 PM Jordan West <jo...@gmail.com> wrote:
>
>> I agree with Josh. We need to be able to protect from a sudden burst of
>> traffic. 19534 went a long way in that regard — at least wrt to minimizing
>> the effects. The challenge with latency and queue depths can be that they
>> trigger when the bad has already occurred.
>>
>> One other thing we are considering internally in that project I mentioned
>> is having separate thresholds. One that starts to slow traffic and one that
>> rejects it.
>>
>> All this said, I think our intuition can guide us to initial
>> implementations but we I’m still pretty adamant that our decisions should
>> be guided by testing and numbers for this CEP.
>>
>> Jordan
>>
>> On Sat, Sep 21, 2024 at 04:36 Josh McKenzie <jm...@apache.org> wrote:
>>
>>> Are those three sufficient to protect against a client that unexpectedly
>>> comes up with 100x a previous provisioned-for workload? Or 100 clients at
>>> 100x concurrently? Given that can be 100x in terms of quantity (helped by
>>> queueing and shedding), but also 100x in terms of *computational and
>>> disk implications*? We don't have control over what users do, and while
>>> in an ideal world client applications have fairly predictable workloads
>>> over time, in practice things spike around various events for different
>>> applications.
>>>
>>> i.e. 1 thing in a queue could produce orders of magnitude more work than
>>> 1 other thing in a queue. Especially as we move into a world with SAI and
>>> Accord.
>>>
>>> we need to solve load balancing (summarized by the above three points)
>>> before we start working on the rate limiter
>>>
>>> I see these 2 things as complementary, not as interdependent. Is there
>>> something I'm missing?
>>>
>>> At least for me, mentally I frame this as "How can nodes protect
>>> themselves from variable and changing user behavior in a way that's
>>> minimally disruptive to the user and requires as little configuration as
>>> possible for operators?". Basically, how do we keep the limits of node
>>> performance from leaking into the scope of user awareness and
>>> responsibility outside simply pushing various exceptions to the client to
>>> indicate what's going on (OverloadedException to the client, etc).
>>>
>>> It seems to me both rate limiting and resource balancing are integral
>>> parts of this, but also parts that could be worked on independently. Were
>>> we to wave a magic wand tomorrow and have robust rate limiting, as we
>>> improved load balancing over time it would raise the ceiling at which rate
>>> limiting kicked in.
>>>
>>> So concretely to the thread, I think I agree with Jon:
>>>
>>> * use the rate of timeouts to limit the depth of the queues for each of
>>> the thread pools
>>> * reject requests when the queue is full with an OverloadedException.
>>>
>>> followed by:
>>>
>>> If you want to follow this up with the ability to dynamically resize
>>> thread pools that could be interesting.
>>>
>>>
>>> Simple is very much a feature here.
>>>
>>> On Sat, Sep 21, 2024, at 5:20 AM, Alex Petrov wrote:
>>>
>>> > Personally, I’m a bit skeptical that we will come up with a metric
>>> based heuristic that works well in most scenarios and doesn’t require
>>> significant knowledge and tuning. I think past implementations of the
>>> dynamic snitch are good evidence of that.
>>>
>>> I am more optimistic on that font. I think we can achieve a lot.
>>> However, in my opinion, we need to focus on balancing the load rather than
>>> rate limiting. Rate limiting is going to be important if/when we decide to
>>> implement workload isolation. Until then, I think we should focus on three
>>> things:
>>>
>>>   * Node health (Nodes should produce useful work and should be stable
>>> and not overloaded)
>>>   * Latency (we always need to find an optimal way to process request
>>> and minimize overall queueing time)
>>>   * Fairness (avoid workload and utilization imbalances)
>>>
>>> All three points are achievable with very straightforward approaches
>>> that will not require much operator involvement.
>>>
>>> I guess my main point is we need to solve load balancing (summarized by
>>> the above three points) before we start working on the rate limiter, but
>>> there's a good chance we may not need one apart from use cases that require
>>> workload isolation.
>>>
>>>
>>> On Fri, Sep 20, 2024, at 8:14 PM, Jordan West wrote:
>>>
>>> +1 to Benedict’s (and others) comments on plugability and low overhead
>>> when disabled. The latter I think needs little justification. The reason I
>>> am big on the former is, in my opinion: decisions on approach need to be
>>> settled with numbers not anecdotes or past experience (including my own).
>>> So I would like to see us compare different approaches (what metrics to
>>> use, etc).
>>>
>>> Personally, I’m a bit skeptical that we will come up with a metric based
>>> heuristic that works well in most scenarios and doesn’t require significant
>>> knowledge and tuning. I think past implementations of the dynamic snitch
>>> are good evidence of that. However, I expressed the same concerns
>>> internally for a client level project where we exposed metrics to induce
>>> back pressure and early experiments are encouraging / contrary to my
>>> expectations. At different layers different approaches can work better or
>>> worse. Same with different workloads. I don’t think we should dismiss
>>> approaches out right in this thread without hard numbers.
>>>
>>> In short, I think the testing and evaluation of this CEP is as important
>>> as its design and implementation. We will need to test a wide variety of
>>> workloads and potentially implementations and that’s where pluggability
>>> will be a huge benefit. I would go as far as saying the CEP should focus
>>> more on a framework for pluggable implementations that has low to zero cost
>>> when disabled than a specific set of metrics to use or specific approach.
>>>
>>> Jordan
>>>
>>> On Thu, Sep 19, 2024 at 14:38 Benedict Elliott Smith <
>>> benedict@apache.org> wrote:
>>>
>>> I just want to flag here that this is a topic I have strong opinions on,
>>> but the CEP is not really specific or detailed enough to understand
>>> precisely how it will be implemented. So, if a patch is already being
>>> produced, most of my feedback is likely to be provided some time after a
>>> patch appears, through the normal review process. I want to flag this now
>>> to avoid any surprise.
>>>
>>> I will say that upfront that, ideally, this system should be designed to
>>> have ~zero overhead when disabled, and with minimal coupling (between its
>>> own components and C* itself), so that entirely orthogonal approaches can
>>> be integrated in future without polluting the codebase.
>>>
>>>
>>> On 19 Sep 2024, at 19:14, Patrick McFadin <pm...@gmail.com> wrote:
>>>
>>> The work has begun but we don't have a VOTE thread for this CEP. Can one
>>> get started?
>>>
>>> On Mon, May 6, 2024 at 9:24 PM Jaydeep Chovatia <
>>> chovatia.jaydeep@gmail.com> wrote:
>>>
>>> Sure, Caleb. I will include the work as part of CASSANDRA-19534
>>> <https://issues.apache.org/jira/browse/CASSANDRA-19534> in the CEP-41.
>>>
>>> Jaydeep
>>>
>>> On Fri, May 3, 2024 at 7:48 AM Caleb Rackliffe <ca...@gmail.com>
>>> wrote:
>>>
>>> FYI, there is some ongoing sort-of-related work going on in
>>> CASSANDRA-19534 <https://issues.apache.org/jira/browse/CASSANDRA-19534>
>>>
>>> On Wed, Apr 10, 2024 at 6:35 PM Jaydeep Chovatia <
>>> chovatia.jaydeep@gmail.com> wrote:
>>>
>>> Just created an official CEP-41
>>> <https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-41+%28DRAFT%29+Apache+Cassandra+Unified+Rate+Limiter>
>>> incorporating the feedback from this discussion. Feel free to let me know
>>> if I may have missed some important feedback in this thread that is not
>>> captured in the CEP-41.
>>>
>>> Jaydeep
>>>
>>> On Thu, Feb 22, 2024 at 11:36 AM Jaydeep Chovatia <
>>> chovatia.jaydeep@gmail.com> wrote:
>>>
>>> Thanks, Josh. I will file an official CEP with all the details in a few
>>> days and update this thread with that CEP number.
>>> Thanks a lot everyone for providing valuable insights!
>>>
>>> Jaydeep
>>>
>>> On Thu, Feb 22, 2024 at 9:24 AM Josh McKenzie <jm...@apache.org>
>>> wrote:
>>>
>>>
>>> Do folks think we should file an official CEP and take it there?
>>>
>>> +1 here.
>>>
>>> Synthesizing your gdoc, Caleb's work, and the feedback from this thread
>>> into a draft seems like a solid next step.
>>>
>>> On Wed, Feb 7, 2024, at 12:31 PM, Jaydeep Chovatia wrote:
>>>
>>> I see a lot of great ideas being discussed or proposed in the past to
>>> cover the most common rate limiter candidate use cases. Do folks think we
>>> should file an official CEP and take it there?
>>>
>>> Jaydeep
>>>
>>> On Fri, Feb 2, 2024 at 8:30 AM Caleb Rackliffe <ca...@gmail.com>
>>> wrote:
>>>
>>> I just remembered the other day that I had done a quick writeup on the
>>> state of compaction stress-related throttling in the project:
>>>
>>>
>>> https://docs.google.com/document/d/1dfTEcKVidRKC1EWu3SO1kE1iVLMdaJ9uY1WMpS3P_hs/edit?usp=sharing
>>>
>>> I'm sure most of it is old news to the people on this thread, but I
>>> figured I'd post it just in case :)
>>>
>>> On Tue, Jan 30, 2024 at 11:58 AM Josh McKenzie <jm...@apache.org>
>>> wrote:
>>>
>>>
>>> 2.) We should make sure the links between the "known" root causes of
>>> cascading failures and the mechanisms we introduce to avoid them remain
>>> very strong.
>>>
>>> Seems to me that our historical strategy was to address individual known
>>> cases one-by-one rather than looking for a more holistic load-balancing and
>>> load-shedding solution. While the engineer in me likes the elegance of a
>>> broad, more-inclusive *actual SEDA-like* approach, the pragmatist in me
>>> wonders how far we think we are today from a stable set-point.
>>>
>>> i.e. are we facing a handful of cases where nodes can still get pushed
>>> over and then cascade that we can surgically address, or are we facing a
>>> broader lack of back-pressure that rears its head in different domains
>>> (client -> coordinator, coordinator -> replica, internode with other
>>> operations, etc) at surprising times and should be considered more
>>> holistically?
>>>
>>> On Tue, Jan 30, 2024, at 12:31 AM, Caleb Rackliffe wrote:
>>>
>>> I almost forgot CASSANDRA-15817, which introduced
>>> reject_repair_compaction_threshold, which provides a mechanism to stop
>>> repairs while compaction is underwater.
>>>
>>> On Jan 26, 2024, at 6:22 PM, Caleb Rackliffe <ca...@gmail.com>
>>> wrote:
>>>
>>> 
>>> Hey all,
>>>
>>> I'm a bit late to the discussion. I see that we've already discussed
>>> CASSANDRA-15013 <https://issues.apache.org/jira/browse/CASSANDRA-15013>
>>>  and CASSANDRA-16663
>>> <https://issues.apache.org/jira/browse/CASSANDRA-16663> at least in
>>> passing. Having written the latter, I'd be the first to admit it's a crude
>>> tool, although it's been useful here and there, and provides a couple
>>> primitives that may be useful for future work. As Scott mentions, while it
>>> is configurable at runtime, it is not adaptive, although we did
>>> make configuration easier in CASSANDRA-17423
>>> <https://issues.apache.org/jira/browse/CASSANDRA-17423>. It also is
>>> global to the node, although we've lightly discussed some ideas around
>>> making it more granular. (For example, keyspace-based limiting, or limiting
>>> "domains" tagged by the client in requests, could be interesting.) It also
>>> does not deal with inter-node traffic, of course.
>>>
>>> Something we've not yet mentioned (that does address internode traffic)
>>> is CASSANDRA-17324
>>> <https://issues.apache.org/jira/browse/CASSANDRA-17324>, which I
>>> proposed shortly after working on the native request limiter (and have just
>>> not had much time to return to). The basic idea is this:
>>>
>>> When a node is struggling under the weight of a compaction backlog and
>>> becomes a cause of increased read latency for clients, we have two safety
>>> valves:
>>>
>>>
>>> 1.) Disabling the native protocol server, which stops the node from
>>> coordinating reads and writes.
>>> 2.) Jacking up the severity on the node, which tells the dynamic snitch
>>> to avoid the node for reads from other coordinators.
>>>
>>>
>>> These are useful, but we don’t appear to have any mechanism that would
>>> allow us to temporarily reject internode hint, batch, and mutation messages
>>> that could further delay resolution of the compaction backlog.
>>>
>>>
>>> Whether it's done as part of a larger framework or on its own, it still
>>> feels like a good idea.
>>>
>>> Thinking in terms of opportunity costs here (i.e. where we spend our
>>> finite engineering time to holistically improve the experience of operating
>>> this database) is healthy, but we probably haven't reached the point of
>>> diminishing returns on nodes being able to protect themselves from clients
>>> and from other nodes. I would just keep in mind two things:
>>>
>>> 1.) The effectiveness of rate-limiting in the system (which includes the
>>> database and all clients) as a whole necessarily decreases as we move from
>>> the application to the lowest-level database internals. Limiting correctly
>>> at the client will save more resources than limiting at the native protocol
>>> server, and limiting correctly at the native protocol server will save more
>>> resources than limiting after we've dispatched requests to some thread pool
>>> for processing.
>>> 2.) We should make sure the links between the "known" root causes of
>>> cascading failures and the mechanisms we introduce to avoid them remain
>>> very strong.
>>>
>>> In any case, I'd be happy to help out in any way I can as this moves
>>> forward (especially as it relates to our past/current attempts to address
>>> this problem space).
>>>
>>>
>>>
>>>
>>>
>>>

Re: [EXTERNAL] [Discuss] Generic Purpose Rate Limiter in Cassandra

[Jon Haddad <jo...@rustyrazorblade.com>]

Can you elaborate what “the bad” is here? Maybe a scenario would help. I’m
trying to visualize what kind of workload would be running where you
wouldn’t have timeouts or a deep queue yet a node is overloaded.  What is
“the bad” if requests aren’t timing out? How is a node overloaded if there
isn’t work queued up?

The levers we have under our control are concurrent work and the queue of
work. Applying a rate limiter to the queue is a roundabout way of limiting
the queue depth, so i view it as unnecessary in a world where the queue has
a limit.

Dynamically sizing the depth of the queue based on timeouts allows us to
adjust it based on the heuristic of the amount of work that was able to be
completed successfully in the past.  If you know that the queue has seen
timeouts when it gets deeper than 100k requests, you stop accepting
requests after 100k is reached.  The rate of rejection will be roughly
correlated to the number of requests that would otherwise timeout.
Basically we say, we can’t serve you, so don’t even get in line, and we use
the length of the line as our throttle.

Fwiw, I fully acknowledge that I didn’t invent any of this, it’s how
hardware works already. I’m just applying the same concepts to our software
queues that cpu, disks, and network cards already successfully use.

Jon

On Sat, Sep 21, 2024 at 12:05 PM Jordan West <jo...@gmail.com> wrote:

> I agree with Josh. We need to be able to protect from a sudden burst of
> traffic. 19534 went a long way in that regard — at least wrt to minimizing
> the effects. The challenge with latency and queue depths can be that they
> trigger when the bad has already occurred.
>
> One other thing we are considering internally in that project I mentioned
> is having separate thresholds. One that starts to slow traffic and one that
> rejects it.
>
> All this said, I think our intuition can guide us to initial
> implementations but we I’m still pretty adamant that our decisions should
> be guided by testing and numbers for this CEP.
>
> Jordan
>
> On Sat, Sep 21, 2024 at 04:36 Josh McKenzie <jm...@apache.org> wrote:
>
>> Are those three sufficient to protect against a client that unexpectedly
>> comes up with 100x a previous provisioned-for workload? Or 100 clients at
>> 100x concurrently? Given that can be 100x in terms of quantity (helped by
>> queueing and shedding), but also 100x in terms of *computational and
>> disk implications*? We don't have control over what users do, and while
>> in an ideal world client applications have fairly predictable workloads
>> over time, in practice things spike around various events for different
>> applications.
>>
>> i.e. 1 thing in a queue could produce orders of magnitude more work than
>> 1 other thing in a queue. Especially as we move into a world with SAI and
>> Accord.
>>
>> we need to solve load balancing (summarized by the above three points)
>> before we start working on the rate limiter
>>
>> I see these 2 things as complementary, not as interdependent. Is there
>> something I'm missing?
>>
>> At least for me, mentally I frame this as "How can nodes protect
>> themselves from variable and changing user behavior in a way that's
>> minimally disruptive to the user and requires as little configuration as
>> possible for operators?". Basically, how do we keep the limits of node
>> performance from leaking into the scope of user awareness and
>> responsibility outside simply pushing various exceptions to the client to
>> indicate what's going on (OverloadedException to the client, etc).
>>
>> It seems to me both rate limiting and resource balancing are integral
>> parts of this, but also parts that could be worked on independently. Were
>> we to wave a magic wand tomorrow and have robust rate limiting, as we
>> improved load balancing over time it would raise the ceiling at which rate
>> limiting kicked in.
>>
>> So concretely to the thread, I think I agree with Jon:
>>
>> * use the rate of timeouts to limit the depth of the queues for each of
>> the thread pools
>> * reject requests when the queue is full with an OverloadedException.
>>
>> followed by:
>>
>> If you want to follow this up with the ability to dynamically resize
>> thread pools that could be interesting.
>>
>>
>> Simple is very much a feature here.
>>
>> On Sat, Sep 21, 2024, at 5:20 AM, Alex Petrov wrote:
>>
>> > Personally, I’m a bit skeptical that we will come up with a metric
>> based heuristic that works well in most scenarios and doesn’t require
>> significant knowledge and tuning. I think past implementations of the
>> dynamic snitch are good evidence of that.
>>
>> I am more optimistic on that font. I think we can achieve a lot. However,
>> in my opinion, we need to focus on balancing the load rather than rate
>> limiting. Rate limiting is going to be important if/when we decide to
>> implement workload isolation. Until then, I think we should focus on three
>> things:
>>
>>   * Node health (Nodes should produce useful work and should be stable
>> and not overloaded)
>>   * Latency (we always need to find an optimal way to process request and
>> minimize overall queueing time)
>>   * Fairness (avoid workload and utilization imbalances)
>>
>> All three points are achievable with very straightforward approaches that
>> will not require much operator involvement.
>>
>> I guess my main point is we need to solve load balancing (summarized by
>> the above three points) before we start working on the rate limiter, but
>> there's a good chance we may not need one apart from use cases that require
>> workload isolation.
>>
>>
>> On Fri, Sep 20, 2024, at 8:14 PM, Jordan West wrote:
>>
>> +1 to Benedict’s (and others) comments on plugability and low overhead
>> when disabled. The latter I think needs little justification. The reason I
>> am big on the former is, in my opinion: decisions on approach need to be
>> settled with numbers not anecdotes or past experience (including my own).
>> So I would like to see us compare different approaches (what metrics to
>> use, etc).
>>
>> Personally, I’m a bit skeptical that we will come up with a metric based
>> heuristic that works well in most scenarios and doesn’t require significant
>> knowledge and tuning. I think past implementations of the dynamic snitch
>> are good evidence of that. However, I expressed the same concerns
>> internally for a client level project where we exposed metrics to induce
>> back pressure and early experiments are encouraging / contrary to my
>> expectations. At different layers different approaches can work better or
>> worse. Same with different workloads. I don’t think we should dismiss
>> approaches out right in this thread without hard numbers.
>>
>> In short, I think the testing and evaluation of this CEP is as important
>> as its design and implementation. We will need to test a wide variety of
>> workloads and potentially implementations and that’s where pluggability
>> will be a huge benefit. I would go as far as saying the CEP should focus
>> more on a framework for pluggable implementations that has low to zero cost
>> when disabled than a specific set of metrics to use or specific approach.
>>
>> Jordan
>>
>> On Thu, Sep 19, 2024 at 14:38 Benedict Elliott Smith <be...@apache.org>
>> wrote:
>>
>> I just want to flag here that this is a topic I have strong opinions on,
>> but the CEP is not really specific or detailed enough to understand
>> precisely how it will be implemented. So, if a patch is already being
>> produced, most of my feedback is likely to be provided some time after a
>> patch appears, through the normal review process. I want to flag this now
>> to avoid any surprise.
>>
>> I will say that upfront that, ideally, this system should be designed to
>> have ~zero overhead when disabled, and with minimal coupling (between its
>> own components and C* itself), so that entirely orthogonal approaches can
>> be integrated in future without polluting the codebase.
>>
>>
>> On 19 Sep 2024, at 19:14, Patrick McFadin <pm...@gmail.com> wrote:
>>
>> The work has begun but we don't have a VOTE thread for this CEP. Can one
>> get started?
>>
>> On Mon, May 6, 2024 at 9:24 PM Jaydeep Chovatia <
>> chovatia.jaydeep@gmail.com> wrote:
>>
>> Sure, Caleb. I will include the work as part of CASSANDRA-19534
>> <https://issues.apache.org/jira/browse/CASSANDRA-19534> in the CEP-41.
>>
>> Jaydeep
>>
>> On Fri, May 3, 2024 at 7:48 AM Caleb Rackliffe <ca...@gmail.com>
>> wrote:
>>
>> FYI, there is some ongoing sort-of-related work going on in
>> CASSANDRA-19534 <https://issues.apache.org/jira/browse/CASSANDRA-19534>
>>
>> On Wed, Apr 10, 2024 at 6:35 PM Jaydeep Chovatia <
>> chovatia.jaydeep@gmail.com> wrote:
>>
>> Just created an official CEP-41
>> <https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-41+%28DRAFT%29+Apache+Cassandra+Unified+Rate+Limiter>
>> incorporating the feedback from this discussion. Feel free to let me know
>> if I may have missed some important feedback in this thread that is not
>> captured in the CEP-41.
>>
>> Jaydeep
>>
>> On Thu, Feb 22, 2024 at 11:36 AM Jaydeep Chovatia <
>> chovatia.jaydeep@gmail.com> wrote:
>>
>> Thanks, Josh. I will file an official CEP with all the details in a few
>> days and update this thread with that CEP number.
>> Thanks a lot everyone for providing valuable insights!
>>
>> Jaydeep
>>
>> On Thu, Feb 22, 2024 at 9:24 AM Josh McKenzie <jm...@apache.org>
>> wrote:
>>
>>
>> Do folks think we should file an official CEP and take it there?
>>
>> +1 here.
>>
>> Synthesizing your gdoc, Caleb's work, and the feedback from this thread
>> into a draft seems like a solid next step.
>>
>> On Wed, Feb 7, 2024, at 12:31 PM, Jaydeep Chovatia wrote:
>>
>> I see a lot of great ideas being discussed or proposed in the past to
>> cover the most common rate limiter candidate use cases. Do folks think we
>> should file an official CEP and take it there?
>>
>> Jaydeep
>>
>> On Fri, Feb 2, 2024 at 8:30 AM Caleb Rackliffe <ca...@gmail.com>
>> wrote:
>>
>> I just remembered the other day that I had done a quick writeup on the
>> state of compaction stress-related throttling in the project:
>>
>>
>> https://docs.google.com/document/d/1dfTEcKVidRKC1EWu3SO1kE1iVLMdaJ9uY1WMpS3P_hs/edit?usp=sharing
>>
>> I'm sure most of it is old news to the people on this thread, but I
>> figured I'd post it just in case :)
>>
>> On Tue, Jan 30, 2024 at 11:58 AM Josh McKenzie <jm...@apache.org>
>> wrote:
>>
>>
>> 2.) We should make sure the links between the "known" root causes of
>> cascading failures and the mechanisms we introduce to avoid them remain
>> very strong.
>>
>> Seems to me that our historical strategy was to address individual known
>> cases one-by-one rather than looking for a more holistic load-balancing and
>> load-shedding solution. While the engineer in me likes the elegance of a
>> broad, more-inclusive *actual SEDA-like* approach, the pragmatist in me
>> wonders how far we think we are today from a stable set-point.
>>
>> i.e. are we facing a handful of cases where nodes can still get pushed
>> over and then cascade that we can surgically address, or are we facing a
>> broader lack of back-pressure that rears its head in different domains
>> (client -> coordinator, coordinator -> replica, internode with other
>> operations, etc) at surprising times and should be considered more
>> holistically?
>>
>> On Tue, Jan 30, 2024, at 12:31 AM, Caleb Rackliffe wrote:
>>
>> I almost forgot CASSANDRA-15817, which introduced
>> reject_repair_compaction_threshold, which provides a mechanism to stop
>> repairs while compaction is underwater.
>>
>> On Jan 26, 2024, at 6:22 PM, Caleb Rackliffe <ca...@gmail.com>
>> wrote:
>>
>> 
>> Hey all,
>>
>> I'm a bit late to the discussion. I see that we've already discussed
>> CASSANDRA-15013 <https://issues.apache.org/jira/browse/CASSANDRA-15013>
>>  and CASSANDRA-16663
>> <https://issues.apache.org/jira/browse/CASSANDRA-16663> at least in
>> passing. Having written the latter, I'd be the first to admit it's a crude
>> tool, although it's been useful here and there, and provides a couple
>> primitives that may be useful for future work. As Scott mentions, while it
>> is configurable at runtime, it is not adaptive, although we did
>> make configuration easier in CASSANDRA-17423
>> <https://issues.apache.org/jira/browse/CASSANDRA-17423>. It also is
>> global to the node, although we've lightly discussed some ideas around
>> making it more granular. (For example, keyspace-based limiting, or limiting
>> "domains" tagged by the client in requests, could be interesting.) It also
>> does not deal with inter-node traffic, of course.
>>
>> Something we've not yet mentioned (that does address internode traffic)
>> is CASSANDRA-17324
>> <https://issues.apache.org/jira/browse/CASSANDRA-17324>, which I
>> proposed shortly after working on the native request limiter (and have just
>> not had much time to return to). The basic idea is this:
>>
>> When a node is struggling under the weight of a compaction backlog and
>> becomes a cause of increased read latency for clients, we have two safety
>> valves:
>>
>>
>> 1.) Disabling the native protocol server, which stops the node from
>> coordinating reads and writes.
>> 2.) Jacking up the severity on the node, which tells the dynamic snitch
>> to avoid the node for reads from other coordinators.
>>
>>
>> These are useful, but we don’t appear to have any mechanism that would
>> allow us to temporarily reject internode hint, batch, and mutation messages
>> that could further delay resolution of the compaction backlog.
>>
>>
>> Whether it's done as part of a larger framework or on its own, it still
>> feels like a good idea.
>>
>> Thinking in terms of opportunity costs here (i.e. where we spend our
>> finite engineering time to holistically improve the experience of operating
>> this database) is healthy, but we probably haven't reached the point of
>> diminishing returns on nodes being able to protect themselves from clients
>> and from other nodes. I would just keep in mind two things:
>>
>> 1.) The effectiveness of rate-limiting in the system (which includes the
>> database and all clients) as a whole necessarily decreases as we move from
>> the application to the lowest-level database internals. Limiting correctly
>> at the client will save more resources than limiting at the native protocol
>> server, and limiting correctly at the native protocol server will save more
>> resources than limiting after we've dispatched requests to some thread pool
>> for processing.
>> 2.) We should make sure the links between the "known" root causes of
>> cascading failures and the mechanisms we introduce to avoid them remain
>> very strong.
>>
>> In any case, I'd be happy to help out in any way I can as this moves
>> forward (especially as it relates to our past/current attempts to address
>> this problem space).
>>
>>
>>
>>
>>
>>

Re: [EXTERNAL] [Discuss] Generic Purpose Rate Limiter in Cassandra

[Jordan West <jo...@gmail.com>]

I agree with Josh. We need to be able to protect from a sudden burst of
traffic. 19534 went a long way in that regard — at least wrt to minimizing
the effects. The challenge with latency and queue depths can be that they
trigger when the bad has already occurred.

One other thing we are considering internally in that project I mentioned
is having separate thresholds. One that starts to slow traffic and one that
rejects it.

All this said, I think our intuition can guide us to initial
implementations but we I’m still pretty adamant that our decisions should
be guided by testing and numbers for this CEP.

Jordan

On Sat, Sep 21, 2024 at 04:36 Josh McKenzie <jm...@apache.org> wrote:

> Are those three sufficient to protect against a client that unexpectedly
> comes up with 100x a previous provisioned-for workload? Or 100 clients at
> 100x concurrently? Given that can be 100x in terms of quantity (helped by
> queueing and shedding), but also 100x in terms of *computational and disk
> implications*? We don't have control over what users do, and while in an
> ideal world client applications have fairly predictable workloads over
> time, in practice things spike around various events for different
> applications.
>
> i.e. 1 thing in a queue could produce orders of magnitude more work than 1
> other thing in a queue. Especially as we move into a world with SAI and
> Accord.
>
> we need to solve load balancing (summarized by the above three points)
> before we start working on the rate limiter
>
> I see these 2 things as complementary, not as interdependent. Is there
> something I'm missing?
>
> At least for me, mentally I frame this as "How can nodes protect
> themselves from variable and changing user behavior in a way that's
> minimally disruptive to the user and requires as little configuration as
> possible for operators?". Basically, how do we keep the limits of node
> performance from leaking into the scope of user awareness and
> responsibility outside simply pushing various exceptions to the client to
> indicate what's going on (OverloadedException to the client, etc).
>
> It seems to me both rate limiting and resource balancing are integral
> parts of this, but also parts that could be worked on independently. Were
> we to wave a magic wand tomorrow and have robust rate limiting, as we
> improved load balancing over time it would raise the ceiling at which rate
> limiting kicked in.
>
> So concretely to the thread, I think I agree with Jon:
>
> * use the rate of timeouts to limit the depth of the queues for each of
> the thread pools
> * reject requests when the queue is full with an OverloadedException.
>
> followed by:
>
> If you want to follow this up with the ability to dynamically resize
> thread pools that could be interesting.
>
>
> Simple is very much a feature here.
>
> On Sat, Sep 21, 2024, at 5:20 AM, Alex Petrov wrote:
>
> > Personally, I’m a bit skeptical that we will come up with a metric based
> heuristic that works well in most scenarios and doesn’t require significant
> knowledge and tuning. I think past implementations of the dynamic snitch
> are good evidence of that.
>
> I am more optimistic on that font. I think we can achieve a lot. However,
> in my opinion, we need to focus on balancing the load rather than rate
> limiting. Rate limiting is going to be important if/when we decide to
> implement workload isolation. Until then, I think we should focus on three
> things:
>
>   * Node health (Nodes should produce useful work and should be stable and
> not overloaded)
>   * Latency (we always need to find an optimal way to process request and minimize
> overall queueing time)
>   * Fairness (avoid workload and utilization imbalances)
>
> All three points are achievable with very straightforward approaches that
> will not require much operator involvement.
>
> I guess my main point is we need to solve load balancing (summarized by
> the above three points) before we start working on the rate limiter, but
> there's a good chance we may not need one apart from use cases that require
> workload isolation.
>
>
> On Fri, Sep 20, 2024, at 8:14 PM, Jordan West wrote:
>
> +1 to Benedict’s (and others) comments on plugability and low overhead
> when disabled. The latter I think needs little justification. The reason I
> am big on the former is, in my opinion: decisions on approach need to be
> settled with numbers not anecdotes or past experience (including my own).
> So I would like to see us compare different approaches (what metrics to
> use, etc).
>
> Personally, I’m a bit skeptical that we will come up with a metric based
> heuristic that works well in most scenarios and doesn’t require significant
> knowledge and tuning. I think past implementations of the dynamic snitch
> are good evidence of that. However, I expressed the same concerns
> internally for a client level project where we exposed metrics to induce
> back pressure and early experiments are encouraging / contrary to my
> expectations. At different layers different approaches can work better or
> worse. Same with different workloads. I don’t think we should dismiss
> approaches out right in this thread without hard numbers.
>
> In short, I think the testing and evaluation of this CEP is as important
> as its design and implementation. We will need to test a wide variety of
> workloads and potentially implementations and that’s where pluggability
> will be a huge benefit. I would go as far as saying the CEP should focus
> more on a framework for pluggable implementations that has low to zero cost
> when disabled than a specific set of metrics to use or specific approach.
>
> Jordan
>
> On Thu, Sep 19, 2024 at 14:38 Benedict Elliott Smith <be...@apache.org>
> wrote:
>
> I just want to flag here that this is a topic I have strong opinions on,
> but the CEP is not really specific or detailed enough to understand
> precisely how it will be implemented. So, if a patch is already being
> produced, most of my feedback is likely to be provided some time after a
> patch appears, through the normal review process. I want to flag this now
> to avoid any surprise.
>
> I will say that upfront that, ideally, this system should be designed to
> have ~zero overhead when disabled, and with minimal coupling (between its
> own components and C* itself), so that entirely orthogonal approaches can
> be integrated in future without polluting the codebase.
>
>
> On 19 Sep 2024, at 19:14, Patrick McFadin <pm...@gmail.com> wrote:
>
> The work has begun but we don't have a VOTE thread for this CEP. Can one
> get started?
>
> On Mon, May 6, 2024 at 9:24 PM Jaydeep Chovatia <
> chovatia.jaydeep@gmail.com> wrote:
>
> Sure, Caleb. I will include the work as part of CASSANDRA-19534
> <https://issues.apache.org/jira/browse/CASSANDRA-19534> in the CEP-41.
>
> Jaydeep
>
> On Fri, May 3, 2024 at 7:48 AM Caleb Rackliffe <ca...@gmail.com>
> wrote:
>
> FYI, there is some ongoing sort-of-related work going on in
> CASSANDRA-19534 <https://issues.apache.org/jira/browse/CASSANDRA-19534>
>
> On Wed, Apr 10, 2024 at 6:35 PM Jaydeep Chovatia <
> chovatia.jaydeep@gmail.com> wrote:
>
> Just created an official CEP-41
> <https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-41+%28DRAFT%29+Apache+Cassandra+Unified+Rate+Limiter>
> incorporating the feedback from this discussion. Feel free to let me know
> if I may have missed some important feedback in this thread that is not
> captured in the CEP-41.
>
> Jaydeep
>
> On Thu, Feb 22, 2024 at 11:36 AM Jaydeep Chovatia <
> chovatia.jaydeep@gmail.com> wrote:
>
> Thanks, Josh. I will file an official CEP with all the details in a few
> days and update this thread with that CEP number.
> Thanks a lot everyone for providing valuable insights!
>
> Jaydeep
>
> On Thu, Feb 22, 2024 at 9:24 AM Josh McKenzie <jm...@apache.org>
> wrote:
>
>
> Do folks think we should file an official CEP and take it there?
>
> +1 here.
>
> Synthesizing your gdoc, Caleb's work, and the feedback from this thread
> into a draft seems like a solid next step.
>
> On Wed, Feb 7, 2024, at 12:31 PM, Jaydeep Chovatia wrote:
>
> I see a lot of great ideas being discussed or proposed in the past to
> cover the most common rate limiter candidate use cases. Do folks think we
> should file an official CEP and take it there?
>
> Jaydeep
>
> On Fri, Feb 2, 2024 at 8:30 AM Caleb Rackliffe <ca...@gmail.com>
> wrote:
>
> I just remembered the other day that I had done a quick writeup on the
> state of compaction stress-related throttling in the project:
>
>
> https://docs.google.com/document/d/1dfTEcKVidRKC1EWu3SO1kE1iVLMdaJ9uY1WMpS3P_hs/edit?usp=sharing
>
> I'm sure most of it is old news to the people on this thread, but I
> figured I'd post it just in case :)
>
> On Tue, Jan 30, 2024 at 11:58 AM Josh McKenzie <jm...@apache.org>
> wrote:
>
>
> 2.) We should make sure the links between the "known" root causes of
> cascading failures and the mechanisms we introduce to avoid them remain
> very strong.
>
> Seems to me that our historical strategy was to address individual known
> cases one-by-one rather than looking for a more holistic load-balancing and
> load-shedding solution. While the engineer in me likes the elegance of a
> broad, more-inclusive *actual SEDA-like* approach, the pragmatist in me
> wonders how far we think we are today from a stable set-point.
>
> i.e. are we facing a handful of cases where nodes can still get pushed
> over and then cascade that we can surgically address, or are we facing a
> broader lack of back-pressure that rears its head in different domains
> (client -> coordinator, coordinator -> replica, internode with other
> operations, etc) at surprising times and should be considered more
> holistically?
>
> On Tue, Jan 30, 2024, at 12:31 AM, Caleb Rackliffe wrote:
>
> I almost forgot CASSANDRA-15817, which introduced
> reject_repair_compaction_threshold, which provides a mechanism to stop
> repairs while compaction is underwater.
>
> On Jan 26, 2024, at 6:22 PM, Caleb Rackliffe <ca...@gmail.com>
> wrote:
>
> 
> Hey all,
>
> I'm a bit late to the discussion. I see that we've already discussed
> CASSANDRA-15013 <https://issues.apache.org/jira/browse/CASSANDRA-15013>
>  and CASSANDRA-16663
> <https://issues.apache.org/jira/browse/CASSANDRA-16663> at least in
> passing. Having written the latter, I'd be the first to admit it's a crude
> tool, although it's been useful here and there, and provides a couple
> primitives that may be useful for future work. As Scott mentions, while it
> is configurable at runtime, it is not adaptive, although we did
> make configuration easier in CASSANDRA-17423
> <https://issues.apache.org/jira/browse/CASSANDRA-17423>. It also is
> global to the node, although we've lightly discussed some ideas around
> making it more granular. (For example, keyspace-based limiting, or limiting
> "domains" tagged by the client in requests, could be interesting.) It also
> does not deal with inter-node traffic, of course.
>
> Something we've not yet mentioned (that does address internode traffic) is
> CASSANDRA-17324 <https://issues.apache.org/jira/browse/CASSANDRA-17324>,
> which I proposed shortly after working on the native request limiter (and
> have just not had much time to return to). The basic idea is this:
>
> When a node is struggling under the weight of a compaction backlog and
> becomes a cause of increased read latency for clients, we have two safety
> valves:
>
>
> 1.) Disabling the native protocol server, which stops the node from
> coordinating reads and writes.
> 2.) Jacking up the severity on the node, which tells the dynamic snitch to
> avoid the node for reads from other coordinators.
>
>
> These are useful, but we don’t appear to have any mechanism that would
> allow us to temporarily reject internode hint, batch, and mutation messages
> that could further delay resolution of the compaction backlog.
>
>
> Whether it's done as part of a larger framework or on its own, it still
> feels like a good idea.
>
> Thinking in terms of opportunity costs here (i.e. where we spend our
> finite engineering time to holistically improve the experience of operating
> this database) is healthy, but we probably haven't reached the point of
> diminishing returns on nodes being able to protect themselves from clients
> and from other nodes. I would just keep in mind two things:
>
> 1.) The effectiveness of rate-limiting in the system (which includes the
> database and all clients) as a whole necessarily decreases as we move from
> the application to the lowest-level database internals. Limiting correctly
> at the client will save more resources than limiting at the native protocol
> server, and limiting correctly at the native protocol server will save more
> resources than limiting after we've dispatched requests to some thread pool
> for processing.
> 2.) We should make sure the links between the "known" root causes of
> cascading failures and the mechanisms we introduce to avoid them remain
> very strong.
>
> In any case, I'd be happy to help out in any way I can as this moves
> forward (especially as it relates to our past/current attempts to address
> this problem space).
>
>
>
>
>
>

Re: [EXTERNAL] [Discuss] Generic Purpose Rate Limiter in Cassandra

[Benedict Elliott Smith <be...@apache.org>]

If we’re debating the overall approach, I think we need to define what we want to achieve before we pursue any specific design.

I think rate limiting is simply a proxy for cluster stability. I think implicitly we also all want to achieve client fairness. Rate limiting is one proposal for achieving the first only - a poor one IMO, driven I think by a lack of guarantees currently provided by the rest of the system.

Given this lens, load balancing is IMO a performance improvement - an important one, but orthogonal to achieving the stability itself. My personal view is that the best next step for achieving both is improved load shedding, building on Alex’s earlier work, which I think is broadly what Jon is saying.

The best improvement IMO is ensuring load shedding is fair. That is, we should shed load to replicas that already have the most work sent to them, and for clients with the most work already queued. That is, we should partition the client work queue by client, and as we get back-pressure signals from overloaded replicas we should being materialising a view of the client work queue attributing targets to this replica. We can then expire work based on the worst queue depth of any queue. We should also use this to override the behaviour of the dynamic snitch; we should not send work to replicas with local queues that cannot flush due to back-pressure.

I think, once this is in place, the internal queue depths Jon discusses would be useful signals to the internode inbound message handler to propagate back to peers’ outbound message queues (and, transitively, the client message queue management).

I think there are also improvements to be made regarding how we select work to be expired, such as partitioning by workload type, predicting wether there is enough time for the system to process a query (e.g. not submitting work that is close to its expiration), as considering the size of the client payload, etc. Also, choosing when to notify a client of timeout/overload - it may be preferential to discard the work but delay notifying the client to avoid the client retrying too quickly.



> On 23 Sep 2024, at 08:39, Alex Petrov <al...@coffeenco.de> wrote:
> 
> > Are those three sufficient to protect against a client that unexpectedly comes up with 100x a previous provisioned-for workload? Or 100 clients at 100x concurrently? Given that can be 100x in terms of quantity (helped by queueing and shedding), but also 100x in terms of computational and disk implications? We don't have control over what users do, and while in an ideal world client applications have fairly predictable workloads over time, in practice things spike around various events for different applications.
> 
> To fully answer your question I would probably have to flesh out a CEP (which I hope to get to ASAP), and I will elaborate on all of the points on Community over Code this year (and will do my best to put it in writing for the ones who will not attend). But to briefly answer, one of the ideas is exactly the fact that one thing in a queue is not the same as a another thing in a queue. 
> 
> > I see these 2 things as complementary, not as interdependent. Is there something I'm missing?
> 
> I think if we start working on rate limiting before we implement good load balancing, we are risking of shedding load that could otherwise have been handled by the cluster. I think you even said it yourself with "over time it would raise the ceiling at which rate limiting kicked in".
> 
> Besides, in CASSANDRA-19534 <https://issues.apache.org/jira/browse/CASSANDRA-19534> I was attempting to show that we need to find a maximal natural possible throughput that the cluster can handle without tipping, and maintain it. And the easiest way to handle this was, naturally, through the user-set timeouts. We can work in resource limits, but as of now I see only marginal improvement over what we can do just with timeouts.
> 
> One of the risk of a misconfigured rate-limiter is avoidable throttling / shedding. For example, CEP mentions a trigger of a 80% CPU utilization. But while testing 19534, we have seen that we can easily burst into high CPU utilization, let the requests that do not satisfy client timeout boundaries get shedded, and continue operating without any additional rate-limiting. My guess is that the existing rate-limiter sees less usage than we wish it to primarily because it is hard to say where to set the limits to for throwing OverloadedException, and TCP throttling/backoff does not work because we lose the queue arrival timestamp and start triggering client timeouts that are invisible to a server (i.e. client retries while request is still in the queue). While latter problem has a trivial solution (i.e. client-set deadlines), former one probably requires some auto-tuning or guidance. 
> 
> Another example is "culprit Keyspaces" from the CEP. If we introduce fairness into our load-balancing, a single keyspace, or a replica set (partition) will not be able to dominate the cluster workload, causing accross the board timeouts. Which means that by simply giving preference to other requests we have naturally shed an imbalance without introducing any rate-limiting.
> 
> Maybe the problem is in the terminology, but I think we should choose "prioritize read/write coordinator workload" over "block any {read/write} {coordinator/replication} traffic for a table". Let me make an example. Let's say we implement some algorithm for replenishing request allowance for write replication for a table, and this table runs out of tokens. If we make a decision to shed this request before waiting until the last moment it potentially could have gotten processed, we are risking shedding a request that can be served. But if we gave such request a lower priority, we can get to it when we get to it given current resources and queues. If we still can process it, we will, even if it is on a very end of our timeout guarantee, and I think it does not need to be shed.
> 
> I think what I have in mind seems to jibe very well with the points you have brought up:
>   * How can nodes protect themselves from variable and changing user behavior in a way that's minimally disruptive to the user and requires as little configuration as possible for operators
>   * How do we keep the limits of node performance from leaking into the scope of user awareness and responsibility outside simply pushing various exceptions to the client to indicate what's going on 
> 
> To summarize, I think good load balancing and workload prioritization combined with just "give up on a request if we know for a fact we can not process it" feels like a simpler way to solve the two problems you mentioned, and it will help us maximize cluster utilization while not shedding the load that could otherwise have been served, _while_ also reducing latencies across the board.
> 
> 
> On Sat, Sep 21, 2024, at 1:35 PM, Josh McKenzie wrote:
>> Are those three sufficient to protect against a client that unexpectedly comes up with 100x a previous provisioned-for workload? Or 100 clients at 100x concurrently? Given that can be 100x in terms of quantity (helped by queueing and shedding), but also 100x in terms of computational and disk implications? We don't have control over what users do, and while in an ideal world client applications have fairly predictable workloads over time, in practice things spike around various events for different applications.
>> 
>> i.e. 1 thing in a queue could produce orders of magnitude more work than 1 other thing in a queue. Especially as we move into a world with SAI and Accord.
>> 
>>> we need to solve load balancing (summarized by the above three points) before we start working on the rate limiter
>> I see these 2 things as complementary, not as interdependent. Is there something I'm missing?
>> 
>> At least for me, mentally I frame this as "How can nodes protect themselves from variable and changing user behavior in a way that's minimally disruptive to the user and requires as little configuration as possible for operators?". Basically, how do we keep the limits of node performance from leaking into the scope of user awareness and responsibility outside simply pushing various exceptions to the client to indicate what's going on (OverloadedException to the client, etc).
>> 
>> It seems to me both rate limiting and resource balancing are integral parts of this, but also parts that could be worked on independently. Were we to wave a magic wand tomorrow and have robust rate limiting, as we improved load balancing over time it would raise the ceiling at which rate limiting kicked in.
>> 
>> So concretely to the thread, I think I agree with Jon:
>>> * use the rate of timeouts to limit the depth of the queues for each of the thread pools
>>> * reject requests when the queue is full with an OverloadedException.
>> followed by:
>>> If you want to follow this up with the ability to dynamically resize thread pools that could be interesting.
>> 
>> Simple is very much a feature here.
>> 
>> On Sat, Sep 21, 2024, at 5:20 AM, Alex Petrov wrote:
>>> > Personally, I’m a bit skeptical that we will come up with a metric based heuristic that works well in most scenarios and doesn’t require significant knowledge and tuning. I think past implementations of the dynamic snitch are good evidence of that.
>>> 
>>> I am more optimistic on that font. I think we can achieve a lot. However, in my opinion, we need to focus on balancing the load rather than rate limiting. Rate limiting is going to be important if/when we decide to implement workload isolation. Until then, I think we should focus on three things:
>>> 
>>>   * Node health (Nodes should produce useful work and should be stable and not overloaded)
>>>   * Latency (we always need to find an optimal way to process request and minimize overall queueing time)
>>>   * Fairness (avoid workload and utilization imbalances)
>>> 
>>> All three points are achievable with very straightforward approaches that will not require much operator involvement.
>>> 
>>> I guess my main point is we need to solve load balancing (summarized by the above three points) before we start working on the rate limiter, but there's a good chance we may not need one apart from use cases that require workload isolation. 
>>> 
>>> 
>>> On Fri, Sep 20, 2024, at 8:14 PM, Jordan West wrote:
>>>> +1 to Benedict’s (and others) comments on plugability and low overhead when disabled. The latter I think needs little justification. The reason I am big on the former is, in my opinion: decisions on approach need to be settled with numbers not anecdotes or past experience (including my own). So I would like to see us compare different approaches (what metrics to use, etc). 
>>>> 
>>>> Personally, I’m a bit skeptical that we will come up with a metric based heuristic that works well in most scenarios and doesn’t require significant knowledge and tuning. I think past implementations of the dynamic snitch are good evidence of that. However, I expressed the same concerns internally for a client level project where we exposed metrics to induce back pressure and early experiments are encouraging / contrary to my expectations. At different layers different approaches can work better or worse. Same with different workloads. I don’t think we should dismiss approaches out right in this thread without hard numbers. 
>>>> 
>>>> In short, I think the testing and evaluation of this CEP is as important as its design and implementation. We will need to test a wide variety of workloads and potentially implementations and that’s where pluggability will be a huge benefit. I would go as far as saying the CEP should focus more on a framework for pluggable implementations that has low to zero cost when disabled than a specific set of metrics to use or specific approach. 
>>>> 
>>>> Jordan 
>>>> 
>>>> On Thu, Sep 19, 2024 at 14:38 Benedict Elliott Smith <benedict@apache.org <ma...@apache.org>> wrote:
>>>> I just want to flag here that this is a topic I have strong opinions on, but the CEP is not really specific or detailed enough to understand precisely how it will be implemented. So, if a patch is already being produced, most of my feedback is likely to be provided some time after a patch appears, through the normal review process. I want to flag this now to avoid any surprise.
>>>> 
>>>> I will say that upfront that, ideally, this system should be designed to have ~zero overhead when disabled, and with minimal coupling (between its own components and C* itself), so that entirely orthogonal approaches can be integrated in future without polluting the codebase.
>>>> 
>>>> 
>>>>> On 19 Sep 2024, at 19:14, Patrick McFadin <pmcfadin@gmail.com <ma...@gmail.com>> wrote:
>>>>> 
>>>>> The work has begun but we don't have a VOTE thread for this CEP. Can one get started?
>>>>> 
>>>>> On Mon, May 6, 2024 at 9:24 PM Jaydeep Chovatia <chovatia.jaydeep@gmail.com <ma...@gmail.com>> wrote:
>>>>> Sure, Caleb. I will include the work as part of CASSANDRA-19534 <https://issues.apache.org/jira/browse/CASSANDRA-19534> in the CEP-41.
>>>>> 
>>>>> Jaydeep
>>>>> 
>>>>> On Fri, May 3, 2024 at 7:48 AM Caleb Rackliffe <calebrackliffe@gmail.com <ma...@gmail.com>> wrote:
>>>>> FYI, there is some ongoing sort-of-related work going on in CASSANDRA-19534 <https://issues.apache.org/jira/browse/CASSANDRA-19534>
>>>>> 
>>>>> On Wed, Apr 10, 2024 at 6:35 PM Jaydeep Chovatia <chovatia.jaydeep@gmail.com <ma...@gmail.com>> wrote:
>>>>> Just created an official CEP-41 <https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-41+%28DRAFT%29+Apache+Cassandra+Unified+Rate+Limiter> incorporating the feedback from this discussion. Feel free to let me know if I may have missed some important feedback in this thread that is not captured in the CEP-41.
>>>>> 
>>>>> Jaydeep
>>>>> 
>>>>> On Thu, Feb 22, 2024 at 11:36 AM Jaydeep Chovatia <chovatia.jaydeep@gmail.com <ma...@gmail.com>> wrote:
>>>>> Thanks, Josh. I will file an official CEP with all the details in a few days and update this thread with that CEP number.
>>>>> Thanks a lot everyone for providing valuable insights!
>>>>> 
>>>>> Jaydeep
>>>>> 
>>>>> On Thu, Feb 22, 2024 at 9:24 AM Josh McKenzie <jmckenzie@apache.org <ma...@apache.org>> wrote:
>>>>> 
>>>>>> Do folks think we should file an official CEP and take it there?
>>>>> +1 here.
>>>>> 
>>>>> Synthesizing your gdoc, Caleb's work, and the feedback from this thread into a draft seems like a solid next step.
>>>>> 
>>>>> On Wed, Feb 7, 2024, at 12:31 PM, Jaydeep Chovatia wrote:
>>>>>> I see a lot of great ideas being discussed or proposed in the past to cover the most common rate limiter candidate use cases. Do folks think we should file an official CEP and take it there?
>>>>>> 
>>>>>> Jaydeep
>>>>>> 
>>>>>> On Fri, Feb 2, 2024 at 8:30 AM Caleb Rackliffe <calebrackliffe@gmail.com <ma...@gmail.com>> wrote:
>>>>>> I just remembered the other day that I had done a quick writeup on the state of compaction stress-related throttling in the project:
>>>>>> 
>>>>>> https://docs.google.com/document/d/1dfTEcKVidRKC1EWu3SO1kE1iVLMdaJ9uY1WMpS3P_hs/edit?usp=sharing
>>>>>> 
>>>>>> I'm sure most of it is old news to the people on this thread, but I figured I'd post it just in case :)
>>>>>> 
>>>>>> On Tue, Jan 30, 2024 at 11:58 AM Josh McKenzie <jmckenzie@apache.org <ma...@apache.org>> wrote:
>>>>>> 
>>>>>>> 2.) We should make sure the links between the "known" root causes of cascading failures and the mechanisms we introduce to avoid them remain very strong.
>>>>>> Seems to me that our historical strategy was to address individual known cases one-by-one rather than looking for a more holistic load-balancing and load-shedding solution. While the engineer in me likes the elegance of a broad, more-inclusive actual SEDA-like approach, the pragmatist in me wonders how far we think we are today from a stable set-point.
>>>>>> 
>>>>>> i.e. are we facing a handful of cases where nodes can still get pushed over and then cascade that we can surgically address, or are we facing a broader lack of back-pressure that rears its head in different domains (client -> coordinator, coordinator -> replica, internode with other operations, etc) at surprising times and should be considered more holistically?
>>>>>> 
>>>>>> On Tue, Jan 30, 2024, at 12:31 AM, Caleb Rackliffe wrote:
>>>>>>> I almost forgot CASSANDRA-15817, which introduced reject_repair_compaction_threshold, which provides a mechanism to stop repairs while compaction is underwater.
>>>>>>> 
>>>>>>>> On Jan 26, 2024, at 6:22 PM, Caleb Rackliffe <calebrackliffe@gmail.com <ma...@gmail.com>> wrote:
>>>>>>>> 
>>>>>>>> Hey all,
>>>>>>>> 
>>>>>>>> I'm a bit late to the discussion. I see that we've already discussed CASSANDRA-15013 <https://issues.apache.org/jira/browse/CASSANDRA-15013> and CASSANDRA-16663 <https://issues.apache.org/jira/browse/CASSANDRA-16663> at least in passing. Having written the latter, I'd be the first to admit it's a crude tool, although it's been useful here and there, and provides a couple primitives that may be useful for future work. As Scott mentions, while it is configurable at runtime, it is not adaptive, although we did make configuration easier in CASSANDRA-17423 <https://issues.apache.org/jira/browse/CASSANDRA-17423>. It also is global to the node, although we've lightly discussed some ideas around making it more granular. (For example, keyspace-based limiting, or limiting "domains" tagged by the client in requests, could be interesting.) It also does not deal with inter-node traffic, of course.
>>>>>>>> 
>>>>>>>> Something we've not yet mentioned (that does address internode traffic) is CASSANDRA-17324 <https://issues.apache.org/jira/browse/CASSANDRA-17324>, which I proposed shortly after working on the native request limiter (and have just not had much time to return to). The basic idea is this:
>>>>>>>> 
>>>>>>>> When a node is struggling under the weight of a compaction backlog and becomes a cause of increased read latency for clients, we have two safety valves:
>>>>>>>> 
>>>>>>>> 1.) Disabling the native protocol server, which stops the node from coordinating reads and writes.
>>>>>>>> 2.) Jacking up the severity on the node, which tells the dynamic snitch to avoid the node for reads from other coordinators.
>>>>>>>> 
>>>>>>>> 
>>>>>>>> These are useful, but we don’t appear to have any mechanism that would allow us to temporarily reject internode hint, batch, and mutation messages that could further delay resolution of the compaction backlog.
>>>>>>>> 
>>>>>>>> Whether it's done as part of a larger framework or on its own, it still feels like a good idea.
>>>>>>>> 
>>>>>>>> Thinking in terms of opportunity costs here (i.e. where we spend our finite engineering time to holistically improve the experience of operating this database) is healthy, but we probably haven't reached the point of diminishing returns on nodes being able to protect themselves from clients and from other nodes. I would just keep in mind two things:
>>>>>>>> 
>>>>>>>> 1.) The effectiveness of rate-limiting in the system (which includes the database and all clients) as a whole necessarily decreases as we move from the application to the lowest-level database internals. Limiting correctly at the client will save more resources than limiting at the native protocol server, and limiting correctly at the native protocol server will save more resources than limiting after we've dispatched requests to some thread pool for processing.
>>>>>>>> 2.) We should make sure the links between the "known" root causes of cascading failures and the mechanisms we introduce to avoid them remain very strong.
>>>>>>>> 
>>>>>>>> In any case, I'd be happy to help out in any way I can as this moves forward (especially as it relates to our past/current attempts to address this problem space).

Re: [EXTERNAL] [Discuss] Generic Purpose Rate Limiter in Cassandra

[Štefan Miklošovič <sm...@apache.org>]

I know it is probably too soon to discuss the implementation details in
depth as it is hard to say precisely how it will look like but I want to
highlight for example this (1). Would some parts of that work touch this
logic?

There is also (2) which tries to solve different but somewhat related
problems. If there is an attacker who just brute-forces passwords, we do
not only want to limit the password hashing, we want to just ban such users
for some time to try again. I do not want to go into details, it is
complicated enough already, I just want to say that this stuff should not
be covered by this work. Auth requests should never be throttled by this
mechanism and we implement a specific rate limiter with additional logic
involved which would solve brute-forces / banning of users for a while for
trying again. We would ideally never submit such a query to a respective
thread pool for further processing if we realize that a particular user is
actually banned for logging because he was trying too often and
unsuccessfully which is rather suspicious.

(1)
https://github.com/apache/cassandra/commit/09b282d1fdd7d6d62542137003011d144c0227be
(2) https://issues.apache.org/jira/browse/CASSANDRA-19734

On Mon, Sep 23, 2024 at 9:41 AM Alex Petrov <al...@coffeenco.de> wrote:

> > Are those three sufficient to protect against a client that unexpectedly
> comes up with 100x a previous provisioned-for workload? Or 100 clients at
> 100x concurrently? Given that can be 100x in terms of quantity (helped by
> queueing and shedding), but also 100x in terms of *computational and disk
> implications*? We don't have control over what users do, and while in an
> ideal world client applications have fairly predictable workloads over
> time, in practice things spike around various events for different
> applications.
>
> To fully answer your question I would probably have to flesh out a CEP
> (which I hope to get to ASAP), and I will elaborate on all of the points on
> Community over Code this year (and will do my best to put it in writing for
> the ones who will not attend). But to briefly answer, one of the ideas is
> exactly the fact that one thing in a queue is not the same as a another
> thing in a queue.
>
> > I see these 2 things as complementary, not as interdependent. Is there
> something I'm missing?
>
> I think if we start working on rate limiting before we implement good load
> balancing, we are risking of shedding load that could otherwise have been
> handled by the cluster. I think you even said it yourself with "over time
> it would raise the ceiling at which rate limiting kicked in".
>
> Besides, in CASSANDRA-19534
> <https://issues.apache.org/jira/browse/CASSANDRA-19534> I was attempting
> to show that we need to find a maximal natural possible throughput that the
> cluster can handle without tipping, and maintain it. And the easiest way to
> handle this was, naturally, through the user-set timeouts. We can work in
> resource limits, but as of now I see only marginal improvement over what we
> can do just with timeouts.
>
> One of the risk of a misconfigured rate-limiter is avoidable throttling /
> shedding. For example, CEP mentions a trigger of a 80% CPU utilization. But
> while testing 19534, we have seen that we can easily burst into high CPU
> utilization, let the requests that do not satisfy client timeout boundaries
> get shedded, and continue operating without any additional rate-limiting.
> My guess is that the existing rate-limiter sees less usage than we wish it
> to primarily because it is hard to say where to set the limits to for
> throwing OverloadedException, and TCP throttling/backoff does not work
> because we lose the queue arrival timestamp and start triggering client
> timeouts that are invisible to a server (i.e. client retries while request
> is still in the queue). While latter problem has a trivial solution (i.e.
> client-set deadlines), former one probably requires some auto-tuning or
> guidance.
>
> Another example is "culprit Keyspaces" from the CEP. If we introduce
> fairness into our load-balancing, a single keyspace, or a replica set
> (partition) will not be able to dominate the cluster workload, causing
> accross the board timeouts. Which means that by simply giving preference to
> other requests we have naturally shed an imbalance without introducing any
> rate-limiting.
>
> Maybe the problem is in the terminology, but I think we should choose
> "prioritize read/write coordinator workload" over "block any {read/write}
> {coordinator/replication} traffic for a table". Let me make an example.
> Let's say we implement some algorithm for replenishing request allowance
> for write replication for a table, and this table runs out of tokens. If we
> make a decision to shed this request before waiting until the last moment
> it potentially could have gotten processed, we are risking shedding a
> request that can be served. But if we gave such request a lower priority,
> we can get to it when we get to it given current resources and queues. If
> we still can process it, we will, even if it is on a very end of our
> timeout guarantee, and I think it does not need to be shed.
>
> I think what I have in mind seems to jibe very well with the points you
> have brought up:
>   * How can nodes protect themselves from variable and changing user
> behavior in a way that's minimally disruptive to the user and requires as
> little configuration as possible for operators
>   * How do we keep the limits of node performance from leaking into the
> scope of user awareness and responsibility outside simply pushing various
> exceptions to the client to indicate what's going on
>
> To summarize, I think good load balancing and workload prioritization
> combined with just "give up on a request if we know for a fact we can not
> process it" feels like a simpler way to solve the two problems you
> mentioned, and it will help us maximize cluster utilization while not
> shedding the load that could otherwise have been served, _while_ also
> reducing latencies across the board.
>
>
> On Sat, Sep 21, 2024, at 1:35 PM, Josh McKenzie wrote:
>
> Are those three sufficient to protect against a client that unexpectedly
> comes up with 100x a previous provisioned-for workload? Or 100 clients at
> 100x concurrently? Given that can be 100x in terms of quantity (helped by
> queueing and shedding), but also 100x in terms of *computational and disk
> implications*? We don't have control over what users do, and while in an
> ideal world client applications have fairly predictable workloads over
> time, in practice things spike around various events for different
> applications.
>
> i.e. 1 thing in a queue could produce orders of magnitude more work than 1
> other thing in a queue. Especially as we move into a world with SAI and
> Accord.
>
> we need to solve load balancing (summarized by the above three points)
> before we start working on the rate limiter
>
> I see these 2 things as complementary, not as interdependent. Is there
> something I'm missing?
>
> At least for me, mentally I frame this as "How can nodes protect
> themselves from variable and changing user behavior in a way that's
> minimally disruptive to the user and requires as little configuration as
> possible for operators?". Basically, how do we keep the limits of node
> performance from leaking into the scope of user awareness and
> responsibility outside simply pushing various exceptions to the client to
> indicate what's going on (OverloadedException to the client, etc).
>
> It seems to me both rate limiting and resource balancing are integral
> parts of this, but also parts that could be worked on independently. Were
> we to wave a magic wand tomorrow and have robust rate limiting, as we
> improved load balancing over time it would raise the ceiling at which rate
> limiting kicked in.
>
> So concretely to the thread, I think I agree with Jon:
>
> * use the rate of timeouts to limit the depth of the queues for each of
> the thread pools
> * reject requests when the queue is full with an OverloadedException.
>
> followed by:
>
> If you want to follow this up with the ability to dynamically resize
> thread pools that could be interesting.
>
>
> Simple is very much a feature here.
>
> On Sat, Sep 21, 2024, at 5:20 AM, Alex Petrov wrote:
>
> > Personally, I’m a bit skeptical that we will come up with a metric based
> heuristic that works well in most scenarios and doesn’t require significant
> knowledge and tuning. I think past implementations of the dynamic snitch
> are good evidence of that.
>
> I am more optimistic on that font. I think we can achieve a lot. However,
> in my opinion, we need to focus on balancing the load rather than rate
> limiting. Rate limiting is going to be important if/when we decide to
> implement workload isolation. Until then, I think we should focus on three
> things:
>
>   * Node health (Nodes should produce useful work and should be stable and
> not overloaded)
>   * Latency (we always need to find an optimal way to process request and minimize
> overall queueing time)
>   * Fairness (avoid workload and utilization imbalances)
>
> All three points are achievable with very straightforward approaches that
> will not require much operator involvement.
>
> I guess my main point is we need to solve load balancing (summarized by
> the above three points) before we start working on the rate limiter, but
> there's a good chance we may not need one apart from use cases that require
> workload isolation.
>
>
> On Fri, Sep 20, 2024, at 8:14 PM, Jordan West wrote:
>
> +1 to Benedict’s (and others) comments on plugability and low overhead
> when disabled. The latter I think needs little justification. The reason I
> am big on the former is, in my opinion: decisions on approach need to be
> settled with numbers not anecdotes or past experience (including my own).
> So I would like to see us compare different approaches (what metrics to
> use, etc).
>
> Personally, I’m a bit skeptical that we will come up with a metric based
> heuristic that works well in most scenarios and doesn’t require significant
> knowledge and tuning. I think past implementations of the dynamic snitch
> are good evidence of that. However, I expressed the same concerns
> internally for a client level project where we exposed metrics to induce
> back pressure and early experiments are encouraging / contrary to my
> expectations. At different layers different approaches can work better or
> worse. Same with different workloads. I don’t think we should dismiss
> approaches out right in this thread without hard numbers.
>
> In short, I think the testing and evaluation of this CEP is as important
> as its design and implementation. We will need to test a wide variety of
> workloads and potentially implementations and that’s where pluggability
> will be a huge benefit. I would go as far as saying the CEP should focus
> more on a framework for pluggable implementations that has low to zero cost
> when disabled than a specific set of metrics to use or specific approach.
>
> Jordan
>
> On Thu, Sep 19, 2024 at 14:38 Benedict Elliott Smith <be...@apache.org>
> wrote:
>
> I just want to flag here that this is a topic I have strong opinions on,
> but the CEP is not really specific or detailed enough to understand
> precisely how it will be implemented. So, if a patch is already being
> produced, most of my feedback is likely to be provided some time after a
> patch appears, through the normal review process. I want to flag this now
> to avoid any surprise.
>
> I will say that upfront that, ideally, this system should be designed to
> have ~zero overhead when disabled, and with minimal coupling (between its
> own components and C* itself), so that entirely orthogonal approaches can
> be integrated in future without polluting the codebase.
>
>
> On 19 Sep 2024, at 19:14, Patrick McFadin <pm...@gmail.com> wrote:
>
> The work has begun but we don't have a VOTE thread for this CEP. Can one
> get started?
>
> On Mon, May 6, 2024 at 9:24 PM Jaydeep Chovatia <
> chovatia.jaydeep@gmail.com> wrote:
>
> Sure, Caleb. I will include the work as part of CASSANDRA-19534
> <https://issues.apache.org/jira/browse/CASSANDRA-19534> in the CEP-41.
>
> Jaydeep
>
> On Fri, May 3, 2024 at 7:48 AM Caleb Rackliffe <ca...@gmail.com>
> wrote:
>
> FYI, there is some ongoing sort-of-related work going on in
> CASSANDRA-19534 <https://issues.apache.org/jira/browse/CASSANDRA-19534>
>
> On Wed, Apr 10, 2024 at 6:35 PM Jaydeep Chovatia <
> chovatia.jaydeep@gmail.com> wrote:
>
> Just created an official CEP-41
> <https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-41+%28DRAFT%29+Apache+Cassandra+Unified+Rate+Limiter>
> incorporating the feedback from this discussion. Feel free to let me know
> if I may have missed some important feedback in this thread that is not
> captured in the CEP-41.
>
> Jaydeep
>
> On Thu, Feb 22, 2024 at 11:36 AM Jaydeep Chovatia <
> chovatia.jaydeep@gmail.com> wrote:
>
> Thanks, Josh. I will file an official CEP with all the details in a few
> days and update this thread with that CEP number.
> Thanks a lot everyone for providing valuable insights!
>
> Jaydeep
>
> On Thu, Feb 22, 2024 at 9:24 AM Josh McKenzie <jm...@apache.org>
> wrote:
>
>
> Do folks think we should file an official CEP and take it there?
>
> +1 here.
>
> Synthesizing your gdoc, Caleb's work, and the feedback from this thread
> into a draft seems like a solid next step.
>
> On Wed, Feb 7, 2024, at 12:31 PM, Jaydeep Chovatia wrote:
>
> I see a lot of great ideas being discussed or proposed in the past to
> cover the most common rate limiter candidate use cases. Do folks think we
> should file an official CEP and take it there?
>
> Jaydeep
>
> On Fri, Feb 2, 2024 at 8:30 AM Caleb Rackliffe <ca...@gmail.com>
> wrote:
>
> I just remembered the other day that I had done a quick writeup on the
> state of compaction stress-related throttling in the project:
>
>
> https://docs.google.com/document/d/1dfTEcKVidRKC1EWu3SO1kE1iVLMdaJ9uY1WMpS3P_hs/edit?usp=sharing
>
> I'm sure most of it is old news to the people on this thread, but I
> figured I'd post it just in case :)
>
> On Tue, Jan 30, 2024 at 11:58 AM Josh McKenzie <jm...@apache.org>
> wrote:
>
>
> 2.) We should make sure the links between the "known" root causes of
> cascading failures and the mechanisms we introduce to avoid them remain
> very strong.
>
> Seems to me that our historical strategy was to address individual known
> cases one-by-one rather than looking for a more holistic load-balancing and
> load-shedding solution. While the engineer in me likes the elegance of a
> broad, more-inclusive *actual SEDA-like* approach, the pragmatist in me
> wonders how far we think we are today from a stable set-point.
>
> i.e. are we facing a handful of cases where nodes can still get pushed
> over and then cascade that we can surgically address, or are we facing a
> broader lack of back-pressure that rears its head in different domains
> (client -> coordinator, coordinator -> replica, internode with other
> operations, etc) at surprising times and should be considered more
> holistically?
>
> On Tue, Jan 30, 2024, at 12:31 AM, Caleb Rackliffe wrote:
>
> I almost forgot CASSANDRA-15817, which introduced
> reject_repair_compaction_threshold, which provides a mechanism to stop
> repairs while compaction is underwater.
>
> On Jan 26, 2024, at 6:22 PM, Caleb Rackliffe <ca...@gmail.com>
> wrote:
>
> 
> Hey all,
>
> I'm a bit late to the discussion. I see that we've already discussed
> CASSANDRA-15013 <https://issues.apache.org/jira/browse/CASSANDRA-15013>
>  and CASSANDRA-16663
> <https://issues.apache.org/jira/browse/CASSANDRA-16663> at least in
> passing. Having written the latter, I'd be the first to admit it's a crude
> tool, although it's been useful here and there, and provides a couple
> primitives that may be useful for future work. As Scott mentions, while it
> is configurable at runtime, it is not adaptive, although we did
> make configuration easier in CASSANDRA-17423
> <https://issues.apache.org/jira/browse/CASSANDRA-17423>. It also is
> global to the node, although we've lightly discussed some ideas around
> making it more granular. (For example, keyspace-based limiting, or limiting
> "domains" tagged by the client in requests, could be interesting.) It also
> does not deal with inter-node traffic, of course.
>
> Something we've not yet mentioned (that does address internode traffic) is
> CASSANDRA-17324 <https://issues.apache.org/jira/browse/CASSANDRA-17324>,
> which I proposed shortly after working on the native request limiter (and
> have just not had much time to return to). The basic idea is this:
>
> When a node is struggling under the weight of a compaction backlog and
> becomes a cause of increased read latency for clients, we have two safety
> valves:
>
>
> 1.) Disabling the native protocol server, which stops the node from
> coordinating reads and writes.
> 2.) Jacking up the severity on the node, which tells the dynamic snitch to
> avoid the node for reads from other coordinators.
>
>
> These are useful, but we don’t appear to have any mechanism that would
> allow us to temporarily reject internode hint, batch, and mutation messages
> that could further delay resolution of the compaction backlog.
>
>
> Whether it's done as part of a larger framework or on its own, it still
> feels like a good idea.
>
> Thinking in terms of opportunity costs here (i.e. where we spend our
> finite engineering time to holistically improve the experience of operating
> this database) is healthy, but we probably haven't reached the point of
> diminishing returns on nodes being able to protect themselves from clients
> and from other nodes. I would just keep in mind two things:
>
> 1.) The effectiveness of rate-limiting in the system (which includes the
> database and all clients) as a whole necessarily decreases as we move from
> the application to the lowest-level database internals. Limiting correctly
> at the client will save more resources than limiting at the native protocol
> server, and limiting correctly at the native protocol server will save more
> resources than limiting after we've dispatched requests to some thread pool
> for processing.
> 2.) We should make sure the links between the "known" root causes of
> cascading failures and the mechanisms we introduce to avoid them remain
> very strong.
>
> In any case, I'd be happy to help out in any way I can as this moves
> forward (especially as it relates to our past/current attempts to address
> this problem space).
>
>
>
>
>
>
>

Re: [EXTERNAL] [Discuss] Generic Purpose Rate Limiter in Cassandra

[Alex Petrov <al...@coffeenco.de>]

> Are those three sufficient to protect against a client that unexpectedly comes up with 100x a previous provisioned-for workload? Or 100 clients at 100x concurrently? Given that can be 100x in terms of quantity (helped by queueing and shedding), but also 100x in terms of *computational and disk implications*? We don't have control over what users do, and while in an ideal world client applications have fairly predictable workloads over time, in practice things spike around various events for different applications.

To fully answer your question I would probably have to flesh out a CEP (which I hope to get to ASAP), and I will elaborate on all of the points on Community over Code this year (and will do my best to put it in writing for the ones who will not attend). But to briefly answer, one of the ideas is exactly the fact that one thing in a queue is not the same as a another thing in a queue. 

> I see these 2 things as complementary, not as interdependent. Is there something I'm missing?

I think if we start working on rate limiting before we implement good load balancing, we are risking of shedding load that could otherwise have been handled by the cluster. I think you even said it yourself with "over time it would raise the ceiling at which rate limiting kicked in".

Besides, in CASSANDRA-19534 <https://issues.apache.org/jira/browse/CASSANDRA-19534> I was attempting to show that we need to find a maximal natural possible throughput that the cluster can handle without tipping, and maintain it. And the easiest way to handle this was, naturally, through the user-set timeouts. We can work in resource limits, but as of now I see only marginal improvement over what we can do just with timeouts.

One of the risk of a misconfigured rate-limiter is avoidable throttling / shedding. For example, CEP mentions a trigger of a 80% CPU utilization. But while testing 19534, we have seen that we can easily burst into high CPU utilization, let the requests that do not satisfy client timeout boundaries get shedded, and continue operating without any additional rate-limiting. My guess is that the existing rate-limiter sees less usage than we wish it to primarily because it is hard to say where to set the limits to for throwing OverloadedException, and TCP throttling/backoff does not work because we lose the queue arrival timestamp and start triggering client timeouts that are invisible to a server (i.e. client retries while request is still in the queue). While latter problem has a trivial solution (i.e. client-set deadlines), former one probably requires some auto-tuning or guidance. 

Another example is "culprit Keyspaces" from the CEP. If we introduce fairness into our load-balancing, a single keyspace, or a replica set (partition) will not be able to dominate the cluster workload, causing accross the board timeouts. Which means that by simply giving preference to other requests we have naturally shed an imbalance without introducing any rate-limiting.

Maybe the problem is in the terminology, but I think we should choose "prioritize read/write coordinator workload" over "block any {read/write} {coordinator/replication} traffic for a table". Let me make an example. Let's say we implement some algorithm for replenishing request allowance for write replication for a table, and this table runs out of tokens. If we make a decision to shed this request before waiting until the last moment it potentially could have gotten processed, we are risking shedding a request that can be served. But if we gave such request a lower priority, we can get to it when we get to it given current resources and queues. If we still can process it, we will, even if it is on a very end of our timeout guarantee, and I think it does not need to be shed.

I think what I have in mind seems to jibe very well with the points you have brought up:
  * How can nodes protect themselves from variable and changing user behavior in a way that's minimally disruptive to the user and requires as little configuration as possible for operators
  * How do we keep the limits of node performance from leaking into the scope of user awareness and responsibility outside simply pushing various exceptions to the client to indicate what's going on 

To summarize, I think good load balancing and workload prioritization combined with just "give up on a request if we know for a fact we can not process it" feels like a simpler way to solve the two problems you mentioned, and it will help us maximize cluster utilization while not shedding the load that could otherwise have been served, _while_ also reducing latencies across the board.


On Sat, Sep 21, 2024, at 1:35 PM, Josh McKenzie wrote:
> Are those three sufficient to protect against a client that unexpectedly comes up with 100x a previous provisioned-for workload? Or 100 clients at 100x concurrently? Given that can be 100x in terms of quantity (helped by queueing and shedding), but also 100x in terms of *computational and disk implications*? We don't have control over what users do, and while in an ideal world client applications have fairly predictable workloads over time, in practice things spike around various events for different applications.
> 
> i.e. 1 thing in a queue could produce orders of magnitude more work than 1 other thing in a queue. Especially as we move into a world with SAI and Accord.
> 
>> we need to solve load balancing (summarized by the above three points) before we start working on the rate limiter
> I see these 2 things as complementary, not as interdependent. Is there something I'm missing?
> 
> At least for me, mentally I frame this as "How can nodes protect themselves from variable and changing user behavior in a way that's minimally disruptive to the user and requires as little configuration as possible for operators?". Basically, how do we keep the limits of node performance from leaking into the scope of user awareness and responsibility outside simply pushing various exceptions to the client to indicate what's going on (OverloadedException to the client, etc).
> 
> It seems to me both rate limiting and resource balancing are integral parts of this, but also parts that could be worked on independently. Were we to wave a magic wand tomorrow and have robust rate limiting, as we improved load balancing over time it would raise the ceiling at which rate limiting kicked in.
> 
> So concretely to the thread, I think I agree with Jon:
>> * use the rate of timeouts to limit the depth of the queues for each of the thread pools
>> * reject requests when the queue is full with an OverloadedException.
> followed by:
>> If you want to follow this up with the ability to dynamically resize thread pools that could be interesting.
> 
> Simple is very much a feature here.
> 
> On Sat, Sep 21, 2024, at 5:20 AM, Alex Petrov wrote:
>> > Personally, I’m a bit skeptical that we will come up with a metric based heuristic that works well in most scenarios and doesn’t require significant knowledge and tuning. I think past implementations of the dynamic snitch are good evidence of that.
>> 
>> I am more optimistic on that font. I think we can achieve a lot. However, in my opinion, we need to focus on balancing the load rather than rate limiting. Rate limiting is going to be important if/when we decide to implement workload isolation. Until then, I think we should focus on three things:
>> 
>>   * Node health (Nodes should produce useful work and should be stable and not overloaded)
>>   * Latency (we always need to find an optimal way to process request and minimize overall queueing time)
>>   * Fairness (avoid workload and utilization imbalances)
>> 
>> All three points are achievable with very straightforward approaches that will not require much operator involvement.
>> 
>> I guess my main point is we need to solve load balancing (summarized by the above three points) before we start working on the rate limiter, but there's a good chance we may not need one apart from use cases that require workload isolation. 
>> 
>> 
>> On Fri, Sep 20, 2024, at 8:14 PM, Jordan West wrote:
>>> +1 to Benedict’s (and others) comments on plugability and low overhead when disabled. The latter I think needs little justification. The reason I am big on the former is, in my opinion: decisions on approach need to be settled with numbers not anecdotes or past experience (including my own). So I would like to see us compare different approaches (what metrics to use, etc). 
>>> 
>>> Personally, I’m a bit skeptical that we will come up with a metric based heuristic that works well in most scenarios and doesn’t require significant knowledge and tuning. I think past implementations of the dynamic snitch are good evidence of that. However, I expressed the same concerns internally for a client level project where we exposed metrics to induce back pressure and early experiments are encouraging / contrary to my expectations. At different layers different approaches can work better or worse. Same with different workloads. I don’t think we should dismiss approaches out right in this thread without hard numbers. 
>>> 
>>> In short, I think the testing and evaluation of this CEP is as important as its design and implementation. We will need to test a wide variety of workloads and potentially implementations and that’s where pluggability will be a huge benefit. I would go as far as saying the CEP should focus more on a framework for pluggable implementations that has low to zero cost when disabled than a specific set of metrics to use or specific approach. 
>>> 
>>> Jordan 
>>> 
>>> On Thu, Sep 19, 2024 at 14:38 Benedict Elliott Smith <be...@apache.org> wrote:
>>>> I just want to flag here that this is a topic I have strong opinions on, but the CEP is not really specific or detailed enough to understand precisely how it will be implemented. So, if a patch is already being produced, most of my feedback is likely to be provided some time after a patch appears, through the normal review process. I want to flag this now to avoid any surprise.
>>>> 
>>>> I will say that upfront that, ideally, this system should be designed to have ~zero overhead when disabled, and with minimal coupling (between its own components and C* itself), so that entirely orthogonal approaches can be integrated in future without polluting the codebase.
>>>> 
>>>> 
>>>>> On 19 Sep 2024, at 19:14, Patrick McFadin <pm...@gmail.com> wrote:
>>>>> 
>>>>> The work has begun but we don't have a VOTE thread for this CEP. Can one get started?
>>>>> 
>>>>> On Mon, May 6, 2024 at 9:24 PM Jaydeep Chovatia <ch...@gmail.com> wrote:
>>>>>> Sure, Caleb. I will include the work as part of CASSANDRA-19534 <https://issues.apache.org/jira/browse/CASSANDRA-19534> in the CEP-41.
>>>>>> 
>>>>>> Jaydeep
>>>>>> 
>>>>>> On Fri, May 3, 2024 at 7:48 AM Caleb Rackliffe <ca...@gmail.com> wrote:
>>>>>>> FYI, there is some ongoing sort-of-related work going on in CASSANDRA-19534 <https://issues.apache.org/jira/browse/CASSANDRA-19534>
>>>>>>> 
>>>>>>> On Wed, Apr 10, 2024 at 6:35 PM Jaydeep Chovatia <ch...@gmail.com> wrote:
>>>>>>>> Just created an official CEP-41 <https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-41+%28DRAFT%29+Apache+Cassandra+Unified+Rate+Limiter> incorporating the feedback from this discussion. Feel free to let me know if I may have missed some important feedback in this thread that is not captured in the CEP-41.
>>>>>>>> 
>>>>>>>> Jaydeep
>>>>>>>> 
>>>>>>>> On Thu, Feb 22, 2024 at 11:36 AM Jaydeep Chovatia <ch...@gmail.com> wrote:
>>>>>>>>> Thanks, Josh. I will file an official CEP with all the details in a few days and update this thread with that CEP number.
>>>>>>>>> Thanks a lot everyone for providing valuable insights!
>>>>>>>>> 
>>>>>>>>> Jaydeep
>>>>>>>>> 
>>>>>>>>> On Thu, Feb 22, 2024 at 9:24 AM Josh McKenzie <jm...@apache.org> wrote:
>>>>>>>>>> __
>>>>>>>>>>> Do folks think we should file an official CEP and take it there?
>>>>>>>>>> +1 here.
>>>>>>>>>> 
>>>>>>>>>> Synthesizing your gdoc, Caleb's work, and the feedback from this thread into a draft seems like a solid next step.
>>>>>>>>>> 
>>>>>>>>>> On Wed, Feb 7, 2024, at 12:31 PM, Jaydeep Chovatia wrote:
>>>>>>>>>>> I see a lot of great ideas being discussed or proposed in the past to cover the most common rate limiter candidate use cases. Do folks think we should file an official CEP and take it there?
>>>>>>>>>>> 
>>>>>>>>>>> Jaydeep
>>>>>>>>>>> 
>>>>>>>>>>> On Fri, Feb 2, 2024 at 8:30 AM Caleb Rackliffe <ca...@gmail.com> wrote:
>>>>>>>>>>>> I just remembered the other day that I had done a quick writeup on the state of compaction stress-related throttling in the project:
>>>>>>>>>>>> 
>>>>>>>>>>>> https://docs.google.com/document/d/1dfTEcKVidRKC1EWu3SO1kE1iVLMdaJ9uY1WMpS3P_hs/edit?usp=sharing
>>>>>>>>>>>> 
>>>>>>>>>>>> I'm sure most of it is old news to the people on this thread, but I figured I'd post it just in case :)
>>>>>>>>>>>> 
>>>>>>>>>>>> On Tue, Jan 30, 2024 at 11:58 AM Josh McKenzie <jm...@apache.org> wrote:
>>>>>>>>>>>>> __
>>>>>>>>>>>>>> 2.) We should make sure the links between the "known" root causes of cascading failures and the mechanisms we introduce to avoid them remain very strong.
>>>>>>>>>>>>> Seems to me that our historical strategy was to address individual known cases one-by-one rather than looking for a more holistic load-balancing and load-shedding solution. While the engineer in me likes the elegance of a broad, more-inclusive *actual SEDA-like* approach, the pragmatist in me wonders how far we think we are today from a stable set-point.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> i.e. are we facing a handful of cases where nodes can still get pushed over and then cascade that we can surgically address, or are we facing a broader lack of back-pressure that rears its head in different domains (client -> coordinator, coordinator -> replica, internode with other operations, etc) at surprising times and should be considered more holistically?
>>>>>>>>>>>>> 
>>>>>>>>>>>>> On Tue, Jan 30, 2024, at 12:31 AM, Caleb Rackliffe wrote:
>>>>>>>>>>>>>> I almost forgot CASSANDRA-15817, which introduced reject_repair_compaction_threshold, which provides a mechanism to stop repairs while compaction is underwater.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> On Jan 26, 2024, at 6:22 PM, Caleb Rackliffe <ca...@gmail.com> wrote:
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Hey all,
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> I'm a bit late to the discussion. I see that we've already discussed CASSANDRA-15013 <https://issues.apache.org/jira/browse/CASSANDRA-15013> and CASSANDRA-16663 <https://issues.apache.org/jira/browse/CASSANDRA-16663> at least in passing. Having written the latter, I'd be the first to admit it's a crude tool, although it's been useful here and there, and provides a couple primitives that may be useful for future work. As Scott mentions, while it is configurable at runtime, it is not adaptive, although we did make configuration easier in CASSANDRA-17423 <https://issues.apache.org/jira/browse/CASSANDRA-17423>. It also is global to the node, although we've lightly discussed some ideas around making it more granular. (For example, keyspace-based limiting, or limiting "domains" tagged by the client in requests, could be interesting.) It also does not deal with inter-node traffic, of course.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Something we've not yet mentioned (that does address internode traffic) is CASSANDRA-17324 <https://issues.apache.org/jira/browse/CASSANDRA-17324>, which I proposed shortly after working on the native request limiter (and have just not had much time to return to). The basic idea is this:
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> When a node is struggling under the weight of a compaction backlog and becomes a cause of increased read latency for clients, we have two safety valves:
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 1.) Disabling the native protocol server, which stops the node from coordinating reads and writes.
>>>>>>>>>>>>>>>> 2.) Jacking up the severity on the node, which tells the dynamic snitch to avoid the node for reads from other coordinators.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> These are useful, but we don’t appear to have any mechanism that would allow us to temporarily reject internode hint, batch, and mutation messages that could further delay resolution of the compaction backlog.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Whether it's done as part of a larger framework or on its own, it still feels like a good idea.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Thinking in terms of opportunity costs here (i.e. where we spend our finite engineering time to holistically improve the experience of operating this database) is healthy, but we probably haven't reached the point of diminishing returns on nodes being able to protect themselves from clients and from other nodes. I would just keep in mind two things:
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 1.) The effectiveness of rate-limiting in the system (which includes the database and all clients) as a whole necessarily decreases as we move from the application to the lowest-level database internals. Limiting correctly at the client will save more resources than limiting at the native protocol server, and limiting correctly at the native protocol server will save more resources than limiting after we've dispatched requests to some thread pool for processing.
>>>>>>>>>>>>>>> 2.) We should make sure the links between the "known" root causes of cascading failures and the mechanisms we introduce to avoid them remain very strong.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> In any case, I'd be happy to help out in any way I can as this moves forward (especially as it relates to our past/current attempts to address this problem space).
>>>>>>>>>>>>> 
>>>>>>>>>> 
>> 
> 

Re: [EXTERNAL] [Discuss] Generic Purpose Rate Limiter in Cassandra

[Josh McKenzie <jm...@apache.org>]

Are those three sufficient to protect against a client that unexpectedly comes up with 100x a previous provisioned-for workload? Or 100 clients at 100x concurrently? Given that can be 100x in terms of quantity (helped by queueing and shedding), but also 100x in terms of *computational and disk implications*? We don't have control over what users do, and while in an ideal world client applications have fairly predictable workloads over time, in practice things spike around various events for different applications.

i.e. 1 thing in a queue could produce orders of magnitude more work than 1 other thing in a queue. Especially as we move into a world with SAI and Accord.

> we need to solve load balancing (summarized by the above three points) before we start working on the rate limiter
I see these 2 things as complementary, not as interdependent. Is there something I'm missing?

At least for me, mentally I frame this as "How can nodes protect themselves from variable and changing user behavior in a way that's minimally disruptive to the user and requires as little configuration as possible for operators?". Basically, how do we keep the limits of node performance from leaking into the scope of user awareness and responsibility outside simply pushing various exceptions to the client to indicate what's going on (OverloadedException to the client, etc).

It seems to me both rate limiting and resource balancing are integral parts of this, but also parts that could be worked on independently. Were we to wave a magic wand tomorrow and have robust rate limiting, as we improved load balancing over time it would raise the ceiling at which rate limiting kicked in.

So concretely to the thread, I think I agree with Jon:
> * use the rate of timeouts to limit the depth of the queues for each of the thread pools
> * reject requests when the queue is full with an OverloadedException.
followed by:
> If you want to follow this up with the ability to dynamically resize thread pools that could be interesting.

Simple is very much a feature here.

On Sat, Sep 21, 2024, at 5:20 AM, Alex Petrov wrote:
> > Personally, I’m a bit skeptical that we will come up with a metric based heuristic that works well in most scenarios and doesn’t require significant knowledge and tuning. I think past implementations of the dynamic snitch are good evidence of that.
> 
> I am more optimistic on that font. I think we can achieve a lot. However, in my opinion, we need to focus on balancing the load rather than rate limiting. Rate limiting is going to be important if/when we decide to implement workload isolation. Until then, I think we should focus on three things:
> 
>   * Node health (Nodes should produce useful work and should be stable and not overloaded)
>   * Latency (we always need to find an optimal way to process request and minimize overall queueing time)
>   * Fairness (avoid workload and utilization imbalances)
> 
> All three points are achievable with very straightforward approaches that will not require much operator involvement.
> 
> I guess my main point is we need to solve load balancing (summarized by the above three points) before we start working on the rate limiter, but there's a good chance we may not need one apart from use cases that require workload isolation. 
> 
> 
> On Fri, Sep 20, 2024, at 8:14 PM, Jordan West wrote:
>> +1 to Benedict’s (and others) comments on plugability and low overhead when disabled. The latter I think needs little justification. The reason I am big on the former is, in my opinion: decisions on approach need to be settled with numbers not anecdotes or past experience (including my own). So I would like to see us compare different approaches (what metrics to use, etc). 
>> 
>> Personally, I’m a bit skeptical that we will come up with a metric based heuristic that works well in most scenarios and doesn’t require significant knowledge and tuning. I think past implementations of the dynamic snitch are good evidence of that. However, I expressed the same concerns internally for a client level project where we exposed metrics to induce back pressure and early experiments are encouraging / contrary to my expectations. At different layers different approaches can work better or worse. Same with different workloads. I don’t think we should dismiss approaches out right in this thread without hard numbers. 
>> 
>> In short, I think the testing and evaluation of this CEP is as important as its design and implementation. We will need to test a wide variety of workloads and potentially implementations and that’s where pluggability will be a huge benefit. I would go as far as saying the CEP should focus more on a framework for pluggable implementations that has low to zero cost when disabled than a specific set of metrics to use or specific approach. 
>> 
>> Jordan 
>> 
>> On Thu, Sep 19, 2024 at 14:38 Benedict Elliott Smith <be...@apache.org> wrote:
>>> I just want to flag here that this is a topic I have strong opinions on, but the CEP is not really specific or detailed enough to understand precisely how it will be implemented. So, if a patch is already being produced, most of my feedback is likely to be provided some time after a patch appears, through the normal review process. I want to flag this now to avoid any surprise.
>>> 
>>> I will say that upfront that, ideally, this system should be designed to have ~zero overhead when disabled, and with minimal coupling (between its own components and C* itself), so that entirely orthogonal approaches can be integrated in future without polluting the codebase.
>>> 
>>> 
>>>> On 19 Sep 2024, at 19:14, Patrick McFadin <pm...@gmail.com> wrote:
>>>> 
>>>> The work has begun but we don't have a VOTE thread for this CEP. Can one get started?
>>>> 
>>>> On Mon, May 6, 2024 at 9:24 PM Jaydeep Chovatia <ch...@gmail.com> wrote:
>>>>> Sure, Caleb. I will include the work as part of CASSANDRA-19534 <https://issues.apache.org/jira/browse/CASSANDRA-19534> in the CEP-41.
>>>>> 
>>>>> Jaydeep
>>>>> 
>>>>> On Fri, May 3, 2024 at 7:48 AM Caleb Rackliffe <ca...@gmail.com> wrote:
>>>>>> FYI, there is some ongoing sort-of-related work going on in CASSANDRA-19534 <https://issues.apache.org/jira/browse/CASSANDRA-19534>
>>>>>> 
>>>>>> On Wed, Apr 10, 2024 at 6:35 PM Jaydeep Chovatia <ch...@gmail.com> wrote:
>>>>>>> Just created an official CEP-41 <https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-41+%28DRAFT%29+Apache+Cassandra+Unified+Rate+Limiter> incorporating the feedback from this discussion. Feel free to let me know if I may have missed some important feedback in this thread that is not captured in the CEP-41.
>>>>>>> 
>>>>>>> Jaydeep
>>>>>>> 
>>>>>>> On Thu, Feb 22, 2024 at 11:36 AM Jaydeep Chovatia <ch...@gmail.com> wrote:
>>>>>>>> Thanks, Josh. I will file an official CEP with all the details in a few days and update this thread with that CEP number.
>>>>>>>> Thanks a lot everyone for providing valuable insights!
>>>>>>>> 
>>>>>>>> Jaydeep
>>>>>>>> 
>>>>>>>> On Thu, Feb 22, 2024 at 9:24 AM Josh McKenzie <jm...@apache.org> wrote:
>>>>>>>>> __
>>>>>>>>>> Do folks think we should file an official CEP and take it there?
>>>>>>>>> +1 here.
>>>>>>>>> 
>>>>>>>>> Synthesizing your gdoc, Caleb's work, and the feedback from this thread into a draft seems like a solid next step.
>>>>>>>>> 
>>>>>>>>> On Wed, Feb 7, 2024, at 12:31 PM, Jaydeep Chovatia wrote:
>>>>>>>>>> I see a lot of great ideas being discussed or proposed in the past to cover the most common rate limiter candidate use cases. Do folks think we should file an official CEP and take it there?
>>>>>>>>>> 
>>>>>>>>>> Jaydeep
>>>>>>>>>> 
>>>>>>>>>> On Fri, Feb 2, 2024 at 8:30 AM Caleb Rackliffe <ca...@gmail.com> wrote:
>>>>>>>>>>> I just remembered the other day that I had done a quick writeup on the state of compaction stress-related throttling in the project:
>>>>>>>>>>> 
>>>>>>>>>>> https://docs.google.com/document/d/1dfTEcKVidRKC1EWu3SO1kE1iVLMdaJ9uY1WMpS3P_hs/edit?usp=sharing
>>>>>>>>>>> 
>>>>>>>>>>> I'm sure most of it is old news to the people on this thread, but I figured I'd post it just in case :)
>>>>>>>>>>> 
>>>>>>>>>>> On Tue, Jan 30, 2024 at 11:58 AM Josh McKenzie <jm...@apache.org> wrote:
>>>>>>>>>>>> __
>>>>>>>>>>>>> 2.) We should make sure the links between the "known" root causes of cascading failures and the mechanisms we introduce to avoid them remain very strong.
>>>>>>>>>>>> Seems to me that our historical strategy was to address individual known cases one-by-one rather than looking for a more holistic load-balancing and load-shedding solution. While the engineer in me likes the elegance of a broad, more-inclusive *actual SEDA-like* approach, the pragmatist in me wonders how far we think we are today from a stable set-point.
>>>>>>>>>>>> 
>>>>>>>>>>>> i.e. are we facing a handful of cases where nodes can still get pushed over and then cascade that we can surgically address, or are we facing a broader lack of back-pressure that rears its head in different domains (client -> coordinator, coordinator -> replica, internode with other operations, etc) at surprising times and should be considered more holistically?
>>>>>>>>>>>> 
>>>>>>>>>>>> On Tue, Jan 30, 2024, at 12:31 AM, Caleb Rackliffe wrote:
>>>>>>>>>>>>> I almost forgot CASSANDRA-15817, which introduced reject_repair_compaction_threshold, which provides a mechanism to stop repairs while compaction is underwater.
>>>>>>>>>>>>> 
>>>>>>>>>>>>>> On Jan 26, 2024, at 6:22 PM, Caleb Rackliffe <ca...@gmail.com> wrote:
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Hey all,
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> I'm a bit late to the discussion. I see that we've already discussed CASSANDRA-15013 <https://issues.apache.org/jira/browse/CASSANDRA-15013> and CASSANDRA-16663 <https://issues.apache.org/jira/browse/CASSANDRA-16663> at least in passing. Having written the latter, I'd be the first to admit it's a crude tool, although it's been useful here and there, and provides a couple primitives that may be useful for future work. As Scott mentions, while it is configurable at runtime, it is not adaptive, although we did make configuration easier in CASSANDRA-17423 <https://issues.apache.org/jira/browse/CASSANDRA-17423>. It also is global to the node, although we've lightly discussed some ideas around making it more granular. (For example, keyspace-based limiting, or limiting "domains" tagged by the client in requests, could be interesting.) It also does not deal with inter-node traffic, of course.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Something we've not yet mentioned (that does address internode traffic) is CASSANDRA-17324 <https://issues.apache.org/jira/browse/CASSANDRA-17324>, which I proposed shortly after working on the native request limiter (and have just not had much time to return to). The basic idea is this:
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> When a node is struggling under the weight of a compaction backlog and becomes a cause of increased read latency for clients, we have two safety valves:
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 1.) Disabling the native protocol server, which stops the node from coordinating reads and writes.
>>>>>>>>>>>>>>> 2.) Jacking up the severity on the node, which tells the dynamic snitch to avoid the node for reads from other coordinators.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> These are useful, but we don’t appear to have any mechanism that would allow us to temporarily reject internode hint, batch, and mutation messages that could further delay resolution of the compaction backlog.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Whether it's done as part of a larger framework or on its own, it still feels like a good idea.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Thinking in terms of opportunity costs here (i.e. where we spend our finite engineering time to holistically improve the experience of operating this database) is healthy, but we probably haven't reached the point of diminishing returns on nodes being able to protect themselves from clients and from other nodes. I would just keep in mind two things:
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 1.) The effectiveness of rate-limiting in the system (which includes the database and all clients) as a whole necessarily decreases as we move from the application to the lowest-level database internals. Limiting correctly at the client will save more resources than limiting at the native protocol server, and limiting correctly at the native protocol server will save more resources than limiting after we've dispatched requests to some thread pool for processing.
>>>>>>>>>>>>>> 2.) We should make sure the links between the "known" root causes of cascading failures and the mechanisms we introduce to avoid them remain very strong.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> In any case, I'd be happy to help out in any way I can as this moves forward (especially as it relates to our past/current attempts to address this problem space).
>>>>>>>>>>>> 
>>>>>>>>> 
> 

Re: [EXTERNAL] [Discuss] Generic Purpose Rate Limiter in Cassandra

[Alex Petrov <al...@coffeenco.de>]

> Personally, I’m a bit skeptical that we will come up with a metric based heuristic that works well in most scenarios and doesn’t require significant knowledge and tuning. I think past implementations of the dynamic snitch are good evidence of that.

I am more optimistic on that font. I think we can achieve a lot. However, in my opinion, we need to focus on balancing the load rather than rate limiting. Rate limiting is going to be important if/when we decide to implement workload isolation. Until then, I think we should focus on three things:

  * Node health (Nodes should produce useful work and should be stable and not overloaded)
  * Latency (we always need to find an optimal way to process request and minimize overall queueing time)
  * Fairness (avoid workload and utilization imbalances)

All three points are achievable with very straightforward approaches that will not require much operator involvement.

I guess my main point is we need to solve load balancing (summarized by the above three points) before we start working on the rate limiter, but there's a good chance we may not need one apart from use cases that require workload isolation. 


On Fri, Sep 20, 2024, at 8:14 PM, Jordan West wrote:
> +1 to Benedict’s (and others) comments on plugability and low overhead when disabled. The latter I think needs little justification. The reason I am big on the former is, in my opinion: decisions on approach need to be settled with numbers not anecdotes or past experience (including my own). So I would like to see us compare different approaches (what metrics to use, etc). 
> 
> Personally, I’m a bit skeptical that we will come up with a metric based heuristic that works well in most scenarios and doesn’t require significant knowledge and tuning. I think past implementations of the dynamic snitch are good evidence of that. However, I expressed the same concerns internally for a client level project where we exposed metrics to induce back pressure and early experiments are encouraging / contrary to my expectations. At different layers different approaches can work better or worse. Same with different workloads. I don’t think we should dismiss approaches out right in this thread without hard numbers. 
> 
> In short, I think the testing and evaluation of this CEP is as important as its design and implementation. We will need to test a wide variety of workloads and potentially implementations and that’s where pluggability will be a huge benefit. I would go as far as saying the CEP should focus more on a framework for pluggable implementations that has low to zero cost when disabled than a specific set of metrics to use or specific approach. 
> 
> Jordan 
> 
> On Thu, Sep 19, 2024 at 14:38 Benedict Elliott Smith <be...@apache.org> wrote:
>> I just want to flag here that this is a topic I have strong opinions on, but the CEP is not really specific or detailed enough to understand precisely how it will be implemented. So, if a patch is already being produced, most of my feedback is likely to be provided some time after a patch appears, through the normal review process. I want to flag this now to avoid any surprise.
>> 
>> I will say that upfront that, ideally, this system should be designed to have ~zero overhead when disabled, and with minimal coupling (between its own components and C* itself), so that entirely orthogonal approaches can be integrated in future without polluting the codebase.
>> 
>> 
>>> On 19 Sep 2024, at 19:14, Patrick McFadin <pm...@gmail.com> wrote:
>>> 
>>> The work has begun but we don't have a VOTE thread for this CEP. Can one get started?
>>> 
>>> On Mon, May 6, 2024 at 9:24 PM Jaydeep Chovatia <ch...@gmail.com> wrote:
>>>> Sure, Caleb. I will include the work as part of CASSANDRA-19534 <https://issues.apache.org/jira/browse/CASSANDRA-19534> in the CEP-41.
>>>> 
>>>> Jaydeep
>>>> 
>>>> On Fri, May 3, 2024 at 7:48 AM Caleb Rackliffe <ca...@gmail.com> wrote:
>>>>> FYI, there is some ongoing sort-of-related work going on in CASSANDRA-19534 <https://issues.apache.org/jira/browse/CASSANDRA-19534>
>>>>> 
>>>>> On Wed, Apr 10, 2024 at 6:35 PM Jaydeep Chovatia <ch...@gmail.com> wrote:
>>>>>> Just created an official CEP-41 <https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-41+%28DRAFT%29+Apache+Cassandra+Unified+Rate+Limiter> incorporating the feedback from this discussion. Feel free to let me know if I may have missed some important feedback in this thread that is not captured in the CEP-41.
>>>>>> 
>>>>>> Jaydeep
>>>>>> 
>>>>>> On Thu, Feb 22, 2024 at 11:36 AM Jaydeep Chovatia <ch...@gmail.com> wrote:
>>>>>>> Thanks, Josh. I will file an official CEP with all the details in a few days and update this thread with that CEP number.
>>>>>>> Thanks a lot everyone for providing valuable insights!
>>>>>>> 
>>>>>>> Jaydeep
>>>>>>> 
>>>>>>> On Thu, Feb 22, 2024 at 9:24 AM Josh McKenzie <jm...@apache.org> wrote:
>>>>>>>> __
>>>>>>>>> Do folks think we should file an official CEP and take it there?
>>>>>>>> +1 here.
>>>>>>>> 
>>>>>>>> Synthesizing your gdoc, Caleb's work, and the feedback from this thread into a draft seems like a solid next step.
>>>>>>>> 
>>>>>>>> On Wed, Feb 7, 2024, at 12:31 PM, Jaydeep Chovatia wrote:
>>>>>>>>> I see a lot of great ideas being discussed or proposed in the past to cover the most common rate limiter candidate use cases. Do folks think we should file an official CEP and take it there?
>>>>>>>>> 
>>>>>>>>> Jaydeep
>>>>>>>>> 
>>>>>>>>> On Fri, Feb 2, 2024 at 8:30 AM Caleb Rackliffe <ca...@gmail.com> wrote:
>>>>>>>>>> I just remembered the other day that I had done a quick writeup on the state of compaction stress-related throttling in the project:
>>>>>>>>>> 
>>>>>>>>>> https://docs.google.com/document/d/1dfTEcKVidRKC1EWu3SO1kE1iVLMdaJ9uY1WMpS3P_hs/edit?usp=sharing
>>>>>>>>>> 
>>>>>>>>>> I'm sure most of it is old news to the people on this thread, but I figured I'd post it just in case :)
>>>>>>>>>> 
>>>>>>>>>> On Tue, Jan 30, 2024 at 11:58 AM Josh McKenzie <jm...@apache.org> wrote:
>>>>>>>>>>> __
>>>>>>>>>>>> 2.) We should make sure the links between the "known" root causes of cascading failures and the mechanisms we introduce to avoid them remain very strong.
>>>>>>>>>>> Seems to me that our historical strategy was to address individual known cases one-by-one rather than looking for a more holistic load-balancing and load-shedding solution. While the engineer in me likes the elegance of a broad, more-inclusive *actual SEDA-like* approach, the pragmatist in me wonders how far we think we are today from a stable set-point.
>>>>>>>>>>> 
>>>>>>>>>>> i.e. are we facing a handful of cases where nodes can still get pushed over and then cascade that we can surgically address, or are we facing a broader lack of back-pressure that rears its head in different domains (client -> coordinator, coordinator -> replica, internode with other operations, etc) at surprising times and should be considered more holistically?
>>>>>>>>>>> 
>>>>>>>>>>> On Tue, Jan 30, 2024, at 12:31 AM, Caleb Rackliffe wrote:
>>>>>>>>>>>> I almost forgot CASSANDRA-15817, which introduced reject_repair_compaction_threshold, which provides a mechanism to stop repairs while compaction is underwater.
>>>>>>>>>>>> 
>>>>>>>>>>>>> On Jan 26, 2024, at 6:22 PM, Caleb Rackliffe <ca...@gmail.com> wrote:
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Hey all,
>>>>>>>>>>>>> 
>>>>>>>>>>>>> I'm a bit late to the discussion. I see that we've already discussed CASSANDRA-15013 <https://issues.apache.org/jira/browse/CASSANDRA-15013> and CASSANDRA-16663 <https://issues.apache.org/jira/browse/CASSANDRA-16663> at least in passing. Having written the latter, I'd be the first to admit it's a crude tool, although it's been useful here and there, and provides a couple primitives that may be useful for future work. As Scott mentions, while it is configurable at runtime, it is not adaptive, although we did make configuration easier in CASSANDRA-17423 <https://issues.apache.org/jira/browse/CASSANDRA-17423>. It also is global to the node, although we've lightly discussed some ideas around making it more granular. (For example, keyspace-based limiting, or limiting "domains" tagged by the client in requests, could be interesting.) It also does not deal with inter-node traffic, of course.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Something we've not yet mentioned (that does address internode traffic) is CASSANDRA-17324 <https://issues.apache.org/jira/browse/CASSANDRA-17324>, which I proposed shortly after working on the native request limiter (and have just not had much time to return to). The basic idea is this:
>>>>>>>>>>>>> 
>>>>>>>>>>>>>> When a node is struggling under the weight of a compaction backlog and becomes a cause of increased read latency for clients, we have two safety valves:
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 1.) Disabling the native protocol server, which stops the node from coordinating reads and writes.
>>>>>>>>>>>>>> 2.) Jacking up the severity on the node, which tells the dynamic snitch to avoid the node for reads from other coordinators.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> These are useful, but we don’t appear to have any mechanism that would allow us to temporarily reject internode hint, batch, and mutation messages that could further delay resolution of the compaction backlog.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Whether it's done as part of a larger framework or on its own, it still feels like a good idea.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Thinking in terms of opportunity costs here (i.e. where we spend our finite engineering time to holistically improve the experience of operating this database) is healthy, but we probably haven't reached the point of diminishing returns on nodes being able to protect themselves from clients and from other nodes. I would just keep in mind two things:
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 1.) The effectiveness of rate-limiting in the system (which includes the database and all clients) as a whole necessarily decreases as we move from the application to the lowest-level database internals. Limiting correctly at the client will save more resources than limiting at the native protocol server, and limiting correctly at the native protocol server will save more resources than limiting after we've dispatched requests to some thread pool for processing.
>>>>>>>>>>>>> 2.) We should make sure the links between the "known" root causes of cascading failures and the mechanisms we introduce to avoid them remain very strong.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> In any case, I'd be happy to help out in any way I can as this moves forward (especially as it relates to our past/current attempts to address this problem space).
>>>>>>>>>>> 
>>>>>>>> 

Re: [EXTERNAL] [Discuss] Generic Purpose Rate Limiter in Cassandra

[Jordan West <jo...@gmail.com>]

+1 to Benedict’s (and others) comments on plugability and low overhead when
disabled. The latter I think needs little justification. The reason I am
big on the former is, in my opinion: decisions on approach need to be
settled with numbers not anecdotes or past experience (including my own).
So I would like to see us compare different approaches (what metrics to
use, etc).

Personally, I’m a bit skeptical that we will come up with a metric based
heuristic that works well in most scenarios and doesn’t require significant
knowledge and tuning. I think past implementations of the dynamic snitch
are good evidence of that. However, I expressed the same concerns
internally for a client level project where we exposed metrics to induce
back pressure and early experiments are encouraging / contrary to my
expectations. At different layers different approaches can work better or
worse. Same with different workloads. I don’t think we should dismiss
approaches out right in this thread without hard numbers.

In short, I think the testing and evaluation of this CEP is as important as
its design and implementation. We will need to test a wide variety of
workloads and potentially implementations and that’s where pluggability
will be a huge benefit. I would go as far as saying the CEP should focus
more on a framework for pluggable implementations that has low to zero cost
when disabled than a specific set of metrics to use or specific approach.

Jordan

On Thu, Sep 19, 2024 at 14:38 Benedict Elliott Smith <be...@apache.org>
wrote:

> I just want to flag here that this is a topic I have strong opinions on,
> but the CEP is not really specific or detailed enough to understand
> precisely how it will be implemented. So, if a patch is already being
> produced, most of my feedback is likely to be provided some time after a
> patch appears, through the normal review process. I want to flag this now
> to avoid any surprise.
>
> I will say that upfront that, ideally, this system should be designed to
> have ~zero overhead when disabled, and with minimal coupling (between its
> own components and C* itself), so that entirely orthogonal approaches can
> be integrated in future without polluting the codebase.
>
>
> On 19 Sep 2024, at 19:14, Patrick McFadin <pm...@gmail.com> wrote:
>
> The work has begun but we don't have a VOTE thread for this CEP. Can one
> get started?
>
> On Mon, May 6, 2024 at 9:24 PM Jaydeep Chovatia <
> chovatia.jaydeep@gmail.com> wrote:
>
>> Sure, Caleb. I will include the work as part of CASSANDRA-19534
>> <https://issues.apache.org/jira/browse/CASSANDRA-19534> in the CEP-41.
>>
>> Jaydeep
>>
>> On Fri, May 3, 2024 at 7:48 AM Caleb Rackliffe <ca...@gmail.com>
>> wrote:
>>
>>> FYI, there is some ongoing sort-of-related work going on in
>>> CASSANDRA-19534 <https://issues.apache.org/jira/browse/CASSANDRA-19534>
>>>
>>> On Wed, Apr 10, 2024 at 6:35 PM Jaydeep Chovatia <
>>> chovatia.jaydeep@gmail.com> wrote:
>>>
>>>> Just created an official CEP-41
>>>> <https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-41+%28DRAFT%29+Apache+Cassandra+Unified+Rate+Limiter>
>>>> incorporating the feedback from this discussion. Feel free to let me know
>>>> if I may have missed some important feedback in this thread that is not
>>>> captured in the CEP-41.
>>>>
>>>> Jaydeep
>>>>
>>>> On Thu, Feb 22, 2024 at 11:36 AM Jaydeep Chovatia <
>>>> chovatia.jaydeep@gmail.com> wrote:
>>>>
>>>>> Thanks, Josh. I will file an official CEP with all the details in a
>>>>> few days and update this thread with that CEP number.
>>>>> Thanks a lot everyone for providing valuable insights!
>>>>>
>>>>> Jaydeep
>>>>>
>>>>> On Thu, Feb 22, 2024 at 9:24 AM Josh McKenzie <jm...@apache.org>
>>>>> wrote:
>>>>>
>>>>>> Do folks think we should file an official CEP and take it there?
>>>>>>
>>>>>> +1 here.
>>>>>>
>>>>>> Synthesizing your gdoc, Caleb's work, and the feedback from this
>>>>>> thread into a draft seems like a solid next step.
>>>>>>
>>>>>> On Wed, Feb 7, 2024, at 12:31 PM, Jaydeep Chovatia wrote:
>>>>>>
>>>>>> I see a lot of great ideas being discussed or proposed in the past to
>>>>>> cover the most common rate limiter candidate use cases. Do folks think we
>>>>>> should file an official CEP and take it there?
>>>>>>
>>>>>> Jaydeep
>>>>>>
>>>>>> On Fri, Feb 2, 2024 at 8:30 AM Caleb Rackliffe <
>>>>>> calebrackliffe@gmail.com> wrote:
>>>>>>
>>>>>> I just remembered the other day that I had done a quick writeup on
>>>>>> the state of compaction stress-related throttling in the project:
>>>>>>
>>>>>>
>>>>>> https://docs.google.com/document/d/1dfTEcKVidRKC1EWu3SO1kE1iVLMdaJ9uY1WMpS3P_hs/edit?usp=sharing
>>>>>>
>>>>>> I'm sure most of it is old news to the people on this thread, but I
>>>>>> figured I'd post it just in case :)
>>>>>>
>>>>>> On Tue, Jan 30, 2024 at 11:58 AM Josh McKenzie <jm...@apache.org>
>>>>>> wrote:
>>>>>>
>>>>>>
>>>>>> 2.) We should make sure the links between the "known" root causes of
>>>>>> cascading failures and the mechanisms we introduce to avoid them remain
>>>>>> very strong.
>>>>>>
>>>>>> Seems to me that our historical strategy was to address individual
>>>>>> known cases one-by-one rather than looking for a more holistic
>>>>>> load-balancing and load-shedding solution. While the engineer in me likes
>>>>>> the elegance of a broad, more-inclusive *actual SEDA-like* approach,
>>>>>> the pragmatist in me wonders how far we think we are today from a stable
>>>>>> set-point.
>>>>>>
>>>>>> i.e. are we facing a handful of cases where nodes can still get
>>>>>> pushed over and then cascade that we can surgically address, or are we
>>>>>> facing a broader lack of back-pressure that rears its head in different
>>>>>> domains (client -> coordinator, coordinator -> replica, internode with
>>>>>> other operations, etc) at surprising times and should be considered more
>>>>>> holistically?
>>>>>>
>>>>>> On Tue, Jan 30, 2024, at 12:31 AM, Caleb Rackliffe wrote:
>>>>>>
>>>>>> I almost forgot CASSANDRA-15817, which introduced
>>>>>> reject_repair_compaction_threshold, which provides a mechanism to stop
>>>>>> repairs while compaction is underwater.
>>>>>>
>>>>>> On Jan 26, 2024, at 6:22 PM, Caleb Rackliffe <
>>>>>> calebrackliffe@gmail.com> wrote:
>>>>>>
>>>>>> 
>>>>>> Hey all,
>>>>>>
>>>>>> I'm a bit late to the discussion. I see that we've already discussed
>>>>>> CASSANDRA-15013
>>>>>> <https://issues.apache.org/jira/browse/CASSANDRA-15013> and
>>>>>> CASSANDRA-16663
>>>>>> <https://issues.apache.org/jira/browse/CASSANDRA-16663> at least in
>>>>>> passing. Having written the latter, I'd be the first to admit it's a crude
>>>>>> tool, although it's been useful here and there, and provides a couple
>>>>>> primitives that may be useful for future work. As Scott mentions, while it
>>>>>> is configurable at runtime, it is not adaptive, although we did
>>>>>> make configuration easier in CASSANDRA-17423
>>>>>> <https://issues.apache.org/jira/browse/CASSANDRA-17423>. It also is
>>>>>> global to the node, although we've lightly discussed some ideas around
>>>>>> making it more granular. (For example, keyspace-based limiting, or limiting
>>>>>> "domains" tagged by the client in requests, could be interesting.) It also
>>>>>> does not deal with inter-node traffic, of course.
>>>>>>
>>>>>> Something we've not yet mentioned (that does address internode
>>>>>> traffic) is CASSANDRA-17324
>>>>>> <https://issues.apache.org/jira/browse/CASSANDRA-17324>, which I
>>>>>> proposed shortly after working on the native request limiter (and have just
>>>>>> not had much time to return to). The basic idea is this:
>>>>>>
>>>>>> When a node is struggling under the weight of a compaction backlog
>>>>>> and becomes a cause of increased read latency for clients, we have two
>>>>>> safety valves:
>>>>>>
>>>>>>
>>>>>> 1.) Disabling the native protocol server, which stops the node from
>>>>>> coordinating reads and writes.
>>>>>> 2.) Jacking up the severity on the node, which tells the dynamic
>>>>>> snitch to avoid the node for reads from other coordinators.
>>>>>>
>>>>>>
>>>>>> These are useful, but we don’t appear to have any mechanism that
>>>>>> would allow us to temporarily reject internode hint, batch, and mutation
>>>>>> messages that could further delay resolution of the compaction backlog.
>>>>>>
>>>>>>
>>>>>> Whether it's done as part of a larger framework or on its own, it
>>>>>> still feels like a good idea.
>>>>>>
>>>>>> Thinking in terms of opportunity costs here (i.e. where we spend our
>>>>>> finite engineering time to holistically improve the experience of operating
>>>>>> this database) is healthy, but we probably haven't reached the point of
>>>>>> diminishing returns on nodes being able to protect themselves from clients
>>>>>> and from other nodes. I would just keep in mind two things:
>>>>>>
>>>>>> 1.) The effectiveness of rate-limiting in the system (which includes
>>>>>> the database and all clients) as a whole necessarily decreases as we move
>>>>>> from the application to the lowest-level database internals. Limiting
>>>>>> correctly at the client will save more resources than limiting at the
>>>>>> native protocol server, and limiting correctly at the native protocol
>>>>>> server will save more resources than limiting after we've dispatched
>>>>>> requests to some thread pool for processing.
>>>>>> 2.) We should make sure the links between the "known" root causes of
>>>>>> cascading failures and the mechanisms we introduce to avoid them remain
>>>>>> very strong.
>>>>>>
>>>>>> In any case, I'd be happy to help out in any way I can as this moves
>>>>>> forward (especially as it relates to our past/current attempts to address
>>>>>> this problem space).
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>

Re: [EXTERNAL] [Discuss] Generic Purpose Rate Limiter in Cassandra

[Jon Haddad <jo...@rustyrazorblade.com>]

Assuming the intent was to migrate the Google Doc to the CEP, I took
another look.  I think there's some ambitious ideas here, and I appreciate
any effort to improve Cassandra's stability.  I think CASSANDRA-19534
<https://issues.apache.org/jira/browse/CASSANDRA-19534> was a massive step
in the right direction, and a bit of what's in the doc complements the work
that Alex did.

I think here, there's a case to be made for less is more.  The doc lists
several things that the system could use to apply a rate limiter, but I
think the strongest signals are the queue depth of individual pools (NTR,
Mutation, Read) and rate of timeouts.  I'd like to suggest the first
iteration of this should only look at those things, rather than trying to
utilize system metrics.  You can have high CPU usage with a low queue
depth, and you can have low CPU usage with a deep queue.  In the former
example, there's no need to throttle, the system is just busy.  In the
second, there's a need to throttle.  This is a common issue with say, LWTs,
where the system hasn't been configured with a high enough
concurrent_writes.

I also think the CEP should be simplified further, by enforcing queue depth
limits instead of a rate limit.  Pending requests piling up have been a
huge problem for system instability, which is why CASSANDRA-19534
<https://issues.apache.org/jira/browse/CASSANDRA-19534> is such a
significant improvement - we're able to shed timed out requests much more
aggressively than we were before which leads to faster recovery times.

So my suggestion is to simplify things, so where we

* use the rate of timeouts to limit the depth of the queues for each of the
thread pools
* reject requests when the queue is full with an OverloadedException.

Ideally the drivers would back off and rate limit, because when you start
rejecting requests that retry, you increase load on other parts of the
system, which affects stability as well.

If you want to follow this up with the ability to dynamically resize thread
pools that could be interesting.  I think that would be the right time to
look a system resources, because if you have 100K pending reads and you're
at 20% CPU w/ low disk latency, you can probably increase
concurrent_reads.

Jon




On Thu, Sep 19, 2024 at 2:38 PM Benedict Elliott Smith <be...@apache.org>
wrote:

> I just want to flag here that this is a topic I have strong opinions on,
> but the CEP is not really specific or detailed enough to understand
> precisely how it will be implemented. So, if a patch is already being
> produced, most of my feedback is likely to be provided some time after a
> patch appears, through the normal review process. I want to flag this now
> to avoid any surprise.
>
> I will say that upfront that, ideally, this system should be designed to
> have ~zero overhead when disabled, and with minimal coupling (between its
> own components and C* itself), so that entirely orthogonal approaches can
> be integrated in future without polluting the codebase.
>
>
> On 19 Sep 2024, at 19:14, Patrick McFadin <pm...@gmail.com> wrote:
>
> The work has begun but we don't have a VOTE thread for this CEP. Can one
> get started?
>
> On Mon, May 6, 2024 at 9:24 PM Jaydeep Chovatia <
> chovatia.jaydeep@gmail.com> wrote:
>
>> Sure, Caleb. I will include the work as part of CASSANDRA-19534
>> <https://issues.apache.org/jira/browse/CASSANDRA-19534> in the CEP-41.
>>
>> Jaydeep
>>
>> On Fri, May 3, 2024 at 7:48 AM Caleb Rackliffe <ca...@gmail.com>
>> wrote:
>>
>>> FYI, there is some ongoing sort-of-related work going on in
>>> CASSANDRA-19534 <https://issues.apache.org/jira/browse/CASSANDRA-19534>
>>>
>>> On Wed, Apr 10, 2024 at 6:35 PM Jaydeep Chovatia <
>>> chovatia.jaydeep@gmail.com> wrote:
>>>
>>>> Just created an official CEP-41
>>>> <https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-41+%28DRAFT%29+Apache+Cassandra+Unified+Rate+Limiter>
>>>> incorporating the feedback from this discussion. Feel free to let me know
>>>> if I may have missed some important feedback in this thread that is not
>>>> captured in the CEP-41.
>>>>
>>>> Jaydeep
>>>>
>>>> On Thu, Feb 22, 2024 at 11:36 AM Jaydeep Chovatia <
>>>> chovatia.jaydeep@gmail.com> wrote:
>>>>
>>>>> Thanks, Josh. I will file an official CEP with all the details in a
>>>>> few days and update this thread with that CEP number.
>>>>> Thanks a lot everyone for providing valuable insights!
>>>>>
>>>>> Jaydeep
>>>>>
>>>>> On Thu, Feb 22, 2024 at 9:24 AM Josh McKenzie <jm...@apache.org>
>>>>> wrote:
>>>>>
>>>>>> Do folks think we should file an official CEP and take it there?
>>>>>>
>>>>>> +1 here.
>>>>>>
>>>>>> Synthesizing your gdoc, Caleb's work, and the feedback from this
>>>>>> thread into a draft seems like a solid next step.
>>>>>>
>>>>>> On Wed, Feb 7, 2024, at 12:31 PM, Jaydeep Chovatia wrote:
>>>>>>
>>>>>> I see a lot of great ideas being discussed or proposed in the past to
>>>>>> cover the most common rate limiter candidate use cases. Do folks think we
>>>>>> should file an official CEP and take it there?
>>>>>>
>>>>>> Jaydeep
>>>>>>
>>>>>> On Fri, Feb 2, 2024 at 8:30 AM Caleb Rackliffe <
>>>>>> calebrackliffe@gmail.com> wrote:
>>>>>>
>>>>>> I just remembered the other day that I had done a quick writeup on
>>>>>> the state of compaction stress-related throttling in the project:
>>>>>>
>>>>>>
>>>>>> https://docs.google.com/document/d/1dfTEcKVidRKC1EWu3SO1kE1iVLMdaJ9uY1WMpS3P_hs/edit?usp=sharing
>>>>>>
>>>>>> I'm sure most of it is old news to the people on this thread, but I
>>>>>> figured I'd post it just in case :)
>>>>>>
>>>>>> On Tue, Jan 30, 2024 at 11:58 AM Josh McKenzie <jm...@apache.org>
>>>>>> wrote:
>>>>>>
>>>>>>
>>>>>> 2.) We should make sure the links between the "known" root causes of
>>>>>> cascading failures and the mechanisms we introduce to avoid them remain
>>>>>> very strong.
>>>>>>
>>>>>> Seems to me that our historical strategy was to address individual
>>>>>> known cases one-by-one rather than looking for a more holistic
>>>>>> load-balancing and load-shedding solution. While the engineer in me likes
>>>>>> the elegance of a broad, more-inclusive *actual SEDA-like* approach,
>>>>>> the pragmatist in me wonders how far we think we are today from a stable
>>>>>> set-point.
>>>>>>
>>>>>> i.e. are we facing a handful of cases where nodes can still get
>>>>>> pushed over and then cascade that we can surgically address, or are we
>>>>>> facing a broader lack of back-pressure that rears its head in different
>>>>>> domains (client -> coordinator, coordinator -> replica, internode with
>>>>>> other operations, etc) at surprising times and should be considered more
>>>>>> holistically?
>>>>>>
>>>>>> On Tue, Jan 30, 2024, at 12:31 AM, Caleb Rackliffe wrote:
>>>>>>
>>>>>> I almost forgot CASSANDRA-15817, which introduced
>>>>>> reject_repair_compaction_threshold, which provides a mechanism to stop
>>>>>> repairs while compaction is underwater.
>>>>>>
>>>>>> On Jan 26, 2024, at 6:22 PM, Caleb Rackliffe <
>>>>>> calebrackliffe@gmail.com> wrote:
>>>>>>
>>>>>> 
>>>>>> Hey all,
>>>>>>
>>>>>> I'm a bit late to the discussion. I see that we've already discussed
>>>>>> CASSANDRA-15013
>>>>>> <https://issues.apache.org/jira/browse/CASSANDRA-15013> and
>>>>>> CASSANDRA-16663
>>>>>> <https://issues.apache.org/jira/browse/CASSANDRA-16663> at least in
>>>>>> passing. Having written the latter, I'd be the first to admit it's a crude
>>>>>> tool, although it's been useful here and there, and provides a couple
>>>>>> primitives that may be useful for future work. As Scott mentions, while it
>>>>>> is configurable at runtime, it is not adaptive, although we did
>>>>>> make configuration easier in CASSANDRA-17423
>>>>>> <https://issues.apache.org/jira/browse/CASSANDRA-17423>. It also is
>>>>>> global to the node, although we've lightly discussed some ideas around
>>>>>> making it more granular. (For example, keyspace-based limiting, or limiting
>>>>>> "domains" tagged by the client in requests, could be interesting.) It also
>>>>>> does not deal with inter-node traffic, of course.
>>>>>>
>>>>>> Something we've not yet mentioned (that does address internode
>>>>>> traffic) is CASSANDRA-17324
>>>>>> <https://issues.apache.org/jira/browse/CASSANDRA-17324>, which I
>>>>>> proposed shortly after working on the native request limiter (and have just
>>>>>> not had much time to return to). The basic idea is this:
>>>>>>
>>>>>> When a node is struggling under the weight of a compaction backlog
>>>>>> and becomes a cause of increased read latency for clients, we have two
>>>>>> safety valves:
>>>>>>
>>>>>>
>>>>>> 1.) Disabling the native protocol server, which stops the node from
>>>>>> coordinating reads and writes.
>>>>>> 2.) Jacking up the severity on the node, which tells the dynamic
>>>>>> snitch to avoid the node for reads from other coordinators.
>>>>>>
>>>>>>
>>>>>> These are useful, but we don’t appear to have any mechanism that
>>>>>> would allow us to temporarily reject internode hint, batch, and mutation
>>>>>> messages that could further delay resolution of the compaction backlog.
>>>>>>
>>>>>>
>>>>>> Whether it's done as part of a larger framework or on its own, it
>>>>>> still feels like a good idea.
>>>>>>
>>>>>> Thinking in terms of opportunity costs here (i.e. where we spend our
>>>>>> finite engineering time to holistically improve the experience of operating
>>>>>> this database) is healthy, but we probably haven't reached the point of
>>>>>> diminishing returns on nodes being able to protect themselves from clients
>>>>>> and from other nodes. I would just keep in mind two things:
>>>>>>
>>>>>> 1.) The effectiveness of rate-limiting in the system (which includes
>>>>>> the database and all clients) as a whole necessarily decreases as we move
>>>>>> from the application to the lowest-level database internals. Limiting
>>>>>> correctly at the client will save more resources than limiting at the
>>>>>> native protocol server, and limiting correctly at the native protocol
>>>>>> server will save more resources than limiting after we've dispatched
>>>>>> requests to some thread pool for processing.
>>>>>> 2.) We should make sure the links between the "known" root causes of
>>>>>> cascading failures and the mechanisms we introduce to avoid them remain
>>>>>> very strong.
>>>>>>
>>>>>> In any case, I'd be happy to help out in any way I can as this moves
>>>>>> forward (especially as it relates to our past/current attempts to address
>>>>>> this problem space).
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>

Re: [EXTERNAL] [Discuss] Generic Purpose Rate Limiter in Cassandra

[Benedict Elliott Smith <be...@apache.org>]

I just want to flag here that this is a topic I have strong opinions on, but the CEP is not really specific or detailed enough to understand precisely how it will be implemented. So, if a patch is already being produced, most of my feedback is likely to be provided some time after a patch appears, through the normal review process. I want to flag this now to avoid any surprise.

I will say that upfront that, ideally, this system should be designed to have ~zero overhead when disabled, and with minimal coupling (between its own components and C* itself), so that entirely orthogonal approaches can be integrated in future without polluting the codebase.


> On 19 Sep 2024, at 19:14, Patrick McFadin <pm...@gmail.com> wrote:
> 
> The work has begun but we don't have a VOTE thread for this CEP. Can one get started?
> 
> On Mon, May 6, 2024 at 9:24 PM Jaydeep Chovatia <chovatia.jaydeep@gmail.com <ma...@gmail.com>> wrote:
>> Sure, Caleb. I will include the work as part of CASSANDRA-19534 <https://issues.apache.org/jira/browse/CASSANDRA-19534> in the CEP-41.
>> 
>> Jaydeep
>> 
>> On Fri, May 3, 2024 at 7:48 AM Caleb Rackliffe <calebrackliffe@gmail.com <ma...@gmail.com>> wrote:
>>> FYI, there is some ongoing sort-of-related work going on in CASSANDRA-19534 <https://issues.apache.org/jira/browse/CASSANDRA-19534>
>>> On Wed, Apr 10, 2024 at 6:35 PM Jaydeep Chovatia <chovatia.jaydeep@gmail.com <ma...@gmail.com>> wrote:
>>>> Just created an official CEP-41 <https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-41+%28DRAFT%29+Apache+Cassandra+Unified+Rate+Limiter> incorporating the feedback from this discussion. Feel free to let me know if I may have missed some important feedback in this thread that is not captured in the CEP-41.
>>>> 
>>>> Jaydeep
>>>> 
>>>> On Thu, Feb 22, 2024 at 11:36 AM Jaydeep Chovatia <chovatia.jaydeep@gmail.com <ma...@gmail.com>> wrote:
>>>>> Thanks, Josh. I will file an official CEP with all the details in a few days and update this thread with that CEP number.
>>>>> Thanks a lot everyone for providing valuable insights!
>>>>> 
>>>>> Jaydeep
>>>>> 
>>>>> On Thu, Feb 22, 2024 at 9:24 AM Josh McKenzie <jmckenzie@apache.org <ma...@apache.org>> wrote:
>>>>>>> Do folks think we should file an official CEP and take it there?
>>>>>> +1 here.
>>>>>> 
>>>>>> Synthesizing your gdoc, Caleb's work, and the feedback from this thread into a draft seems like a solid next step.
>>>>>> 
>>>>>> On Wed, Feb 7, 2024, at 12:31 PM, Jaydeep Chovatia wrote:
>>>>>>> I see a lot of great ideas being discussed or proposed in the past to cover the most common rate limiter candidate use cases. Do folks think we should file an official CEP and take it there?
>>>>>>> 
>>>>>>> Jaydeep
>>>>>>> 
>>>>>>> On Fri, Feb 2, 2024 at 8:30 AM Caleb Rackliffe <calebrackliffe@gmail.com <ma...@gmail.com>> wrote:
>>>>>>> I just remembered the other day that I had done a quick writeup on the state of compaction stress-related throttling in the project:
>>>>>>> 
>>>>>>> https://docs.google.com/document/d/1dfTEcKVidRKC1EWu3SO1kE1iVLMdaJ9uY1WMpS3P_hs/edit?usp=sharing
>>>>>>> 
>>>>>>> I'm sure most of it is old news to the people on this thread, but I figured I'd post it just in case :)
>>>>>>> 
>>>>>>> On Tue, Jan 30, 2024 at 11:58 AM Josh McKenzie <jmckenzie@apache.org <ma...@apache.org>> wrote:
>>>>>>> 
>>>>>>>> 2.) We should make sure the links between the "known" root causes of cascading failures and the mechanisms we introduce to avoid them remain very strong.
>>>>>>> Seems to me that our historical strategy was to address individual known cases one-by-one rather than looking for a more holistic load-balancing and load-shedding solution. While the engineer in me likes the elegance of a broad, more-inclusive actual SEDA-like approach, the pragmatist in me wonders how far we think we are today from a stable set-point.
>>>>>>> 
>>>>>>> i.e. are we facing a handful of cases where nodes can still get pushed over and then cascade that we can surgically address, or are we facing a broader lack of back-pressure that rears its head in different domains (client -> coordinator, coordinator -> replica, internode with other operations, etc) at surprising times and should be considered more holistically?
>>>>>>> 
>>>>>>> On Tue, Jan 30, 2024, at 12:31 AM, Caleb Rackliffe wrote:
>>>>>>>> I almost forgot CASSANDRA-15817, which introduced reject_repair_compaction_threshold, which provides a mechanism to stop repairs while compaction is underwater.
>>>>>>>> 
>>>>>>>>> On Jan 26, 2024, at 6:22 PM, Caleb Rackliffe <calebrackliffe@gmail.com <ma...@gmail.com>> wrote:
>>>>>>>>> 
>>>>>>>>> Hey all,
>>>>>>>>> 
>>>>>>>>> I'm a bit late to the discussion. I see that we've already discussed CASSANDRA-15013 <https://issues.apache.org/jira/browse/CASSANDRA-15013> and CASSANDRA-16663 <https://issues.apache.org/jira/browse/CASSANDRA-16663> at least in passing. Having written the latter, I'd be the first to admit it's a crude tool, although it's been useful here and there, and provides a couple primitives that may be useful for future work. As Scott mentions, while it is configurable at runtime, it is not adaptive, although we did make configuration easier in CASSANDRA-17423 <https://issues.apache.org/jira/browse/CASSANDRA-17423>. It also is global to the node, although we've lightly discussed some ideas around making it more granular. (For example, keyspace-based limiting, or limiting "domains" tagged by the client in requests, could be interesting.) It also does not deal with inter-node traffic, of course.
>>>>>>>>> 
>>>>>>>>> Something we've not yet mentioned (that does address internode traffic) is CASSANDRA-17324 <https://issues.apache.org/jira/browse/CASSANDRA-17324>, which I proposed shortly after working on the native request limiter (and have just not had much time to return to). The basic idea is this:
>>>>>>>>> 
>>>>>>>>> When a node is struggling under the weight of a compaction backlog and becomes a cause of increased read latency for clients, we have two safety valves:
>>>>>>>>> 
>>>>>>>>> 1.) Disabling the native protocol server, which stops the node from coordinating reads and writes.
>>>>>>>>> 2.) Jacking up the severity on the node, which tells the dynamic snitch to avoid the node for reads from other coordinators.
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> These are useful, but we don’t appear to have any mechanism that would allow us to temporarily reject internode hint, batch, and mutation messages that could further delay resolution of the compaction backlog.
>>>>>>>>> 
>>>>>>>>> Whether it's done as part of a larger framework or on its own, it still feels like a good idea.
>>>>>>>>> 
>>>>>>>>> Thinking in terms of opportunity costs here (i.e. where we spend our finite engineering time to holistically improve the experience of operating this database) is healthy, but we probably haven't reached the point of diminishing returns on nodes being able to protect themselves from clients and from other nodes. I would just keep in mind two things:
>>>>>>>>> 
>>>>>>>>> 1.) The effectiveness of rate-limiting in the system (which includes the database and all clients) as a whole necessarily decreases as we move from the application to the lowest-level database internals. Limiting correctly at the client will save more resources than limiting at the native protocol server, and limiting correctly at the native protocol server will save more resources than limiting after we've dispatched requests to some thread pool for processing.
>>>>>>>>> 2.) We should make sure the links between the "known" root causes of cascading failures and the mechanisms we introduce to avoid them remain very strong.
>>>>>>>>> 
>>>>>>>>> In any case, I'd be happy to help out in any way I can as this moves forward (especially as it relates to our past/current attempts to address this problem space).
>>>>>>> 
>>>>>> 

Re: [EXTERNAL] Re: [Discuss] Generic Purpose Rate Limiter in Cassandra

[Patrick McFadin <pm...@gmail.com>]

The work has begun but we don't have a VOTE thread for this CEP. Can one
get started?

On Mon, May 6, 2024 at 9:24 PM Jaydeep Chovatia <ch...@gmail.com>
wrote:

> Sure, Caleb. I will include the work as part of CASSANDRA-19534
> <https://issues.apache.org/jira/browse/CASSANDRA-19534> in the CEP-41.
>
> Jaydeep
>
> On Fri, May 3, 2024 at 7:48 AM Caleb Rackliffe <ca...@gmail.com>
> wrote:
>
>> FYI, there is some ongoing sort-of-related work going on in
>> CASSANDRA-19534 <https://issues.apache.org/jira/browse/CASSANDRA-19534>
>>
>> On Wed, Apr 10, 2024 at 6:35 PM Jaydeep Chovatia <
>> chovatia.jaydeep@gmail.com> wrote:
>>
>>> Just created an official CEP-41
>>> <https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-41+%28DRAFT%29+Apache+Cassandra+Unified+Rate+Limiter>
>>> incorporating the feedback from this discussion. Feel free to let me know
>>> if I may have missed some important feedback in this thread that is not
>>> captured in the CEP-41.
>>>
>>> Jaydeep
>>>
>>> On Thu, Feb 22, 2024 at 11:36 AM Jaydeep Chovatia <
>>> chovatia.jaydeep@gmail.com> wrote:
>>>
>>>> Thanks, Josh. I will file an official CEP with all the details in a few
>>>> days and update this thread with that CEP number.
>>>> Thanks a lot everyone for providing valuable insights!
>>>>
>>>> Jaydeep
>>>>
>>>> On Thu, Feb 22, 2024 at 9:24 AM Josh McKenzie <jm...@apache.org>
>>>> wrote:
>>>>
>>>>> Do folks think we should file an official CEP and take it there?
>>>>>
>>>>> +1 here.
>>>>>
>>>>> Synthesizing your gdoc, Caleb's work, and the feedback from this
>>>>> thread into a draft seems like a solid next step.
>>>>>
>>>>> On Wed, Feb 7, 2024, at 12:31 PM, Jaydeep Chovatia wrote:
>>>>>
>>>>> I see a lot of great ideas being discussed or proposed in the past to
>>>>> cover the most common rate limiter candidate use cases. Do folks think we
>>>>> should file an official CEP and take it there?
>>>>>
>>>>> Jaydeep
>>>>>
>>>>> On Fri, Feb 2, 2024 at 8:30 AM Caleb Rackliffe <
>>>>> calebrackliffe@gmail.com> wrote:
>>>>>
>>>>> I just remembered the other day that I had done a quick writeup on the
>>>>> state of compaction stress-related throttling in the project:
>>>>>
>>>>>
>>>>> https://docs.google.com/document/d/1dfTEcKVidRKC1EWu3SO1kE1iVLMdaJ9uY1WMpS3P_hs/edit?usp=sharing
>>>>>
>>>>> I'm sure most of it is old news to the people on this thread, but I
>>>>> figured I'd post it just in case :)
>>>>>
>>>>> On Tue, Jan 30, 2024 at 11:58 AM Josh McKenzie <jm...@apache.org>
>>>>> wrote:
>>>>>
>>>>>
>>>>> 2.) We should make sure the links between the "known" root causes of
>>>>> cascading failures and the mechanisms we introduce to avoid them remain
>>>>> very strong.
>>>>>
>>>>> Seems to me that our historical strategy was to address individual
>>>>> known cases one-by-one rather than looking for a more holistic
>>>>> load-balancing and load-shedding solution. While the engineer in me likes
>>>>> the elegance of a broad, more-inclusive *actual SEDA-like* approach,
>>>>> the pragmatist in me wonders how far we think we are today from a stable
>>>>> set-point.
>>>>>
>>>>> i.e. are we facing a handful of cases where nodes can still get pushed
>>>>> over and then cascade that we can surgically address, or are we facing a
>>>>> broader lack of back-pressure that rears its head in different domains
>>>>> (client -> coordinator, coordinator -> replica, internode with other
>>>>> operations, etc) at surprising times and should be considered more
>>>>> holistically?
>>>>>
>>>>> On Tue, Jan 30, 2024, at 12:31 AM, Caleb Rackliffe wrote:
>>>>>
>>>>> I almost forgot CASSANDRA-15817, which introduced
>>>>> reject_repair_compaction_threshold, which provides a mechanism to stop
>>>>> repairs while compaction is underwater.
>>>>>
>>>>> On Jan 26, 2024, at 6:22 PM, Caleb Rackliffe <ca...@gmail.com>
>>>>> wrote:
>>>>>
>>>>> 
>>>>> Hey all,
>>>>>
>>>>> I'm a bit late to the discussion. I see that we've already discussed
>>>>> CASSANDRA-15013
>>>>> <https://issues.apache.org/jira/browse/CASSANDRA-15013> and
>>>>> CASSANDRA-16663
>>>>> <https://issues.apache.org/jira/browse/CASSANDRA-16663> at least in
>>>>> passing. Having written the latter, I'd be the first to admit it's a crude
>>>>> tool, although it's been useful here and there, and provides a couple
>>>>> primitives that may be useful for future work. As Scott mentions, while it
>>>>> is configurable at runtime, it is not adaptive, although we did
>>>>> make configuration easier in CASSANDRA-17423
>>>>> <https://issues.apache.org/jira/browse/CASSANDRA-17423>. It also is
>>>>> global to the node, although we've lightly discussed some ideas around
>>>>> making it more granular. (For example, keyspace-based limiting, or limiting
>>>>> "domains" tagged by the client in requests, could be interesting.) It also
>>>>> does not deal with inter-node traffic, of course.
>>>>>
>>>>> Something we've not yet mentioned (that does address internode
>>>>> traffic) is CASSANDRA-17324
>>>>> <https://issues.apache.org/jira/browse/CASSANDRA-17324>, which I
>>>>> proposed shortly after working on the native request limiter (and have just
>>>>> not had much time to return to). The basic idea is this:
>>>>>
>>>>> When a node is struggling under the weight of a compaction backlog and
>>>>> becomes a cause of increased read latency for clients, we have two safety
>>>>> valves:
>>>>>
>>>>>
>>>>> 1.) Disabling the native protocol server, which stops the node from
>>>>> coordinating reads and writes.
>>>>> 2.) Jacking up the severity on the node, which tells the dynamic
>>>>> snitch to avoid the node for reads from other coordinators.
>>>>>
>>>>>
>>>>> These are useful, but we don’t appear to have any mechanism that would
>>>>> allow us to temporarily reject internode hint, batch, and mutation messages
>>>>> that could further delay resolution of the compaction backlog.
>>>>>
>>>>>
>>>>> Whether it's done as part of a larger framework or on its own, it
>>>>> still feels like a good idea.
>>>>>
>>>>> Thinking in terms of opportunity costs here (i.e. where we spend our
>>>>> finite engineering time to holistically improve the experience of operating
>>>>> this database) is healthy, but we probably haven't reached the point of
>>>>> diminishing returns on nodes being able to protect themselves from clients
>>>>> and from other nodes. I would just keep in mind two things:
>>>>>
>>>>> 1.) The effectiveness of rate-limiting in the system (which includes
>>>>> the database and all clients) as a whole necessarily decreases as we move
>>>>> from the application to the lowest-level database internals. Limiting
>>>>> correctly at the client will save more resources than limiting at the
>>>>> native protocol server, and limiting correctly at the native protocol
>>>>> server will save more resources than limiting after we've dispatched
>>>>> requests to some thread pool for processing.
>>>>> 2.) We should make sure the links between the "known" root causes of
>>>>> cascading failures and the mechanisms we introduce to avoid them remain
>>>>> very strong.
>>>>>
>>>>> In any case, I'd be happy to help out in any way I can as this moves
>>>>> forward (especially as it relates to our past/current attempts to address
>>>>> this problem space).
>>>>>
>>>>>
>>>>>
>>>>>

Re: [EXTERNAL] Re: [Discuss] Generic Purpose Rate Limiter in Cassandra

[Jaydeep Chovatia <ch...@gmail.com>]

Sure, Caleb. I will include the work as part of CASSANDRA-19534
<https://issues.apache.org/jira/browse/CASSANDRA-19534> in the CEP-41.

Jaydeep

On Fri, May 3, 2024 at 7:48 AM Caleb Rackliffe <ca...@gmail.com>
wrote:

> FYI, there is some ongoing sort-of-related work going on in
> CASSANDRA-19534 <https://issues.apache.org/jira/browse/CASSANDRA-19534>
>
> On Wed, Apr 10, 2024 at 6:35 PM Jaydeep Chovatia <
> chovatia.jaydeep@gmail.com> wrote:
>
>> Just created an official CEP-41
>> <https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-41+%28DRAFT%29+Apache+Cassandra+Unified+Rate+Limiter>
>> incorporating the feedback from this discussion. Feel free to let me know
>> if I may have missed some important feedback in this thread that is not
>> captured in the CEP-41.
>>
>> Jaydeep
>>
>> On Thu, Feb 22, 2024 at 11:36 AM Jaydeep Chovatia <
>> chovatia.jaydeep@gmail.com> wrote:
>>
>>> Thanks, Josh. I will file an official CEP with all the details in a few
>>> days and update this thread with that CEP number.
>>> Thanks a lot everyone for providing valuable insights!
>>>
>>> Jaydeep
>>>
>>> On Thu, Feb 22, 2024 at 9:24 AM Josh McKenzie <jm...@apache.org>
>>> wrote:
>>>
>>>> Do folks think we should file an official CEP and take it there?
>>>>
>>>> +1 here.
>>>>
>>>> Synthesizing your gdoc, Caleb's work, and the feedback from this thread
>>>> into a draft seems like a solid next step.
>>>>
>>>> On Wed, Feb 7, 2024, at 12:31 PM, Jaydeep Chovatia wrote:
>>>>
>>>> I see a lot of great ideas being discussed or proposed in the past to
>>>> cover the most common rate limiter candidate use cases. Do folks think we
>>>> should file an official CEP and take it there?
>>>>
>>>> Jaydeep
>>>>
>>>> On Fri, Feb 2, 2024 at 8:30 AM Caleb Rackliffe <
>>>> calebrackliffe@gmail.com> wrote:
>>>>
>>>> I just remembered the other day that I had done a quick writeup on the
>>>> state of compaction stress-related throttling in the project:
>>>>
>>>>
>>>> https://docs.google.com/document/d/1dfTEcKVidRKC1EWu3SO1kE1iVLMdaJ9uY1WMpS3P_hs/edit?usp=sharing
>>>>
>>>> I'm sure most of it is old news to the people on this thread, but I
>>>> figured I'd post it just in case :)
>>>>
>>>> On Tue, Jan 30, 2024 at 11:58 AM Josh McKenzie <jm...@apache.org>
>>>> wrote:
>>>>
>>>>
>>>> 2.) We should make sure the links between the "known" root causes of
>>>> cascading failures and the mechanisms we introduce to avoid them remain
>>>> very strong.
>>>>
>>>> Seems to me that our historical strategy was to address individual
>>>> known cases one-by-one rather than looking for a more holistic
>>>> load-balancing and load-shedding solution. While the engineer in me likes
>>>> the elegance of a broad, more-inclusive *actual SEDA-like* approach,
>>>> the pragmatist in me wonders how far we think we are today from a stable
>>>> set-point.
>>>>
>>>> i.e. are we facing a handful of cases where nodes can still get pushed
>>>> over and then cascade that we can surgically address, or are we facing a
>>>> broader lack of back-pressure that rears its head in different domains
>>>> (client -> coordinator, coordinator -> replica, internode with other
>>>> operations, etc) at surprising times and should be considered more
>>>> holistically?
>>>>
>>>> On Tue, Jan 30, 2024, at 12:31 AM, Caleb Rackliffe wrote:
>>>>
>>>> I almost forgot CASSANDRA-15817, which introduced
>>>> reject_repair_compaction_threshold, which provides a mechanism to stop
>>>> repairs while compaction is underwater.
>>>>
>>>> On Jan 26, 2024, at 6:22 PM, Caleb Rackliffe <ca...@gmail.com>
>>>> wrote:
>>>>
>>>> 
>>>> Hey all,
>>>>
>>>> I'm a bit late to the discussion. I see that we've already discussed
>>>> CASSANDRA-15013 <https://issues.apache.org/jira/browse/CASSANDRA-15013>
>>>>  and CASSANDRA-16663
>>>> <https://issues.apache.org/jira/browse/CASSANDRA-16663> at least in
>>>> passing. Having written the latter, I'd be the first to admit it's a crude
>>>> tool, although it's been useful here and there, and provides a couple
>>>> primitives that may be useful for future work. As Scott mentions, while it
>>>> is configurable at runtime, it is not adaptive, although we did
>>>> make configuration easier in CASSANDRA-17423
>>>> <https://issues.apache.org/jira/browse/CASSANDRA-17423>. It also is
>>>> global to the node, although we've lightly discussed some ideas around
>>>> making it more granular. (For example, keyspace-based limiting, or limiting
>>>> "domains" tagged by the client in requests, could be interesting.) It also
>>>> does not deal with inter-node traffic, of course.
>>>>
>>>> Something we've not yet mentioned (that does address internode traffic)
>>>> is CASSANDRA-17324
>>>> <https://issues.apache.org/jira/browse/CASSANDRA-17324>, which I
>>>> proposed shortly after working on the native request limiter (and have just
>>>> not had much time to return to). The basic idea is this:
>>>>
>>>> When a node is struggling under the weight of a compaction backlog and
>>>> becomes a cause of increased read latency for clients, we have two safety
>>>> valves:
>>>>
>>>>
>>>> 1.) Disabling the native protocol server, which stops the node from
>>>> coordinating reads and writes.
>>>> 2.) Jacking up the severity on the node, which tells the dynamic snitch
>>>> to avoid the node for reads from other coordinators.
>>>>
>>>>
>>>> These are useful, but we don’t appear to have any mechanism that would
>>>> allow us to temporarily reject internode hint, batch, and mutation messages
>>>> that could further delay resolution of the compaction backlog.
>>>>
>>>>
>>>> Whether it's done as part of a larger framework or on its own, it still
>>>> feels like a good idea.
>>>>
>>>> Thinking in terms of opportunity costs here (i.e. where we spend our
>>>> finite engineering time to holistically improve the experience of operating
>>>> this database) is healthy, but we probably haven't reached the point of
>>>> diminishing returns on nodes being able to protect themselves from clients
>>>> and from other nodes. I would just keep in mind two things:
>>>>
>>>> 1.) The effectiveness of rate-limiting in the system (which includes
>>>> the database and all clients) as a whole necessarily decreases as we move
>>>> from the application to the lowest-level database internals. Limiting
>>>> correctly at the client will save more resources than limiting at the
>>>> native protocol server, and limiting correctly at the native protocol
>>>> server will save more resources than limiting after we've dispatched
>>>> requests to some thread pool for processing.
>>>> 2.) We should make sure the links between the "known" root causes of
>>>> cascading failures and the mechanisms we introduce to avoid them remain
>>>> very strong.
>>>>
>>>> In any case, I'd be happy to help out in any way I can as this moves
>>>> forward (especially as it relates to our past/current attempts to address
>>>> this problem space).
>>>>
>>>>
>>>>
>>>>

Re: [EXTERNAL] Re: [Discuss] Generic Purpose Rate Limiter in Cassandra

[Caleb Rackliffe <ca...@gmail.com>]

FYI, there is some ongoing sort-of-related work going on in CASSANDRA-19534
<https://issues.apache.org/jira/browse/CASSANDRA-19534>

On Wed, Apr 10, 2024 at 6:35 PM Jaydeep Chovatia <ch...@gmail.com>
wrote:

> Just created an official CEP-41
> <https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-41+%28DRAFT%29+Apache+Cassandra+Unified+Rate+Limiter>
> incorporating the feedback from this discussion. Feel free to let me know
> if I may have missed some important feedback in this thread that is not
> captured in the CEP-41.
>
> Jaydeep
>
> On Thu, Feb 22, 2024 at 11:36 AM Jaydeep Chovatia <
> chovatia.jaydeep@gmail.com> wrote:
>
>> Thanks, Josh. I will file an official CEP with all the details in a few
>> days and update this thread with that CEP number.
>> Thanks a lot everyone for providing valuable insights!
>>
>> Jaydeep
>>
>> On Thu, Feb 22, 2024 at 9:24 AM Josh McKenzie <jm...@apache.org>
>> wrote:
>>
>>> Do folks think we should file an official CEP and take it there?
>>>
>>> +1 here.
>>>
>>> Synthesizing your gdoc, Caleb's work, and the feedback from this thread
>>> into a draft seems like a solid next step.
>>>
>>> On Wed, Feb 7, 2024, at 12:31 PM, Jaydeep Chovatia wrote:
>>>
>>> I see a lot of great ideas being discussed or proposed in the past to
>>> cover the most common rate limiter candidate use cases. Do folks think we
>>> should file an official CEP and take it there?
>>>
>>> Jaydeep
>>>
>>> On Fri, Feb 2, 2024 at 8:30 AM Caleb Rackliffe <ca...@gmail.com>
>>> wrote:
>>>
>>> I just remembered the other day that I had done a quick writeup on the
>>> state of compaction stress-related throttling in the project:
>>>
>>>
>>> https://docs.google.com/document/d/1dfTEcKVidRKC1EWu3SO1kE1iVLMdaJ9uY1WMpS3P_hs/edit?usp=sharing
>>>
>>> I'm sure most of it is old news to the people on this thread, but I
>>> figured I'd post it just in case :)
>>>
>>> On Tue, Jan 30, 2024 at 11:58 AM Josh McKenzie <jm...@apache.org>
>>> wrote:
>>>
>>>
>>> 2.) We should make sure the links between the "known" root causes of
>>> cascading failures and the mechanisms we introduce to avoid them remain
>>> very strong.
>>>
>>> Seems to me that our historical strategy was to address individual known
>>> cases one-by-one rather than looking for a more holistic load-balancing and
>>> load-shedding solution. While the engineer in me likes the elegance of a
>>> broad, more-inclusive *actual SEDA-like* approach, the pragmatist in me
>>> wonders how far we think we are today from a stable set-point.
>>>
>>> i.e. are we facing a handful of cases where nodes can still get pushed
>>> over and then cascade that we can surgically address, or are we facing a
>>> broader lack of back-pressure that rears its head in different domains
>>> (client -> coordinator, coordinator -> replica, internode with other
>>> operations, etc) at surprising times and should be considered more
>>> holistically?
>>>
>>> On Tue, Jan 30, 2024, at 12:31 AM, Caleb Rackliffe wrote:
>>>
>>> I almost forgot CASSANDRA-15817, which introduced
>>> reject_repair_compaction_threshold, which provides a mechanism to stop
>>> repairs while compaction is underwater.
>>>
>>> On Jan 26, 2024, at 6:22 PM, Caleb Rackliffe <ca...@gmail.com>
>>> wrote:
>>>
>>> 
>>> Hey all,
>>>
>>> I'm a bit late to the discussion. I see that we've already discussed
>>> CASSANDRA-15013 <https://issues.apache.org/jira/browse/CASSANDRA-15013>
>>>  and CASSANDRA-16663
>>> <https://issues.apache.org/jira/browse/CASSANDRA-16663> at least in
>>> passing. Having written the latter, I'd be the first to admit it's a crude
>>> tool, although it's been useful here and there, and provides a couple
>>> primitives that may be useful for future work. As Scott mentions, while it
>>> is configurable at runtime, it is not adaptive, although we did
>>> make configuration easier in CASSANDRA-17423
>>> <https://issues.apache.org/jira/browse/CASSANDRA-17423>. It also is
>>> global to the node, although we've lightly discussed some ideas around
>>> making it more granular. (For example, keyspace-based limiting, or limiting
>>> "domains" tagged by the client in requests, could be interesting.) It also
>>> does not deal with inter-node traffic, of course.
>>>
>>> Something we've not yet mentioned (that does address internode traffic)
>>> is CASSANDRA-17324
>>> <https://issues.apache.org/jira/browse/CASSANDRA-17324>, which I
>>> proposed shortly after working on the native request limiter (and have just
>>> not had much time to return to). The basic idea is this:
>>>
>>> When a node is struggling under the weight of a compaction backlog and
>>> becomes a cause of increased read latency for clients, we have two safety
>>> valves:
>>>
>>>
>>> 1.) Disabling the native protocol server, which stops the node from
>>> coordinating reads and writes.
>>> 2.) Jacking up the severity on the node, which tells the dynamic snitch
>>> to avoid the node for reads from other coordinators.
>>>
>>>
>>> These are useful, but we don’t appear to have any mechanism that would
>>> allow us to temporarily reject internode hint, batch, and mutation messages
>>> that could further delay resolution of the compaction backlog.
>>>
>>>
>>> Whether it's done as part of a larger framework or on its own, it still
>>> feels like a good idea.
>>>
>>> Thinking in terms of opportunity costs here (i.e. where we spend our
>>> finite engineering time to holistically improve the experience of operating
>>> this database) is healthy, but we probably haven't reached the point of
>>> diminishing returns on nodes being able to protect themselves from clients
>>> and from other nodes. I would just keep in mind two things:
>>>
>>> 1.) The effectiveness of rate-limiting in the system (which includes the
>>> database and all clients) as a whole necessarily decreases as we move from
>>> the application to the lowest-level database internals. Limiting correctly
>>> at the client will save more resources than limiting at the native protocol
>>> server, and limiting correctly at the native protocol server will save more
>>> resources than limiting after we've dispatched requests to some thread pool
>>> for processing.
>>> 2.) We should make sure the links between the "known" root causes of
>>> cascading failures and the mechanisms we introduce to avoid them remain
>>> very strong.
>>>
>>> In any case, I'd be happy to help out in any way I can as this moves
>>> forward (especially as it relates to our past/current attempts to address
>>> this problem space).
>>>
>>>
>>>
>>>

Re: [EXTERNAL] Re: [Discuss] Generic Purpose Rate Limiter in Cassandra

[Jaydeep Chovatia <ch...@gmail.com>]

Just created an official CEP-41
<https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-41+%28DRAFT%29+Apache+Cassandra+Unified+Rate+Limiter>
incorporating the feedback from this discussion. Feel free to let me know
if I may have missed some important feedback in this thread that is not
captured in the CEP-41.

Jaydeep

On Thu, Feb 22, 2024 at 11:36 AM Jaydeep Chovatia <
chovatia.jaydeep@gmail.com> wrote:

> Thanks, Josh. I will file an official CEP with all the details in a few
> days and update this thread with that CEP number.
> Thanks a lot everyone for providing valuable insights!
>
> Jaydeep
>
> On Thu, Feb 22, 2024 at 9:24 AM Josh McKenzie <jm...@apache.org>
> wrote:
>
>> Do folks think we should file an official CEP and take it there?
>>
>> +1 here.
>>
>> Synthesizing your gdoc, Caleb's work, and the feedback from this thread
>> into a draft seems like a solid next step.
>>
>> On Wed, Feb 7, 2024, at 12:31 PM, Jaydeep Chovatia wrote:
>>
>> I see a lot of great ideas being discussed or proposed in the past to
>> cover the most common rate limiter candidate use cases. Do folks think we
>> should file an official CEP and take it there?
>>
>> Jaydeep
>>
>> On Fri, Feb 2, 2024 at 8:30 AM Caleb Rackliffe <ca...@gmail.com>
>> wrote:
>>
>> I just remembered the other day that I had done a quick writeup on the
>> state of compaction stress-related throttling in the project:
>>
>>
>> https://docs.google.com/document/d/1dfTEcKVidRKC1EWu3SO1kE1iVLMdaJ9uY1WMpS3P_hs/edit?usp=sharing
>>
>> I'm sure most of it is old news to the people on this thread, but I
>> figured I'd post it just in case :)
>>
>> On Tue, Jan 30, 2024 at 11:58 AM Josh McKenzie <jm...@apache.org>
>> wrote:
>>
>>
>> 2.) We should make sure the links between the "known" root causes of
>> cascading failures and the mechanisms we introduce to avoid them remain
>> very strong.
>>
>> Seems to me that our historical strategy was to address individual known
>> cases one-by-one rather than looking for a more holistic load-balancing and
>> load-shedding solution. While the engineer in me likes the elegance of a
>> broad, more-inclusive *actual SEDA-like* approach, the pragmatist in me
>> wonders how far we think we are today from a stable set-point.
>>
>> i.e. are we facing a handful of cases where nodes can still get pushed
>> over and then cascade that we can surgically address, or are we facing a
>> broader lack of back-pressure that rears its head in different domains
>> (client -> coordinator, coordinator -> replica, internode with other
>> operations, etc) at surprising times and should be considered more
>> holistically?
>>
>> On Tue, Jan 30, 2024, at 12:31 AM, Caleb Rackliffe wrote:
>>
>> I almost forgot CASSANDRA-15817, which introduced
>> reject_repair_compaction_threshold, which provides a mechanism to stop
>> repairs while compaction is underwater.
>>
>> On Jan 26, 2024, at 6:22 PM, Caleb Rackliffe <ca...@gmail.com>
>> wrote:
>>
>> 
>> Hey all,
>>
>> I'm a bit late to the discussion. I see that we've already discussed
>> CASSANDRA-15013 <https://issues.apache.org/jira/browse/CASSANDRA-15013>
>>  and CASSANDRA-16663
>> <https://issues.apache.org/jira/browse/CASSANDRA-16663> at least in
>> passing. Having written the latter, I'd be the first to admit it's a crude
>> tool, although it's been useful here and there, and provides a couple
>> primitives that may be useful for future work. As Scott mentions, while it
>> is configurable at runtime, it is not adaptive, although we did
>> make configuration easier in CASSANDRA-17423
>> <https://issues.apache.org/jira/browse/CASSANDRA-17423>. It also is
>> global to the node, although we've lightly discussed some ideas around
>> making it more granular. (For example, keyspace-based limiting, or limiting
>> "domains" tagged by the client in requests, could be interesting.) It also
>> does not deal with inter-node traffic, of course.
>>
>> Something we've not yet mentioned (that does address internode traffic)
>> is CASSANDRA-17324
>> <https://issues.apache.org/jira/browse/CASSANDRA-17324>, which I
>> proposed shortly after working on the native request limiter (and have just
>> not had much time to return to). The basic idea is this:
>>
>> When a node is struggling under the weight of a compaction backlog and
>> becomes a cause of increased read latency for clients, we have two safety
>> valves:
>>
>>
>> 1.) Disabling the native protocol server, which stops the node from
>> coordinating reads and writes.
>> 2.) Jacking up the severity on the node, which tells the dynamic snitch
>> to avoid the node for reads from other coordinators.
>>
>>
>> These are useful, but we don’t appear to have any mechanism that would
>> allow us to temporarily reject internode hint, batch, and mutation messages
>> that could further delay resolution of the compaction backlog.
>>
>>
>> Whether it's done as part of a larger framework or on its own, it still
>> feels like a good idea.
>>
>> Thinking in terms of opportunity costs here (i.e. where we spend our
>> finite engineering time to holistically improve the experience of operating
>> this database) is healthy, but we probably haven't reached the point of
>> diminishing returns on nodes being able to protect themselves from clients
>> and from other nodes. I would just keep in mind two things:
>>
>> 1.) The effectiveness of rate-limiting in the system (which includes the
>> database and all clients) as a whole necessarily decreases as we move from
>> the application to the lowest-level database internals. Limiting correctly
>> at the client will save more resources than limiting at the native protocol
>> server, and limiting correctly at the native protocol server will save more
>> resources than limiting after we've dispatched requests to some thread pool
>> for processing.
>> 2.) We should make sure the links between the "known" root causes of
>> cascading failures and the mechanisms we introduce to avoid them remain
>> very strong.
>>
>> In any case, I'd be happy to help out in any way I can as this moves
>> forward (especially as it relates to our past/current attempts to address
>> this problem space).
>>
>>
>>
>>

Re: [EXTERNAL] Re: [Discuss] Generic Purpose Rate Limiter in Cassandra

[Jaydeep Chovatia <ch...@gmail.com>]

Thanks, Josh. I will file an official CEP with all the details in a few
days and update this thread with that CEP number.
Thanks a lot everyone for providing valuable insights!

Jaydeep

On Thu, Feb 22, 2024 at 9:24 AM Josh McKenzie <jm...@apache.org> wrote:

> Do folks think we should file an official CEP and take it there?
>
> +1 here.
>
> Synthesizing your gdoc, Caleb's work, and the feedback from this thread
> into a draft seems like a solid next step.
>
> On Wed, Feb 7, 2024, at 12:31 PM, Jaydeep Chovatia wrote:
>
> I see a lot of great ideas being discussed or proposed in the past to
> cover the most common rate limiter candidate use cases. Do folks think we
> should file an official CEP and take it there?
>
> Jaydeep
>
> On Fri, Feb 2, 2024 at 8:30 AM Caleb Rackliffe <ca...@gmail.com>
> wrote:
>
> I just remembered the other day that I had done a quick writeup on the
> state of compaction stress-related throttling in the project:
>
>
> https://docs.google.com/document/d/1dfTEcKVidRKC1EWu3SO1kE1iVLMdaJ9uY1WMpS3P_hs/edit?usp=sharing
>
> I'm sure most of it is old news to the people on this thread, but I
> figured I'd post it just in case :)
>
> On Tue, Jan 30, 2024 at 11:58 AM Josh McKenzie <jm...@apache.org>
> wrote:
>
>
> 2.) We should make sure the links between the "known" root causes of
> cascading failures and the mechanisms we introduce to avoid them remain
> very strong.
>
> Seems to me that our historical strategy was to address individual known
> cases one-by-one rather than looking for a more holistic load-balancing and
> load-shedding solution. While the engineer in me likes the elegance of a
> broad, more-inclusive *actual SEDA-like* approach, the pragmatist in me
> wonders how far we think we are today from a stable set-point.
>
> i.e. are we facing a handful of cases where nodes can still get pushed
> over and then cascade that we can surgically address, or are we facing a
> broader lack of back-pressure that rears its head in different domains
> (client -> coordinator, coordinator -> replica, internode with other
> operations, etc) at surprising times and should be considered more
> holistically?
>
> On Tue, Jan 30, 2024, at 12:31 AM, Caleb Rackliffe wrote:
>
> I almost forgot CASSANDRA-15817, which introduced
> reject_repair_compaction_threshold, which provides a mechanism to stop
> repairs while compaction is underwater.
>
> On Jan 26, 2024, at 6:22 PM, Caleb Rackliffe <ca...@gmail.com>
> wrote:
>
> 
> Hey all,
>
> I'm a bit late to the discussion. I see that we've already discussed
> CASSANDRA-15013 <https://issues.apache.org/jira/browse/CASSANDRA-15013>
>  and CASSANDRA-16663
> <https://issues.apache.org/jira/browse/CASSANDRA-16663> at least in
> passing. Having written the latter, I'd be the first to admit it's a crude
> tool, although it's been useful here and there, and provides a couple
> primitives that may be useful for future work. As Scott mentions, while it
> is configurable at runtime, it is not adaptive, although we did
> make configuration easier in CASSANDRA-17423
> <https://issues.apache.org/jira/browse/CASSANDRA-17423>. It also is
> global to the node, although we've lightly discussed some ideas around
> making it more granular. (For example, keyspace-based limiting, or limiting
> "domains" tagged by the client in requests, could be interesting.) It also
> does not deal with inter-node traffic, of course.
>
> Something we've not yet mentioned (that does address internode traffic) is
> CASSANDRA-17324 <https://issues.apache.org/jira/browse/CASSANDRA-17324>,
> which I proposed shortly after working on the native request limiter (and
> have just not had much time to return to). The basic idea is this:
>
> When a node is struggling under the weight of a compaction backlog and
> becomes a cause of increased read latency for clients, we have two safety
> valves:
>
>
> 1.) Disabling the native protocol server, which stops the node from
> coordinating reads and writes.
> 2.) Jacking up the severity on the node, which tells the dynamic snitch to
> avoid the node for reads from other coordinators.
>
>
> These are useful, but we don’t appear to have any mechanism that would
> allow us to temporarily reject internode hint, batch, and mutation messages
> that could further delay resolution of the compaction backlog.
>
>
> Whether it's done as part of a larger framework or on its own, it still
> feels like a good idea.
>
> Thinking in terms of opportunity costs here (i.e. where we spend our
> finite engineering time to holistically improve the experience of operating
> this database) is healthy, but we probably haven't reached the point of
> diminishing returns on nodes being able to protect themselves from clients
> and from other nodes. I would just keep in mind two things:
>
> 1.) The effectiveness of rate-limiting in the system (which includes the
> database and all clients) as a whole necessarily decreases as we move from
> the application to the lowest-level database internals. Limiting correctly
> at the client will save more resources than limiting at the native protocol
> server, and limiting correctly at the native protocol server will save more
> resources than limiting after we've dispatched requests to some thread pool
> for processing.
> 2.) We should make sure the links between the "known" root causes of
> cascading failures and the mechanisms we introduce to avoid them remain
> very strong.
>
> In any case, I'd be happy to help out in any way I can as this moves
> forward (especially as it relates to our past/current attempts to address
> this problem space).
>
>
>
>

Re: [EXTERNAL] Re: [Discuss] Generic Purpose Rate Limiter in Cassandra

[Josh McKenzie <jm...@apache.org>]

> Do folks think we should file an official CEP and take it there?
+1 here.

Synthesizing your gdoc, Caleb's work, and the feedback from this thread into a draft seems like a solid next step.

On Wed, Feb 7, 2024, at 12:31 PM, Jaydeep Chovatia wrote:
> I see a lot of great ideas being discussed or proposed in the past to cover the most common rate limiter candidate use cases. Do folks think we should file an official CEP and take it there?
> 
> Jaydeep
> 
> On Fri, Feb 2, 2024 at 8:30 AM Caleb Rackliffe <ca...@gmail.com> wrote:
>> I just remembered the other day that I had done a quick writeup on the state of compaction stress-related throttling in the project:
>> 
>> https://docs.google.com/document/d/1dfTEcKVidRKC1EWu3SO1kE1iVLMdaJ9uY1WMpS3P_hs/edit?usp=sharing
>> 
>> I'm sure most of it is old news to the people on this thread, but I figured I'd post it just in case :)
>> 
>> On Tue, Jan 30, 2024 at 11:58 AM Josh McKenzie <jm...@apache.org> wrote:
>>> __
>>>> 2.) We should make sure the links between the "known" root causes of cascading failures and the mechanisms we introduce to avoid them remain very strong.
>>> Seems to me that our historical strategy was to address individual known cases one-by-one rather than looking for a more holistic load-balancing and load-shedding solution. While the engineer in me likes the elegance of a broad, more-inclusive *actual SEDA-like* approach, the pragmatist in me wonders how far we think we are today from a stable set-point.
>>> 
>>> i.e. are we facing a handful of cases where nodes can still get pushed over and then cascade that we can surgically address, or are we facing a broader lack of back-pressure that rears its head in different domains (client -> coordinator, coordinator -> replica, internode with other operations, etc) at surprising times and should be considered more holistically?
>>> 
>>> On Tue, Jan 30, 2024, at 12:31 AM, Caleb Rackliffe wrote:
>>>> I almost forgot CASSANDRA-15817, which introduced reject_repair_compaction_threshold, which provides a mechanism to stop repairs while compaction is underwater.
>>>> 
>>>>> On Jan 26, 2024, at 6:22 PM, Caleb Rackliffe <ca...@gmail.com> wrote:
>>>>> 
>>>>> Hey all,
>>>>> 
>>>>> I'm a bit late to the discussion. I see that we've already discussed CASSANDRA-15013 <https://issues.apache.org/jira/browse/CASSANDRA-15013> and CASSANDRA-16663 <https://issues.apache.org/jira/browse/CASSANDRA-16663> at least in passing. Having written the latter, I'd be the first to admit it's a crude tool, although it's been useful here and there, and provides a couple primitives that may be useful for future work. As Scott mentions, while it is configurable at runtime, it is not adaptive, although we did make configuration easier in CASSANDRA-17423 <https://issues.apache.org/jira/browse/CASSANDRA-17423>. It also is global to the node, although we've lightly discussed some ideas around making it more granular. (For example, keyspace-based limiting, or limiting "domains" tagged by the client in requests, could be interesting.) It also does not deal with inter-node traffic, of course.
>>>>> 
>>>>> Something we've not yet mentioned (that does address internode traffic) is CASSANDRA-17324 <https://issues.apache.org/jira/browse/CASSANDRA-17324>, which I proposed shortly after working on the native request limiter (and have just not had much time to return to). The basic idea is this:
>>>>> 
>>>>>> When a node is struggling under the weight of a compaction backlog and becomes a cause of increased read latency for clients, we have two safety valves:
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 1.) Disabling the native protocol server, which stops the node from coordinating reads and writes.
>>>>>> 2.) Jacking up the severity on the node, which tells the dynamic snitch to avoid the node for reads from other coordinators.
>>>>>> 
>>>>>> 
>>>>>> These are useful, but we don’t appear to have any mechanism that would allow us to temporarily reject internode hint, batch, and mutation messages that could further delay resolution of the compaction backlog.
>>>>>> 
>>>>> 
>>>>> Whether it's done as part of a larger framework or on its own, it still feels like a good idea.
>>>>> 
>>>>> Thinking in terms of opportunity costs here (i.e. where we spend our finite engineering time to holistically improve the experience of operating this database) is healthy, but we probably haven't reached the point of diminishing returns on nodes being able to protect themselves from clients and from other nodes. I would just keep in mind two things:
>>>>> 
>>>>> 1.) The effectiveness of rate-limiting in the system (which includes the database and all clients) as a whole necessarily decreases as we move from the application to the lowest-level database internals. Limiting correctly at the client will save more resources than limiting at the native protocol server, and limiting correctly at the native protocol server will save more resources than limiting after we've dispatched requests to some thread pool for processing.
>>>>> 2.) We should make sure the links between the "known" root causes of cascading failures and the mechanisms we introduce to avoid them remain very strong.
>>>>> 
>>>>> In any case, I'd be happy to help out in any way I can as this moves forward (especially as it relates to our past/current attempts to address this problem space).
>>> 

Re: [EXTERNAL] Re: [Discuss] Generic Purpose Rate Limiter in Cassandra

[Jaydeep Chovatia <ch...@gmail.com>]

I see a lot of great ideas being discussed or proposed in the past to cover
the most common rate limiter candidate use cases. Do folks think we should
file an official CEP and take it there?

Jaydeep

On Fri, Feb 2, 2024 at 8:30 AM Caleb Rackliffe <ca...@gmail.com>
wrote:

> I just remembered the other day that I had done a quick writeup on the
> state of compaction stress-related throttling in the project:
>
>
> https://docs.google.com/document/d/1dfTEcKVidRKC1EWu3SO1kE1iVLMdaJ9uY1WMpS3P_hs/edit?usp=sharing
>
> I'm sure most of it is old news to the people on this thread, but I
> figured I'd post it just in case :)
>
> On Tue, Jan 30, 2024 at 11:58 AM Josh McKenzie <jm...@apache.org>
> wrote:
>
>> 2.) We should make sure the links between the "known" root causes of
>> cascading failures and the mechanisms we introduce to avoid them remain
>> very strong.
>>
>> Seems to me that our historical strategy was to address individual known
>> cases one-by-one rather than looking for a more holistic load-balancing and
>> load-shedding solution. While the engineer in me likes the elegance of a
>> broad, more-inclusive *actual SEDA-like* approach, the pragmatist in me
>> wonders how far we think we are today from a stable set-point.
>>
>> i.e. are we facing a handful of cases where nodes can still get pushed
>> over and then cascade that we can surgically address, or are we facing a
>> broader lack of back-pressure that rears its head in different domains
>> (client -> coordinator, coordinator -> replica, internode with other
>> operations, etc) at surprising times and should be considered more
>> holistically?
>>
>> On Tue, Jan 30, 2024, at 12:31 AM, Caleb Rackliffe wrote:
>>
>> I almost forgot CASSANDRA-15817, which introduced
>> reject_repair_compaction_threshold, which provides a mechanism to stop
>> repairs while compaction is underwater.
>>
>> On Jan 26, 2024, at 6:22 PM, Caleb Rackliffe <ca...@gmail.com>
>> wrote:
>>
>> 
>> Hey all,
>>
>> I'm a bit late to the discussion. I see that we've already discussed
>> CASSANDRA-15013 <https://issues.apache.org/jira/browse/CASSANDRA-15013>
>>  and CASSANDRA-16663
>> <https://issues.apache.org/jira/browse/CASSANDRA-16663> at least in
>> passing. Having written the latter, I'd be the first to admit it's a crude
>> tool, although it's been useful here and there, and provides a couple
>> primitives that may be useful for future work. As Scott mentions, while it
>> is configurable at runtime, it is not adaptive, although we did
>> make configuration easier in CASSANDRA-17423
>> <https://issues.apache.org/jira/browse/CASSANDRA-17423>. It also is
>> global to the node, although we've lightly discussed some ideas around
>> making it more granular. (For example, keyspace-based limiting, or limiting
>> "domains" tagged by the client in requests, could be interesting.) It also
>> does not deal with inter-node traffic, of course.
>>
>> Something we've not yet mentioned (that does address internode traffic)
>> is CASSANDRA-17324
>> <https://issues.apache.org/jira/browse/CASSANDRA-17324>, which I
>> proposed shortly after working on the native request limiter (and have just
>> not had much time to return to). The basic idea is this:
>>
>> When a node is struggling under the weight of a compaction backlog and
>> becomes a cause of increased read latency for clients, we have two safety
>> valves:
>>
>> 1.) Disabling the native protocol server, which stops the node from
>> coordinating reads and writes.
>> 2.) Jacking up the severity on the node, which tells the dynamic snitch
>> to avoid the node for reads from other coordinators.
>>
>> These are useful, but we don’t appear to have any mechanism that would
>> allow us to temporarily reject internode hint, batch, and mutation messages
>> that could further delay resolution of the compaction backlog.
>>
>>
>> Whether it's done as part of a larger framework or on its own, it still
>> feels like a good idea.
>>
>> Thinking in terms of opportunity costs here (i.e. where we spend our
>> finite engineering time to holistically improve the experience of operating
>> this database) is healthy, but we probably haven't reached the point of
>> diminishing returns on nodes being able to protect themselves from clients
>> and from other nodes. I would just keep in mind two things:
>>
>> 1.) The effectiveness of rate-limiting in the system (which includes the
>> database and all clients) as a whole necessarily decreases as we move from
>> the application to the lowest-level database internals. Limiting correctly
>> at the client will save more resources than limiting at the native protocol
>> server, and limiting correctly at the native protocol server will save more
>> resources than limiting after we've dispatched requests to some thread pool
>> for processing.
>> 2.) We should make sure the links between the "known" root causes of
>> cascading failures and the mechanisms we introduce to avoid them remain
>> very strong.
>>
>> In any case, I'd be happy to help out in any way I can as this moves
>> forward (especially as it relates to our past/current attempts to address
>> this problem space).
>>
>>
>>

Re: [EXTERNAL] Re: [Discuss] Generic Purpose Rate Limiter in Cassandra

[Caleb Rackliffe <ca...@gmail.com>]

I just remembered the other day that I had done a quick writeup on the
state of compaction stress-related throttling in the project:

https://docs.google.com/document/d/1dfTEcKVidRKC1EWu3SO1kE1iVLMdaJ9uY1WMpS3P_hs/edit?usp=sharing

I'm sure most of it is old news to the people on this thread, but I figured
I'd post it just in case :)

On Tue, Jan 30, 2024 at 11:58 AM Josh McKenzie <jm...@apache.org> wrote:

> 2.) We should make sure the links between the "known" root causes of
> cascading failures and the mechanisms we introduce to avoid them remain
> very strong.
>
> Seems to me that our historical strategy was to address individual known
> cases one-by-one rather than looking for a more holistic load-balancing and
> load-shedding solution. While the engineer in me likes the elegance of a
> broad, more-inclusive *actual SEDA-like* approach, the pragmatist in me
> wonders how far we think we are today from a stable set-point.
>
> i.e. are we facing a handful of cases where nodes can still get pushed
> over and then cascade that we can surgically address, or are we facing a
> broader lack of back-pressure that rears its head in different domains
> (client -> coordinator, coordinator -> replica, internode with other
> operations, etc) at surprising times and should be considered more
> holistically?
>
> On Tue, Jan 30, 2024, at 12:31 AM, Caleb Rackliffe wrote:
>
> I almost forgot CASSANDRA-15817, which introduced
> reject_repair_compaction_threshold, which provides a mechanism to stop
> repairs while compaction is underwater.
>
> On Jan 26, 2024, at 6:22 PM, Caleb Rackliffe <ca...@gmail.com>
> wrote:
>
> 
> Hey all,
>
> I'm a bit late to the discussion. I see that we've already discussed
> CASSANDRA-15013 <https://issues.apache.org/jira/browse/CASSANDRA-15013>
>  and CASSANDRA-16663
> <https://issues.apache.org/jira/browse/CASSANDRA-16663> at least in
> passing. Having written the latter, I'd be the first to admit it's a crude
> tool, although it's been useful here and there, and provides a couple
> primitives that may be useful for future work. As Scott mentions, while it
> is configurable at runtime, it is not adaptive, although we did
> make configuration easier in CASSANDRA-17423
> <https://issues.apache.org/jira/browse/CASSANDRA-17423>. It also is
> global to the node, although we've lightly discussed some ideas around
> making it more granular. (For example, keyspace-based limiting, or limiting
> "domains" tagged by the client in requests, could be interesting.) It also
> does not deal with inter-node traffic, of course.
>
> Something we've not yet mentioned (that does address internode traffic) is
> CASSANDRA-17324 <https://issues.apache.org/jira/browse/CASSANDRA-17324>,
> which I proposed shortly after working on the native request limiter (and
> have just not had much time to return to). The basic idea is this:
>
> When a node is struggling under the weight of a compaction backlog and
> becomes a cause of increased read latency for clients, we have two safety
> valves:
>
> 1.) Disabling the native protocol server, which stops the node from
> coordinating reads and writes.
> 2.) Jacking up the severity on the node, which tells the dynamic snitch to
> avoid the node for reads from other coordinators.
>
> These are useful, but we don’t appear to have any mechanism that would
> allow us to temporarily reject internode hint, batch, and mutation messages
> that could further delay resolution of the compaction backlog.
>
>
> Whether it's done as part of a larger framework or on its own, it still
> feels like a good idea.
>
> Thinking in terms of opportunity costs here (i.e. where we spend our
> finite engineering time to holistically improve the experience of operating
> this database) is healthy, but we probably haven't reached the point of
> diminishing returns on nodes being able to protect themselves from clients
> and from other nodes. I would just keep in mind two things:
>
> 1.) The effectiveness of rate-limiting in the system (which includes the
> database and all clients) as a whole necessarily decreases as we move from
> the application to the lowest-level database internals. Limiting correctly
> at the client will save more resources than limiting at the native protocol
> server, and limiting correctly at the native protocol server will save more
> resources than limiting after we've dispatched requests to some thread pool
> for processing.
> 2.) We should make sure the links between the "known" root causes of
> cascading failures and the mechanisms we introduce to avoid them remain
> very strong.
>
> In any case, I'd be happy to help out in any way I can as this moves
> forward (especially as it relates to our past/current attempts to address
> this problem space).
>
>
>

Re: [EXTERNAL] Re: [Discuss] Generic Purpose Rate Limiter in Cassandra

[Josh McKenzie <jm...@apache.org>]

> 2.) We should make sure the links between the "known" root causes of cascading failures and the mechanisms we introduce to avoid them remain very strong.
Seems to me that our historical strategy was to address individual known cases one-by-one rather than looking for a more holistic load-balancing and load-shedding solution. While the engineer in me likes the elegance of a broad, more-inclusive *actual SEDA-like* approach, the pragmatist in me wonders how far we think we are today from a stable set-point. 

i.e. are we facing a handful of cases where nodes can still get pushed over and then cascade that we can surgically address, or are we facing a broader lack of back-pressure that rears its head in different domains (client -> coordinator, coordinator -> replica, internode with other operations, etc) at surprising times and should be considered more holistically?

On Tue, Jan 30, 2024, at 12:31 AM, Caleb Rackliffe wrote:
> I almost forgot CASSANDRA-15817, which introduced reject_repair_compaction_threshold, which provides a mechanism to stop repairs while compaction is underwater.
> 
>> On Jan 26, 2024, at 6:22 PM, Caleb Rackliffe <ca...@gmail.com> wrote:
>> 
>> Hey all,
>> 
>> I'm a bit late to the discussion. I see that we've already discussed CASSANDRA-15013 <https://issues.apache.org/jira/browse/CASSANDRA-15013> and CASSANDRA-16663 <https://issues.apache.org/jira/browse/CASSANDRA-16663> at least in passing. Having written the latter, I'd be the first to admit it's a crude tool, although it's been useful here and there, and provides a couple primitives that may be useful for future work. As Scott mentions, while it is configurable at runtime, it is not adaptive, although we did make configuration easier in CASSANDRA-17423 <https://issues.apache.org/jira/browse/CASSANDRA-17423>. It also is global to the node, although we've lightly discussed some ideas around making it more granular. (For example, keyspace-based limiting, or limiting "domains" tagged by the client in requests, could be interesting.) It also does not deal with inter-node traffic, of course.
>> 
>> Something we've not yet mentioned (that does address internode traffic) is CASSANDRA-17324 <https://issues.apache.org/jira/browse/CASSANDRA-17324>, which I proposed shortly after working on the native request limiter (and have just not had much time to return to). The basic idea is this:
>> 
>>> When a node is struggling under the weight of a compaction backlog and becomes a cause of increased read latency for clients, we have two safety valves:
>>> 
>>> 
>>> 1.) Disabling the native protocol server, which stops the node from coordinating reads and writes.
>>> 2.) Jacking up the severity on the node, which tells the dynamic snitch to avoid the node for reads from other coordinators.
>>> 
>>> These are useful, but we don’t appear to have any mechanism that would allow us to temporarily reject internode hint, batch, and mutation messages that could further delay resolution of the compaction backlog.
>>> 
>> 
>> Whether it's done as part of a larger framework or on its own, it still feels like a good idea.
>> 
>> Thinking in terms of opportunity costs here (i.e. where we spend our finite engineering time to holistically improve the experience of operating this database) is healthy, but we probably haven't reached the point of diminishing returns on nodes being able to protect themselves from clients and from other nodes. I would just keep in mind two things:
>> 
>> 1.) The effectiveness of rate-limiting in the system (which includes the database and all clients) as a whole necessarily decreases as we move from the application to the lowest-level database internals. Limiting correctly at the client will save more resources than limiting at the native protocol server, and limiting correctly at the native protocol server will save more resources than limiting after we've dispatched requests to some thread pool for processing.
>> 2.) We should make sure the links between the "known" root causes of cascading failures and the mechanisms we introduce to avoid them remain very strong.
>> 
>> In any case, I'd be happy to help out in any way I can as this moves forward (especially as it relates to our past/current attempts to address this problem space).

Re: [EXTERNAL] Re: [Discuss] Generic Purpose Rate Limiter in Cassandra

[Caleb Rackliffe <ca...@gmail.com>]

I almost forgot CASSANDRA-15817, which introduced
reject_repair_compaction_threshold, which provides a mechanism to stop repairs
while compaction is underwater.

  

> On Jan 26, 2024, at 6:22 PM, Caleb Rackliffe <ca...@gmail.com>
> wrote:  
>  
>

> 
>
> Hey all,
>
>  
>
>
> I'm a bit late to the discussion. I see that we've already discussed
> [CASSANDRA-15013](https://issues.apache.org/jira/browse/CASSANDRA-15013) and
> [CASSANDRA-16663](https://issues.apache.org/jira/browse/CASSANDRA-16663) at
> least in passing. Having written the latter, I'd be the first to admit it's
> a crude tool, although it's been useful here and there, and provides a
> couple primitives that may be useful for future work. As Scott mentions,
> while it is configurable at runtime, it is not adaptive, although we did
> make configuration easier in
> [CASSANDRA-17423](https://issues.apache.org/jira/browse/CASSANDRA-17423). It
> also is global to the node, although we've lightly discussed some ideas
> around making it more granular. (For example, keyspace-based limiting, or
> limiting "domains" tagged by the client in requests, could be interesting.)
> It also does not deal with inter-node traffic, of course.
>
>  
>
>
> Something we've not yet mentioned (that does address internode traffic) is
> [CASSANDRA-17324](https://issues.apache.org/jira/browse/CASSANDRA-17324),
> which I proposed shortly after working on the native request limiter (and
> have just not had much time to return to). The basic idea is this:
>
>  
>
>

>> When a node is struggling under the weight of a compaction backlog and
becomes a cause of increased read latency for clients, we have two safety
valves:

>>

>> 1.) Disabling the native protocol server, which stops the node from
coordinating reads and writes.  
> 2.) Jacking up the severity on the node, which tells the dynamic snitch to
> avoid the node for reads from other coordinators.
>>

>> These are useful, but we don’t appear to have any mechanism that would
allow us to temporarily reject internode hint, batch, and mutation messages
that could further delay resolution of the compaction backlog.

>
>  
>
>
> Whether it's done as part of a larger framework or on its own, it still
> feels like a good idea.
>
>  
>
>
> Thinking in terms of opportunity costs here (i.e. where we spend our finite
> engineering time to holistically improve the experience of operating this
> database) is healthy, but we probably haven't reached the point of
> diminishing returns on nodes being able to protect themselves from clients
> and from other nodes. I would just keep in mind two things:
>
>  
>
>
> 1.) The effectiveness of rate-limiting in the system (which includes the
> database and all clients) as a whole necessarily decreases as we move from
> the application to the lowest-level database internals. Limiting correctly
> at the client will save more resources than limiting at the native protocol
> server, and limiting correctly at the native protocol server will save more
> resources than limiting after we've dispatched requests to some thread pool
> for processing.
>
> 2.) We should make sure the links between the "known" root causes of
> cascading failures and the mechanisms we introduce to avoid them remain very
> strong.
>
>  
>
>
> In any case, I'd be happy to help out in any way I can as this moves forward
> (especially as it relates to our past/current attempts to address this
> problem space).

Re: [EXTERNAL] Re: [Discuss] Generic Purpose Rate Limiter in Cassandra

[Caleb Rackliffe <ca...@gmail.com>]

Hey all,

I'm a bit late to the discussion. I see that we've already discussed
CASSANDRA-15013 <https://issues.apache.org/jira/browse/CASSANDRA-15013> and
CASSANDRA-16663 <https://issues.apache.org/jira/browse/CASSANDRA-16663> at
least in passing. Having written the latter, I'd be the first to admit it's
a crude tool, although it's been useful here and there, and provides a
couple primitives that may be useful for future work. As Scott mentions,
while it is configurable at runtime, it is not adaptive, although we did
make configuration easier in CASSANDRA-17423
<https://issues.apache.org/jira/browse/CASSANDRA-17423>. It also is global
to the node, although we've lightly discussed some ideas around making it
more granular. (For example, keyspace-based limiting, or limiting "domains"
tagged by the client in requests, could be interesting.) It also does not
deal with inter-node traffic, of course.

Something we've not yet mentioned (that does address internode traffic) is
CASSANDRA-17324 <https://issues.apache.org/jira/browse/CASSANDRA-17324>,
which I proposed shortly after working on the native request limiter (and
have just not had much time to return to). The basic idea is this:

When a node is struggling under the weight of a compaction backlog and
> becomes a cause of increased read latency for clients, we have two safety
> valves:
>
> 1.) Disabling the native protocol server, which stops the node from
> coordinating reads and writes.
> 2.) Jacking up the severity on the node, which tells the dynamic snitch to
> avoid the node for reads from other coordinators.
>
> These are useful, but we don’t appear to have any mechanism that would
> allow us to temporarily reject internode hint, batch, and mutation messages
> that could further delay resolution of the compaction backlog.
>

Whether it's done as part of a larger framework or on its own, it still
feels like a good idea.

Thinking in terms of opportunity costs here (i.e. where we spend our finite
engineering time to holistically improve the experience of operating this
database) is healthy, but we probably haven't reached the point of
diminishing returns on nodes being able to protect themselves from clients
and from other nodes. I would just keep in mind two things:

1.) The effectiveness of rate-limiting in the system (which includes the
database and all clients) as a whole necessarily decreases as we move from
the application to the lowest-level database internals. Limiting correctly
at the client will save more resources than limiting at the native protocol
server, and limiting correctly at the native protocol server will save more
resources than limiting after we've dispatched requests to some thread pool
for processing.
2.) We should make sure the links between the "known" root causes of
cascading failures and the mechanisms we introduce to avoid them remain
very strong.

In any case, I'd be happy to help out in any way I can as this moves
forward (especially as it relates to our past/current attempts to address
this problem space).

Re: [EXTERNAL] Re: [Discuss] Generic Purpose Rate Limiter in Cassandra

[Jaydeep Chovatia <ch...@gmail.com>]

Hi Jon and Jeff,

First of all sorry for the late reply as I got a bit busy with a production
bug we have discovered in 4.0
https://issues.apache.org/jira/browse/CASSANDRA-17401, please see if you or
someone could prioritize the review of the submitted PR.

Thanks for your valuable feedback!

I just think there's a ton more value in the drivers being able to throttle
requests to deal than server side.

There is no denying that we can do a ton of things on the client side as
far as the rate limiter is concerned, but the server has more fine-grained
indicators, so if we tackle the server-side first, then we will get a lot
of benefits. In an ideal world, we would need rate limiters at various
layers, i.e., at server layer and at the client layers.

Yes, I agree using dropped metrics (errors) is useful, as well as queue
length.  I can't remember offhand all the details of the request queue and
how load shedding works there, I need to go back and look.  If we don't
already have load shedding based on queue depth that seems like an easy
thing to do immediately, and is a high quality signal.  Maybe someone can
remind me if we have that already?

The Cassandra queues, especially, pending reads, mutations, and transport
queues usually indicate some serious slowdowns.

My issue with using CPU to rate limit clients is that I think it's a very
low quality signal, and I suspect it'll trigger a ton of false positives.

In the rate limiter, CPU is not only going to be the only source, but it
will be one of the key indicators. Along with the CPU, we will consider
Cassandra internal queues as well and then make a decision on whether a
server node is running hot or not.

Again, the "Generic" name I have given of this project might be
misleading. This is not the full and final rate limiter implementation for
Cassandra, but one step towards protecting the Cassandra server due to the
slowdowns caused by the clients, such as pushing 10x load, reading/writing
large partitions, tombstones, etc. On purpose, in this design, we are not
touching system activities such as compaction, system traffic, etc. because
they are essential for Cassandra server to stay alive. For sure, we can
enhance this framework in future and make it more smarter and smarter so
that it will be able to predict the cause of the slowdowns and then apply
back pressure depending upon the situations, such as user traffic or
compaction or repair, etc.
But in the first phase, the target is the slowdowns caused by client
traffic, and especially avoid ripple effect on the entire Cassandra server
ring, e.g., a few hot nodes eventually slowing down the entire cluster.

Jaydeep



On Thu, Jan 18, 2024 at 8:24 PM Jon Haddad <jo...@jonhaddad.com> wrote:

> > The problem with generalizing things is if you’re behind on compaction,
> reads get expensive, so you pause compaction completely, you’re SOL and
> you’ll eventually have to throttle traffic to recover
>
> Yeah - there's definitely quite a few ways this can go sideways, and this
> is a good example that won't be consistent across deployments.  There's a
> lot of variables to consider.  I agree that building the machinery for
> operators to make adjustments is the right first step.  Assuming we get all
> the rate limiting options available over CQL and stats available via
> virtual tables, operators can make whatever decisions they feel is best,
> and we'd hopefully get some good feedback about what works well and what
> doesn't.
>
> Jon
>
>
>
>
> On Thu, Jan 18, 2024 at 4:16 PM Jeff Jirsa <jj...@gmail.com> wrote:
>
>> The problem with generalizing things is if you’re behind on compaction,
>> reads get expensive, so you pause compaction completely, you’re SOL and
>> you’ll eventually have to throttle traffic to recover
>>
>> The SEDA model is bad at back pressure and deferred cost makes it
>> non-obvious which resource to slow to ensure stability
>>
>> Just start by exposing it instead of pretending we can outsmart the very
>> complex system
>>
>> On Jan 18, 2024, at 4:56 PM, Jon Haddad <jo...@jonhaddad.com> wrote:
>>
>> 
>> I am definitely +1 on the ability to rate limit operations to tables and
>> keyspaces, and if we can do it at a granular level per user I'm +1 to that
>> as well.  I think this would need to be exposed to the operator regardless
>> of any automatic rate limiter.
>>
>> Thinking about the bigger picture for a minute, I think there's a few
>> things we could throttle dynamically on the server before limiting the
>> client requests.  I've long wanted to see a dynamic rate limiter with
>> compaction and any streaming operation - using resources when they're
>> available but slowing down to allow an influx of requests.  Being able to
>> throttle background operations to free up resources to ensure the DB stays
>> online and healthy would be a big win.
>>
>> > The major challenge with latency based rate limiters is that the
>> latency is subjective from one workload to another.
>>
>> You're absolutely right.  This goes to my other suggestion that
>> client-side rate limiting would be a higher priority (on my list at least)
>> as it is perfectly suited for multiple varying workloads.  Of course, if
>> you're not interested in working on the drivers and only on C* itself, this
>> is a moot point.  You're free to work on whatever you want - I just think
>> there's a ton more value in the drivers being able to throttle requests to
>> deal than server side.
>>
>> > And if these two are +ve then consider the server under pressure. And
>> once it is under the pressure, then shed the traffic from less aggressive
>> to more aggressive, etc. The idea is to prevent Cassandra server from
>> melting (by considering the above two signals to begin with and add any
>> more based on the learnings)
>>
>> Yes, I agree using dropped metrics (errors) is useful, as well as queue
>> length.  I can't remember offhand all the details of the request queue and
>> how load shedding works there, I need to go back and look.  If we don't
>> already have load shedding based on queue depth that seems like an easy
>> thing to do immediately, and is a high quality signal.  Maybe someone can
>> remind me if we have that already?
>>
>> My issue with using CPU to rate limit clients is that I think it's a very
>> low quality signal, and I suspect it'll trigger a ton of false positives.
>> For example, there's a big difference from performance being impacted by
>> repair vs large reads vs backing up a snapshot to an object store, but they
>> have similar effects on the CPU - high I/O, high CPU usage, both sustained
>> over time.  Imo it would be a pretty bad decision to throttle clients when
>> we should be throttling repair instead, and we should only do so if it's
>> actually causing an issue for the client, something CPU usage can't tell
>> us, only the response time and error rates can.
>>
>> In the case of a backup, throttling might make sense, or might not, it
>> really depends on the environment and if backups are happening
>> concurrently.  If a backup's configured with nice +19 (as it should be),
>> I'd consider throttling user requests to be a false positive, potentially
>> one that does more harm than good to the cluster, since the OS should be
>> deprioritizing the backup for us rather than us deprioritizing C*.
>>
>> In my ideal world, if C* detected problematic response times (possibly
>> violating a per-table, target latency time) or query timeouts, it would
>> start by throttling back compactions, repairs, and streaming to ensure
>> client requests can be serviced.  I think we'd need to define the latency
>> targets in order for this to work optimally, b/c you might not want to wait
>> for query timeouts before you throttle.  I think there's a lot of value in
>> dynamically adaptive compaction, repair, and streaming since it would
>> prioritize user requests, but again, if you're not willing to work on that,
>> it's your call.
>>
>> Anyways - I like the idea of putting more safeguards in the database
>> itself, we're fundamentally in agreement there.  I see a ton of value in
>> having flexible rate limiters, whether it be per-table, keyspace, or
>> user+table combination.  I'd also like to ensure the feature doesn't cause
>> more disruptions than it solves, which I think would be the case from using
>> CPU usage as a signal.
>>
>> Jon
>>
>>
>> On Wed, Jan 17, 2024 at 10:26 AM Jaydeep Chovatia <
>> chovatia.jaydeep@gmail.com> wrote:
>>
>>> Jon,
>>>
>>> The major challenge with latency based rate limiters is that the latency
>>> is subjective from one workload to another. As a result, in the proposal I
>>> have described, the idea is to make decision on the following combinations:
>>>
>>>    1. System parameters (such as CPU usage, etc.)
>>>    2. Cassandra thread pools health (are they dropping requests, etc.)
>>>
>>> And if these two are +ve then consider the server under pressure. And
>>> once it is under the pressure, then shed the traffic from less aggressive
>>> to more aggressive, etc. The idea is to prevent Cassandra server from
>>> melting (by considering the above two signals to begin with and add any
>>> more based on the learnings)
>>>
>>> Scott,
>>>
>>> Yes, I did look at some of the implementations, but they are all great
>>> systems and helping quite a lot. But they are still not relying on system
>>> health, etc. and also not in the generic coordinator/replication read/write
>>> path. The idea here is on the similar lines as the existing
>>> implementations, but making it a bit more generic and trying to cover as
>>> many paths as possible.
>>>
>>> German,
>>>
>>> Sure, let's first continue the discussions here. If it turns out that
>>> there is no widespread interest in the idea then we can do 1:1 and see how
>>> we can help each other on a private fork, etc.
>>>
>>> Jaydeep
>>>
>>> On Wed, Jan 17, 2024 at 7:57 AM German Eichberger via dev <
>>> dev@cassandra.apache.org> wrote:
>>>
>>>> Jaydeep,
>>>>
>>>> I concur with Stefan that extensibility of this  should be a design
>>>> goal:
>>>>
>>>>    - It should be easy to add additional metrics (e.g. write queue
>>>>    depth) and decision logic
>>>>    - There should be a way to interact with other systems to signal a
>>>>    resource need  which then could kick off things like scaling
>>>>
>>>>
>>>> Super interested in this and we have been thinking about siimilar
>>>> things internally 😉
>>>>
>>>> Thanks,
>>>> German
>>>> ------------------------------
>>>> *From:* Jaydeep Chovatia <ch...@gmail.com>
>>>> *Sent:* Tuesday, January 16, 2024 1:16 PM
>>>> *To:* dev@cassandra.apache.org <de...@cassandra.apache.org>
>>>> *Subject:* [EXTERNAL] Re: [Discuss] Generic Purpose Rate Limiter in
>>>> Cassandra
>>>>
>>>> You don't often get email from chovatia.jaydeep@gmail.com. Learn why
>>>> this is important <https://aka.ms/LearnAboutSenderIdentification>
>>>> Hi Stefan,
>>>>
>>>> Please find my response below:
>>>> 1) Currently, I am keeping the signals as interface, so one can
>>>> override with a different implementation, but a point noted that even the
>>>> interface APIs could be also made dynamic so one can define APIs and its
>>>> implementation, if they wish to override.
>>>> 2) I've not looked into that yet, but I will look into it and see if it
>>>> can be easily integrated into the Guardrails framework.
>>>> 3) On the server side, when the framework detects that a node is
>>>> overloaded, then it will throw *OverloadedException* back to the
>>>> client. Because if the node while busy continues to serve additional
>>>> requests, then it will slow down other peer nodes due to dependencies on
>>>> meeting the QUORUM, etc. In this, we are at least preventing server nodes
>>>> from melting down, and giving the control to the client via
>>>> *OverloadedException.* Now, it will be up to the client policy, if
>>>> client wishes to retry immediately on a different server node then
>>>> eventually that server node might be impacted, but if client wishes to do
>>>> exponential back off or throw exception back to the application then that
>>>> server node will not be impacted.
>>>>
>>>>
>>>> Jaydeep
>>>>
>>>> On Tue, Jan 16, 2024 at 10:03 AM Štefan Miklošovič <
>>>> stefan.miklosovic@gmail.com> wrote:
>>>>
>>>> Hi Jaydeep,
>>>>
>>>> That seems quite interesting. Couple points though:
>>>>
>>>> 1) It would be nice if there is a way to "subscribe" to decisions your
>>>> detection framework comes up with. Integration with e.g. diagnostics
>>>> subsystem would be beneficial. This should be pluggable - just coding up an
>>>> interface to dump / react on the decisions how I want. This might also act
>>>> as a notifier to other systems, e-mail, slack channels ...
>>>>
>>>> 2) Have you tried to incorporate this with the Guardrails framework? I
>>>> think that if something is detected to be throttled or rejected (e.g
>>>> writing to a table), there might be a guardrail which would be triggered
>>>> dynamically in runtime. Guardrails are useful as such but here we might
>>>> reuse them so we do not need to code it twice.
>>>>
>>>> 3) I am curious how complex this detection framework would be, it can
>>>> be complicated pretty fast I guess. What would be desirable is to act on it
>>>> in such a way that you will not put that node under even more pressure. In
>>>> other words, your detection system should work in such a way that there
>>>> will not be any "doom loop" whereby mere throttling of various parts of
>>>> Cassandra you make it even worse for other nodes in the cluster. For
>>>> example, if a particular node starts to be overwhelmed and you detect this
>>>> and requests start to be rejected, is it not possible that Java driver
>>>> would start to see this node as "erroneous" with delayed response time etc
>>>> and it would start to prefer other nodes in the cluster when deciding what
>>>> node to contact for query coordination? So you would put more load on other
>>>> nodes, making them more susceptible to be throttled as well ...
>>>>
>>>> Regards
>>>>
>>>> Stefan Miklosovic
>>>>
>>>> On Tue, Jan 16, 2024 at 6:41 PM Jaydeep Chovatia <
>>>> chovatia.jaydeep@gmail.com> wrote:
>>>>
>>>> Hi,
>>>>
>>>> Happy New Year!
>>>>
>>>> I would like to discuss the following idea:
>>>>
>>>> Open-source Cassandra (CASSANDRA-15013
>>>> <https://issues.apache.org/jira/browse/CASSANDRA-15013>) has an
>>>> elementary built-in memory rate limiter based on the incoming payload from
>>>> user requests. This rate limiter activates if any incoming user request’s
>>>> payload exceeds certain thresholds. However, the existing rate limiter only
>>>> solves limited-scope issues. Cassandra's server-side meltdown due to
>>>> overload is a known problem. Often we see that a couple of busy nodes take
>>>> down the entire Cassandra ring due to the ripple effect. The following
>>>> document proposes a generic purpose comprehensive rate limiter that works
>>>> considering system signals, such as CPU, and internal signals, such as
>>>> thread pools. The rate limiter will have knobs to filter out internal
>>>> traffic, system traffic, replication traffic, and furthermore based on the
>>>> types of queries.
>>>>
>>>> More design details to this doc: [OSS] Cassandra Generic Purpose Rate
>>>> Limiter - Google Docs
>>>> <https://docs.google.com/document/d/1w-A3fnoeBS6tS1ffBda_R0QR90olzFoMqLE7znFEUrQ/edit>
>>>>
>>>> Please let me know your thoughts.
>>>>
>>>> Jaydeep
>>>>
>>>>

Re: [EXTERNAL] Re: [Discuss] Generic Purpose Rate Limiter in Cassandra

[Jon Haddad <jo...@jonhaddad.com>]

> The problem with generalizing things is if you’re behind on compaction,
reads get expensive, so you pause compaction completely, you’re SOL and
you’ll eventually have to throttle traffic to recover

Yeah - there's definitely quite a few ways this can go sideways, and this
is a good example that won't be consistent across deployments.  There's a
lot of variables to consider.  I agree that building the machinery for
operators to make adjustments is the right first step.  Assuming we get all
the rate limiting options available over CQL and stats available via
virtual tables, operators can make whatever decisions they feel is best,
and we'd hopefully get some good feedback about what works well and what
doesn't.

Jon




On Thu, Jan 18, 2024 at 4:16 PM Jeff Jirsa <jj...@gmail.com> wrote:

> The problem with generalizing things is if you’re behind on compaction,
> reads get expensive, so you pause compaction completely, you’re SOL and
> you’ll eventually have to throttle traffic to recover
>
> The SEDA model is bad at back pressure and deferred cost makes it
> non-obvious which resource to slow to ensure stability
>
> Just start by exposing it instead of pretending we can outsmart the very
> complex system
>
> On Jan 18, 2024, at 4:56 PM, Jon Haddad <jo...@jonhaddad.com> wrote:
>
> 
> I am definitely +1 on the ability to rate limit operations to tables and
> keyspaces, and if we can do it at a granular level per user I'm +1 to that
> as well.  I think this would need to be exposed to the operator regardless
> of any automatic rate limiter.
>
> Thinking about the bigger picture for a minute, I think there's a few
> things we could throttle dynamically on the server before limiting the
> client requests.  I've long wanted to see a dynamic rate limiter with
> compaction and any streaming operation - using resources when they're
> available but slowing down to allow an influx of requests.  Being able to
> throttle background operations to free up resources to ensure the DB stays
> online and healthy would be a big win.
>
> > The major challenge with latency based rate limiters is that the latency
> is subjective from one workload to another.
>
> You're absolutely right.  This goes to my other suggestion that
> client-side rate limiting would be a higher priority (on my list at least)
> as it is perfectly suited for multiple varying workloads.  Of course, if
> you're not interested in working on the drivers and only on C* itself, this
> is a moot point.  You're free to work on whatever you want - I just think
> there's a ton more value in the drivers being able to throttle requests to
> deal than server side.
>
> > And if these two are +ve then consider the server under pressure. And
> once it is under the pressure, then shed the traffic from less aggressive
> to more aggressive, etc. The idea is to prevent Cassandra server from
> melting (by considering the above two signals to begin with and add any
> more based on the learnings)
>
> Yes, I agree using dropped metrics (errors) is useful, as well as queue
> length.  I can't remember offhand all the details of the request queue and
> how load shedding works there, I need to go back and look.  If we don't
> already have load shedding based on queue depth that seems like an easy
> thing to do immediately, and is a high quality signal.  Maybe someone can
> remind me if we have that already?
>
> My issue with using CPU to rate limit clients is that I think it's a very
> low quality signal, and I suspect it'll trigger a ton of false positives.
> For example, there's a big difference from performance being impacted by
> repair vs large reads vs backing up a snapshot to an object store, but they
> have similar effects on the CPU - high I/O, high CPU usage, both sustained
> over time.  Imo it would be a pretty bad decision to throttle clients when
> we should be throttling repair instead, and we should only do so if it's
> actually causing an issue for the client, something CPU usage can't tell
> us, only the response time and error rates can.
>
> In the case of a backup, throttling might make sense, or might not, it
> really depends on the environment and if backups are happening
> concurrently.  If a backup's configured with nice +19 (as it should be),
> I'd consider throttling user requests to be a false positive, potentially
> one that does more harm than good to the cluster, since the OS should be
> deprioritizing the backup for us rather than us deprioritizing C*.
>
> In my ideal world, if C* detected problematic response times (possibly
> violating a per-table, target latency time) or query timeouts, it would
> start by throttling back compactions, repairs, and streaming to ensure
> client requests can be serviced.  I think we'd need to define the latency
> targets in order for this to work optimally, b/c you might not want to wait
> for query timeouts before you throttle.  I think there's a lot of value in
> dynamically adaptive compaction, repair, and streaming since it would
> prioritize user requests, but again, if you're not willing to work on that,
> it's your call.
>
> Anyways - I like the idea of putting more safeguards in the database
> itself, we're fundamentally in agreement there.  I see a ton of value in
> having flexible rate limiters, whether it be per-table, keyspace, or
> user+table combination.  I'd also like to ensure the feature doesn't cause
> more disruptions than it solves, which I think would be the case from using
> CPU usage as a signal.
>
> Jon
>
>
> On Wed, Jan 17, 2024 at 10:26 AM Jaydeep Chovatia <
> chovatia.jaydeep@gmail.com> wrote:
>
>> Jon,
>>
>> The major challenge with latency based rate limiters is that the latency
>> is subjective from one workload to another. As a result, in the proposal I
>> have described, the idea is to make decision on the following combinations:
>>
>>    1. System parameters (such as CPU usage, etc.)
>>    2. Cassandra thread pools health (are they dropping requests, etc.)
>>
>> And if these two are +ve then consider the server under pressure. And
>> once it is under the pressure, then shed the traffic from less aggressive
>> to more aggressive, etc. The idea is to prevent Cassandra server from
>> melting (by considering the above two signals to begin with and add any
>> more based on the learnings)
>>
>> Scott,
>>
>> Yes, I did look at some of the implementations, but they are all great
>> systems and helping quite a lot. But they are still not relying on system
>> health, etc. and also not in the generic coordinator/replication read/write
>> path. The idea here is on the similar lines as the existing
>> implementations, but making it a bit more generic and trying to cover as
>> many paths as possible.
>>
>> German,
>>
>> Sure, let's first continue the discussions here. If it turns out that
>> there is no widespread interest in the idea then we can do 1:1 and see how
>> we can help each other on a private fork, etc.
>>
>> Jaydeep
>>
>> On Wed, Jan 17, 2024 at 7:57 AM German Eichberger via dev <
>> dev@cassandra.apache.org> wrote:
>>
>>> Jaydeep,
>>>
>>> I concur with Stefan that extensibility of this  should be a design goal:
>>>
>>>    - It should be easy to add additional metrics (e.g. write queue
>>>    depth) and decision logic
>>>    - There should be a way to interact with other systems to signal a
>>>    resource need  which then could kick off things like scaling
>>>
>>>
>>> Super interested in this and we have been thinking about siimilar things
>>> internally 😉
>>>
>>> Thanks,
>>> German
>>> ------------------------------
>>> *From:* Jaydeep Chovatia <ch...@gmail.com>
>>> *Sent:* Tuesday, January 16, 2024 1:16 PM
>>> *To:* dev@cassandra.apache.org <de...@cassandra.apache.org>
>>> *Subject:* [EXTERNAL] Re: [Discuss] Generic Purpose Rate Limiter in
>>> Cassandra
>>>
>>> You don't often get email from chovatia.jaydeep@gmail.com. Learn why
>>> this is important <https://aka.ms/LearnAboutSenderIdentification>
>>> Hi Stefan,
>>>
>>> Please find my response below:
>>> 1) Currently, I am keeping the signals as interface, so one can override
>>> with a different implementation, but a point noted that even the interface
>>> APIs could be also made dynamic so one can define APIs and its
>>> implementation, if they wish to override.
>>> 2) I've not looked into that yet, but I will look into it and see if it
>>> can be easily integrated into the Guardrails framework.
>>> 3) On the server side, when the framework detects that a node is
>>> overloaded, then it will throw *OverloadedException* back to the
>>> client. Because if the node while busy continues to serve additional
>>> requests, then it will slow down other peer nodes due to dependencies on
>>> meeting the QUORUM, etc. In this, we are at least preventing server nodes
>>> from melting down, and giving the control to the client via
>>> *OverloadedException.* Now, it will be up to the client policy, if
>>> client wishes to retry immediately on a different server node then
>>> eventually that server node might be impacted, but if client wishes to do
>>> exponential back off or throw exception back to the application then that
>>> server node will not be impacted.
>>>
>>>
>>> Jaydeep
>>>
>>> On Tue, Jan 16, 2024 at 10:03 AM Štefan Miklošovič <
>>> stefan.miklosovic@gmail.com> wrote:
>>>
>>> Hi Jaydeep,
>>>
>>> That seems quite interesting. Couple points though:
>>>
>>> 1) It would be nice if there is a way to "subscribe" to decisions your
>>> detection framework comes up with. Integration with e.g. diagnostics
>>> subsystem would be beneficial. This should be pluggable - just coding up an
>>> interface to dump / react on the decisions how I want. This might also act
>>> as a notifier to other systems, e-mail, slack channels ...
>>>
>>> 2) Have you tried to incorporate this with the Guardrails framework? I
>>> think that if something is detected to be throttled or rejected (e.g
>>> writing to a table), there might be a guardrail which would be triggered
>>> dynamically in runtime. Guardrails are useful as such but here we might
>>> reuse them so we do not need to code it twice.
>>>
>>> 3) I am curious how complex this detection framework would be, it can be
>>> complicated pretty fast I guess. What would be desirable is to act on it in
>>> such a way that you will not put that node under even more pressure. In
>>> other words, your detection system should work in such a way that there
>>> will not be any "doom loop" whereby mere throttling of various parts of
>>> Cassandra you make it even worse for other nodes in the cluster. For
>>> example, if a particular node starts to be overwhelmed and you detect this
>>> and requests start to be rejected, is it not possible that Java driver
>>> would start to see this node as "erroneous" with delayed response time etc
>>> and it would start to prefer other nodes in the cluster when deciding what
>>> node to contact for query coordination? So you would put more load on other
>>> nodes, making them more susceptible to be throttled as well ...
>>>
>>> Regards
>>>
>>> Stefan Miklosovic
>>>
>>> On Tue, Jan 16, 2024 at 6:41 PM Jaydeep Chovatia <
>>> chovatia.jaydeep@gmail.com> wrote:
>>>
>>> Hi,
>>>
>>> Happy New Year!
>>>
>>> I would like to discuss the following idea:
>>>
>>> Open-source Cassandra (CASSANDRA-15013
>>> <https://issues.apache.org/jira/browse/CASSANDRA-15013>) has an
>>> elementary built-in memory rate limiter based on the incoming payload from
>>> user requests. This rate limiter activates if any incoming user request’s
>>> payload exceeds certain thresholds. However, the existing rate limiter only
>>> solves limited-scope issues. Cassandra's server-side meltdown due to
>>> overload is a known problem. Often we see that a couple of busy nodes take
>>> down the entire Cassandra ring due to the ripple effect. The following
>>> document proposes a generic purpose comprehensive rate limiter that works
>>> considering system signals, such as CPU, and internal signals, such as
>>> thread pools. The rate limiter will have knobs to filter out internal
>>> traffic, system traffic, replication traffic, and furthermore based on the
>>> types of queries.
>>>
>>> More design details to this doc: [OSS] Cassandra Generic Purpose Rate
>>> Limiter - Google Docs
>>> <https://docs.google.com/document/d/1w-A3fnoeBS6tS1ffBda_R0QR90olzFoMqLE7znFEUrQ/edit>
>>>
>>> Please let me know your thoughts.
>>>
>>> Jaydeep
>>>
>>>

Re: [EXTERNAL] Re: [Discuss] Generic Purpose Rate Limiter in Cassandra

[Jeff Jirsa <jj...@gmail.com>]

The problem with generalizing things is if you’re behind on compaction, reads
get expensive, so you pause compaction completely, you’re SOL and you’ll
eventually have to throttle traffic to recover

  

The SEDA model is bad at back pressure and deferred cost makes it non-obvious
which resource to slow to ensure stability

  

Just start by exposing it instead of pretending we can outsmart the very
complex system

  

> On Jan 18, 2024, at 4:56 PM, Jon Haddad <jo...@jonhaddad.com> wrote:  
>  
>

> 
>
> I am definitely +1 on the ability to rate limit operations to tables and
> keyspaces, and if we can do it at a granular level per user I'm +1 to that
> as well.  I think this would need to be exposed to the operator regardless
> of any automatic rate limiter.
>
>  
>
>
> Thinking about the bigger picture for a minute, I think there's a few things
> we could throttle dynamically on the server before limiting the client
> requests.  I've long wanted to see a dynamic rate limiter with compaction
> and any streaming operation - using resources when they're available but
> slowing down to allow an influx of requests.  Being able to throttle
> background operations to free up resources to ensure the DB stays online and
> healthy would be a big win.
>
>  
>
>
> > The major challenge with latency based rate limiters is that the latency
> is subjective from one workload to another.  
>  
> You're absolutely right.  This goes to my other suggestion that client-side
> rate limiting would be a higher priority (on my list at least) as it is
> perfectly suited for multiple varying workloads.  Of course, if you're not
> interested in working on the drivers and only on C* itself, this is a moot
> point.  You're free to work on whatever you want - I just think there's a
> ton more value in the drivers being able to throttle requests to deal than
> server side.
>
>  
>
>
> > And if these two are +ve then consider the server under pressure. And once
> it is under the pressure, then shed the traffic from less aggressive to more
> aggressive, etc. The idea is to prevent Cassandra server from melting (by
> considering the above two signals to begin with and add any more based on
> the learnings)
>
>  
>
>
> Yes, I agree using dropped metrics (errors) is useful, as well as queue
> length.  I can't remember offhand all the details of the request queue and
> how load shedding works there, I need to go back and look.  If we don't
> already have load shedding based on queue depth that seems like an easy
> thing to do immediately, and is a high quality signal.  Maybe someone can
> remind me if we have that already?
>
>  
>
>
> My issue with using CPU to rate limit clients is that I think it's a very
> low quality signal, and I suspect it'll trigger a ton of false positives.
> For example, there's a big difference from performance being impacted by
> repair vs large reads vs backing up a snapshot to an object store, but they
> have similar effects on the CPU - high I/O, high CPU usage, both sustained
> over time.  Imo it would be a pretty bad decision to throttle clients when
> we should be throttling repair instead, and we should only do so if it's
> actually causing an issue for the client, something CPU usage can't tell us,
> only the response time and error rates can.  
>
>  
>
>
> In the case of a backup, throttling might make sense, or might not, it
> really depends on the environment and if backups are happening concurrently.
> If a backup's configured with nice +19 (as it should be), I'd consider
> throttling user requests to be a false positive, potentially one that does
> more harm than good to the cluster, since the OS should be deprioritizing
> the backup for us rather than us deprioritizing C*.  
>
>  
>
>
> In my ideal world, if C* detected problematic response times (possibly
> violating a per-table, target latency time) or query timeouts, it would
> start by throttling back compactions, repairs, and streaming to ensure
> client requests can be serviced.  I think we'd need to define the latency
> targets in order for this to work optimally, b/c you might not want to wait
> for query timeouts before you throttle.  I think there's a lot of value in
> dynamically adaptive compaction, repair, and streaming since it would
> prioritize user requests, but again, if you're not willing to work on that,
> it's your call.  
>
>  
>
>
> Anyways - I like the idea of putting more safeguards in the database itself,
> we're fundamentally in agreement there.  I see a ton of value in having
> flexible rate limiters, whether it be per-table, keyspace, or user+table
> combination.  I'd also like to ensure the feature doesn't cause more
> disruptions than it solves, which I think would be the case from using CPU
> usage as a signal.
>
>  
>
>
> Jon
>
>  
>
>
>  
>
>
> On Wed, Jan 17, 2024 at 10:26 AM Jaydeep Chovatia
> <[chovatia.jaydeep@gmail.com](mailto:chovatia.jaydeep@gmail.com)> wrote:  
>
>

>> Jon,

>>

>>  
>
>>

>> The major challenge with latency based rate limiters is that the latency is
subjective from one workload to another. As a result, in the proposal I have
described, the idea is to make decision on the following combinations:

>>

>>   1. System parameters (such as CPU usage, etc.)

>>   2. Cassandra thread pools health (are they dropping requests, etc.)

>>

>>

>> And if these two are +ve then consider the server under pressure. And once
it is under the pressure, then shed the traffic from less aggressive to more
aggressive, etc. The idea is to prevent Cassandra server from melting (by
considering the above two signals to begin with and add any more based on the
learnings)

>>

>>  
>
>>

>> Scott,

>>

>>  
>
>>

>> Yes, I did look at some of the implementations, but they are all great
systems and helping quite a lot. But they are still not relying on system
health, etc. and also not in the generic coordinator/replication read/write
path. The idea here is on the similar lines as the existing implementations,
but making it a bit more generic and trying to cover as many paths as
possible.

>>

>>  
>
>>

>> German,

>>

>>  
>
>>

>> Sure, let's first continue the discussions here. If it turns out that there
is no widespread interest in the idea then we can do 1:1 and see how we can
help each other on a private fork, etc.

>>

>>  
>
>>

>> Jaydeep

>>

>>  
>
>>

>> On Wed, Jan 17, 2024 at 7:57 AM German Eichberger via dev
<[dev@cassandra.apache.org](mailto:dev@cassandra.apache.org)> wrote:  
>
>>

>>> Jaydeep,

>>>

>>>  
>
>>>

>>> I concur with Stefan that extensibility of this  should be a design goal:

>>>

>>>   * It should be easy to add additional metrics (e.g. write queue depth)
and decision logic

>>>   * There should be a way to interact with other systems to signal a
resource need  which then could kick off things like scaling

>>>

>>>

>>>  
>
>>>

>>> Super interested in this and we have been thinking about siimilar things
internally 😉

>>>

>>>  
>
>>>

>>> Thanks,

>>>

>>> German

>>>

>>> * * *

>>>

>>> **From:** Jaydeep Chovatia
<[chovatia.jaydeep@gmail.com](mailto:chovatia.jaydeep@gmail.com)>  
>  **Sent:** Tuesday, January 16, 2024 1:16 PM  
>  **To:** [dev@cassandra.apache.org](mailto:dev@cassandra.apache.org)
> <[dev@cassandra.apache.org](mailto:dev@cassandra.apache.org)>  
>  **Subject:** [EXTERNAL] Re: [Discuss] Generic Purpose Rate Limiter in
> Cassandra
>>>

>>>  
>>>

>>> |

>>>

>>> You don't often get email from
[chovatia.jaydeep@gmail.com](mailto:chovatia.jaydeep@gmail.com). [ Learn why
this is important](https://aka.ms/LearnAboutSenderIdentification)

>>>

>>> |  
>>> ---|---|---  
>>>  
>>> Hi Stefan,  
>
>>>

>>>  
>
>>>

>>> Please find my response below:

>>>

>>> 1) Currently, I am keeping the signals as interface, so one can override
with a different implementation, but a point noted that even the interface
APIs could be also made dynamic so one can define APIs and its implementation,
if they wish to override.

>>>

>>> 2) I've not looked into that yet, but I will look into it and see if it
can be easily integrated into the Guardrails framework.

>>>

>>> 3) On the server side, when the framework detects that a node is
overloaded, then it will throw _OverloadedException_ back to the client.
Because if the node while busy continues to serve additional requests, then it
will slow down other peer nodes due to dependencies on meeting the QUORUM,
etc. In this, we are at least preventing server nodes from melting down, and
giving the control to the client via  _OverloadedException._ Now, it will be
up to the client policy, if client wishes to retry immediately on a different
server node then eventually that server node might be impacted, but if client
wishes to do exponential back off or throw exception back to the application
then that server node will not be impacted.

>>>

>>>  
>
>>>

>>>  
>
>>>

>>> Jaydeep

>>>

>>>  
>
>>>

>>> On Tue, Jan 16, 2024 at 10:03 AM Štefan Miklošovič
<[stefan.miklosovic@gmail.com](mailto:stefan.miklosovic@gmail.com)> wrote:  
>
>>>

>>>> Hi Jaydeep,  
>  
>  That seems quite interesting. Couple points though:  
>  
>  1) It would be nice if there is a way to "subscribe" to decisions your
> detection framework comes up with. Integration with e.g. diagnostics
> subsystem would be beneficial. This should be pluggable - just coding up an
> interface to dump / react on the decisions how I want. This might also act
> as a notifier to other systems, e-mail, slack channels ...
>>>>

>>>>  
>
>>>>

>>>> 2) Have you tried to incorporate this with the Guardrails framework? I
think that if something is detected to be throttled or rejected (e.g writing
to a table), there might be a guardrail which would be triggered dynamically
in runtime. Guardrails are useful as such but here we might reuse them so we
do not need to code it twice.  
>  
>  3) I am curious how complex this detection framework would be, it can be
> complicated pretty fast I guess. What would be desirable is to act on it in
> such a way that you will not put that node under even more pressure. In
> other words, your detection system should work in such a way that there will
> not be any "doom loop" whereby mere throttling of various parts of Cassandra
> you make it even worse for other nodes in the cluster. For example, if a
> particular node starts to be overwhelmed and you detect this and requests
> start to be rejected, is it not possible that Java driver would start to see
> this node as "erroneous" with delayed response time etc and it would start
> to prefer other nodes in the cluster when deciding what node to contact for
> query coordination? So you would put more load on other nodes, making them
> more susceptible to be throttled as well ...  
>
>>>>

>>>>  
>
>>>>

>>>> Regards

>>>>

>>>>  
>
>>>>

>>>> Stefan Miklosovic

>>>>

>>>>  
>
>>>>

>>>> On Tue, Jan 16, 2024 at 6:41 PM Jaydeep Chovatia
<[chovatia.jaydeep@gmail.com](mailto:chovatia.jaydeep@gmail.com)> wrote:  
>
>>>>

>>>>> Hi,

>>>>>

>>>>>  
>
>>>>>

>>>>> Happy New Year!

>>>>>

>>>>>  
>
>>>>>

>>>>> I would like to discuss the following idea:

>>>>>

>>>>>  
>
>>>>>

>>>>> Open-source Cassandra
([CASSANDRA-15013](https://issues.apache.org/jira/browse/CASSANDRA-15013)) has
an elementary built-in memory rate limiter based on the incoming payload from
user requests. This rate limiter activates if any incoming user request’s
payload exceeds certain thresholds. However, the existing rate limiter only
solves limited-scope issues. Cassandra's server-side meltdown due to overload
is a known problem. Often we see that a couple of busy nodes take down the
entire Cassandra ring due to the ripple effect. The following document
proposes a generic purpose comprehensive rate limiter that works considering
system signals, such as CPU, and internal signals, such as thread pools. The
rate limiter will have knobs to filter out internal traffic, system traffic,
replication traffic, and furthermore based on the types of queries.  
>
>>>>>

>>>>>  
>
>>>>>

>>>>> More design details to this doc: [[OSS] Cassandra Generic Purpose Rate
Limiter - Google
Docs](https://docs.google.com/document/d/1w-A3fnoeBS6tS1ffBda_R0QR90olzFoMqLE7znFEUrQ/edit)  
>
>>>>>

>>>>>  
>
>>>>>

>>>>> Please let me know your thoughts.

>>>>>

>>>>>  
>
>>>>>

>>>>> Jaydeep

Re: [EXTERNAL] Re: [Discuss] Generic Purpose Rate Limiter in Cassandra

[Jon Haddad <jo...@jonhaddad.com>]

I am definitely +1 on the ability to rate limit operations to tables and
keyspaces, and if we can do it at a granular level per user I'm +1 to that
as well.  I think this would need to be exposed to the operator regardless
of any automatic rate limiter.

Thinking about the bigger picture for a minute, I think there's a few
things we could throttle dynamically on the server before limiting the
client requests.  I've long wanted to see a dynamic rate limiter with
compaction and any streaming operation - using resources when they're
available but slowing down to allow an influx of requests.  Being able to
throttle background operations to free up resources to ensure the DB stays
online and healthy would be a big win.

> The major challenge with latency based rate limiters is that the latency
is subjective from one workload to another.

You're absolutely right.  This goes to my other suggestion that client-side
rate limiting would be a higher priority (on my list at least) as it is
perfectly suited for multiple varying workloads.  Of course, if you're not
interested in working on the drivers and only on C* itself, this is a moot
point.  You're free to work on whatever you want - I just think there's a
ton more value in the drivers being able to throttle requests to deal than
server side.

> And if these two are +ve then consider the server under pressure. And
once it is under the pressure, then shed the traffic from less aggressive
to more aggressive, etc. The idea is to prevent Cassandra server from
melting (by considering the above two signals to begin with and add any
more based on the learnings)

Yes, I agree using dropped metrics (errors) is useful, as well as queue
length.  I can't remember offhand all the details of the request queue and
how load shedding works there, I need to go back and look.  If we don't
already have load shedding based on queue depth that seems like an easy
thing to do immediately, and is a high quality signal.  Maybe someone can
remind me if we have that already?

My issue with using CPU to rate limit clients is that I think it's a very
low quality signal, and I suspect it'll trigger a ton of false positives.
For example, there's a big difference from performance being impacted by
repair vs large reads vs backing up a snapshot to an object store, but they
have similar effects on the CPU - high I/O, high CPU usage, both sustained
over time.  Imo it would be a pretty bad decision to throttle clients when
we should be throttling repair instead, and we should only do so if it's
actually causing an issue for the client, something CPU usage can't tell
us, only the response time and error rates can.

In the case of a backup, throttling might make sense, or might not, it
really depends on the environment and if backups are happening
concurrently.  If a backup's configured with nice +19 (as it should be),
I'd consider throttling user requests to be a false positive, potentially
one that does more harm than good to the cluster, since the OS should be
deprioritizing the backup for us rather than us deprioritizing C*.

In my ideal world, if C* detected problematic response times (possibly
violating a per-table, target latency time) or query timeouts, it would
start by throttling back compactions, repairs, and streaming to ensure
client requests can be serviced.  I think we'd need to define the latency
targets in order for this to work optimally, b/c you might not want to wait
for query timeouts before you throttle.  I think there's a lot of value in
dynamically adaptive compaction, repair, and streaming since it would
prioritize user requests, but again, if you're not willing to work on that,
it's your call.

Anyways - I like the idea of putting more safeguards in the database
itself, we're fundamentally in agreement there.  I see a ton of value in
having flexible rate limiters, whether it be per-table, keyspace, or
user+table combination.  I'd also like to ensure the feature doesn't cause
more disruptions than it solves, which I think would be the case from using
CPU usage as a signal.

Jon


On Wed, Jan 17, 2024 at 10:26 AM Jaydeep Chovatia <
chovatia.jaydeep@gmail.com> wrote:

> Jon,
>
> The major challenge with latency based rate limiters is that the latency
> is subjective from one workload to another. As a result, in the proposal I
> have described, the idea is to make decision on the following combinations:
>
>    1. System parameters (such as CPU usage, etc.)
>    2. Cassandra thread pools health (are they dropping requests, etc.)
>
> And if these two are +ve then consider the server under pressure. And once
> it is under the pressure, then shed the traffic from less aggressive to
> more aggressive, etc. The idea is to prevent Cassandra server from melting
> (by considering the above two signals to begin with and add any more based
> on the learnings)
>
> Scott,
>
> Yes, I did look at some of the implementations, but they are all great
> systems and helping quite a lot. But they are still not relying on system
> health, etc. and also not in the generic coordinator/replication read/write
> path. The idea here is on the similar lines as the existing
> implementations, but making it a bit more generic and trying to cover as
> many paths as possible.
>
> German,
>
> Sure, let's first continue the discussions here. If it turns out that
> there is no widespread interest in the idea then we can do 1:1 and see how
> we can help each other on a private fork, etc.
>
> Jaydeep
>
> On Wed, Jan 17, 2024 at 7:57 AM German Eichberger via dev <
> dev@cassandra.apache.org> wrote:
>
>> Jaydeep,
>>
>> I concur with Stefan that extensibility of this  should be a design goal:
>>
>>    - It should be easy to add additional metrics (e.g. write queue
>>    depth) and decision logic
>>    - There should be a way to interact with other systems to signal a
>>    resource need  which then could kick off things like scaling
>>
>>
>> Super interested in this and we have been thinking about siimilar things
>> internally 😉
>>
>> Thanks,
>> German
>> ------------------------------
>> *From:* Jaydeep Chovatia <ch...@gmail.com>
>> *Sent:* Tuesday, January 16, 2024 1:16 PM
>> *To:* dev@cassandra.apache.org <de...@cassandra.apache.org>
>> *Subject:* [EXTERNAL] Re: [Discuss] Generic Purpose Rate Limiter in
>> Cassandra
>>
>> You don't often get email from chovatia.jaydeep@gmail.com. Learn why
>> this is important <https://aka.ms/LearnAboutSenderIdentification>
>> Hi Stefan,
>>
>> Please find my response below:
>> 1) Currently, I am keeping the signals as interface, so one can override
>> with a different implementation, but a point noted that even the interface
>> APIs could be also made dynamic so one can define APIs and its
>> implementation, if they wish to override.
>> 2) I've not looked into that yet, but I will look into it and see if it
>> can be easily integrated into the Guardrails framework.
>> 3) On the server side, when the framework detects that a node is
>> overloaded, then it will throw *OverloadedException* back to the client.
>> Because if the node while busy continues to serve additional requests, then
>> it will slow down other peer nodes due to dependencies on meeting the
>> QUORUM, etc. In this, we are at least preventing server nodes from melting
>> down, and giving the control to the client via *OverloadedException.*
>> Now, it will be up to the client policy, if client wishes to retry
>> immediately on a different server node then eventually that server node
>> might be impacted, but if client wishes to do exponential back off or throw
>> exception back to the application then that server node will not be
>> impacted.
>>
>>
>> Jaydeep
>>
>> On Tue, Jan 16, 2024 at 10:03 AM Štefan Miklošovič <
>> stefan.miklosovic@gmail.com> wrote:
>>
>> Hi Jaydeep,
>>
>> That seems quite interesting. Couple points though:
>>
>> 1) It would be nice if there is a way to "subscribe" to decisions your
>> detection framework comes up with. Integration with e.g. diagnostics
>> subsystem would be beneficial. This should be pluggable - just coding up an
>> interface to dump / react on the decisions how I want. This might also act
>> as a notifier to other systems, e-mail, slack channels ...
>>
>> 2) Have you tried to incorporate this with the Guardrails framework? I
>> think that if something is detected to be throttled or rejected (e.g
>> writing to a table), there might be a guardrail which would be triggered
>> dynamically in runtime. Guardrails are useful as such but here we might
>> reuse them so we do not need to code it twice.
>>
>> 3) I am curious how complex this detection framework would be, it can be
>> complicated pretty fast I guess. What would be desirable is to act on it in
>> such a way that you will not put that node under even more pressure. In
>> other words, your detection system should work in such a way that there
>> will not be any "doom loop" whereby mere throttling of various parts of
>> Cassandra you make it even worse for other nodes in the cluster. For
>> example, if a particular node starts to be overwhelmed and you detect this
>> and requests start to be rejected, is it not possible that Java driver
>> would start to see this node as "erroneous" with delayed response time etc
>> and it would start to prefer other nodes in the cluster when deciding what
>> node to contact for query coordination? So you would put more load on other
>> nodes, making them more susceptible to be throttled as well ...
>>
>> Regards
>>
>> Stefan Miklosovic
>>
>> On Tue, Jan 16, 2024 at 6:41 PM Jaydeep Chovatia <
>> chovatia.jaydeep@gmail.com> wrote:
>>
>> Hi,
>>
>> Happy New Year!
>>
>> I would like to discuss the following idea:
>>
>> Open-source Cassandra (CASSANDRA-15013
>> <https://issues.apache.org/jira/browse/CASSANDRA-15013>) has an
>> elementary built-in memory rate limiter based on the incoming payload from
>> user requests. This rate limiter activates if any incoming user request’s
>> payload exceeds certain thresholds. However, the existing rate limiter only
>> solves limited-scope issues. Cassandra's server-side meltdown due to
>> overload is a known problem. Often we see that a couple of busy nodes take
>> down the entire Cassandra ring due to the ripple effect. The following
>> document proposes a generic purpose comprehensive rate limiter that works
>> considering system signals, such as CPU, and internal signals, such as
>> thread pools. The rate limiter will have knobs to filter out internal
>> traffic, system traffic, replication traffic, and furthermore based on the
>> types of queries.
>>
>> More design details to this doc: [OSS] Cassandra Generic Purpose Rate
>> Limiter - Google Docs
>> <https://docs.google.com/document/d/1w-A3fnoeBS6tS1ffBda_R0QR90olzFoMqLE7znFEUrQ/edit>
>>
>> Please let me know your thoughts.
>>
>> Jaydeep
>>
>>

Re: [EXTERNAL] Re: [Discuss] Generic Purpose Rate Limiter in Cassandra

[Jaydeep Chovatia <ch...@gmail.com>]

Jon,

The major challenge with latency based rate limiters is that the latency is
subjective from one workload to another. As a result, in the proposal I
have described, the idea is to make decision on the following combinations:

   1. System parameters (such as CPU usage, etc.)
   2. Cassandra thread pools health (are they dropping requests, etc.)

And if these two are +ve then consider the server under pressure. And once
it is under the pressure, then shed the traffic from less aggressive to
more aggressive, etc. The idea is to prevent Cassandra server from melting
(by considering the above two signals to begin with and add any more based
on the learnings)

Scott,

Yes, I did look at some of the implementations, but they are all great
systems and helping quite a lot. But they are still not relying on system
health, etc. and also not in the generic coordinator/replication read/write
path. The idea here is on the similar lines as the existing
implementations, but making it a bit more generic and trying to cover as
many paths as possible.

German,

Sure, let's first continue the discussions here. If it turns out that there
is no widespread interest in the idea then we can do 1:1 and see how we can
help each other on a private fork, etc.

Jaydeep

On Wed, Jan 17, 2024 at 7:57 AM German Eichberger via dev <
dev@cassandra.apache.org> wrote:

> Jaydeep,
>
> I concur with Stefan that extensibility of this  should be a design goal:
>
>    - It should be easy to add additional metrics (e.g. write queue depth)
>    and decision logic
>    - There should be a way to interact with other systems to signal a
>    resource need  which then could kick off things like scaling
>
>
> Super interested in this and we have been thinking about siimilar things
> internally 😉
>
> Thanks,
> German
> ------------------------------
> *From:* Jaydeep Chovatia <ch...@gmail.com>
> *Sent:* Tuesday, January 16, 2024 1:16 PM
> *To:* dev@cassandra.apache.org <de...@cassandra.apache.org>
> *Subject:* [EXTERNAL] Re: [Discuss] Generic Purpose Rate Limiter in
> Cassandra
>
> You don't often get email from chovatia.jaydeep@gmail.com. Learn why this
> is important <https://aka.ms/LearnAboutSenderIdentification>
> Hi Stefan,
>
> Please find my response below:
> 1) Currently, I am keeping the signals as interface, so one can override
> with a different implementation, but a point noted that even the interface
> APIs could be also made dynamic so one can define APIs and its
> implementation, if they wish to override.
> 2) I've not looked into that yet, but I will look into it and see if it
> can be easily integrated into the Guardrails framework.
> 3) On the server side, when the framework detects that a node is
> overloaded, then it will throw *OverloadedException* back to the client.
> Because if the node while busy continues to serve additional requests, then
> it will slow down other peer nodes due to dependencies on meeting the
> QUORUM, etc. In this, we are at least preventing server nodes from melting
> down, and giving the control to the client via *OverloadedException.*
> Now, it will be up to the client policy, if client wishes to retry
> immediately on a different server node then eventually that server node
> might be impacted, but if client wishes to do exponential back off or throw
> exception back to the application then that server node will not be
> impacted.
>
>
> Jaydeep
>
> On Tue, Jan 16, 2024 at 10:03 AM Štefan Miklošovič <
> stefan.miklosovic@gmail.com> wrote:
>
> Hi Jaydeep,
>
> That seems quite interesting. Couple points though:
>
> 1) It would be nice if there is a way to "subscribe" to decisions your
> detection framework comes up with. Integration with e.g. diagnostics
> subsystem would be beneficial. This should be pluggable - just coding up an
> interface to dump / react on the decisions how I want. This might also act
> as a notifier to other systems, e-mail, slack channels ...
>
> 2) Have you tried to incorporate this with the Guardrails framework? I
> think that if something is detected to be throttled or rejected (e.g
> writing to a table), there might be a guardrail which would be triggered
> dynamically in runtime. Guardrails are useful as such but here we might
> reuse them so we do not need to code it twice.
>
> 3) I am curious how complex this detection framework would be, it can be
> complicated pretty fast I guess. What would be desirable is to act on it in
> such a way that you will not put that node under even more pressure. In
> other words, your detection system should work in such a way that there
> will not be any "doom loop" whereby mere throttling of various parts of
> Cassandra you make it even worse for other nodes in the cluster. For
> example, if a particular node starts to be overwhelmed and you detect this
> and requests start to be rejected, is it not possible that Java driver
> would start to see this node as "erroneous" with delayed response time etc
> and it would start to prefer other nodes in the cluster when deciding what
> node to contact for query coordination? So you would put more load on other
> nodes, making them more susceptible to be throttled as well ...
>
> Regards
>
> Stefan Miklosovic
>
> On Tue, Jan 16, 2024 at 6:41 PM Jaydeep Chovatia <
> chovatia.jaydeep@gmail.com> wrote:
>
> Hi,
>
> Happy New Year!
>
> I would like to discuss the following idea:
>
> Open-source Cassandra (CASSANDRA-15013
> <https://issues.apache.org/jira/browse/CASSANDRA-15013>) has an
> elementary built-in memory rate limiter based on the incoming payload from
> user requests. This rate limiter activates if any incoming user request’s
> payload exceeds certain thresholds. However, the existing rate limiter only
> solves limited-scope issues. Cassandra's server-side meltdown due to
> overload is a known problem. Often we see that a couple of busy nodes take
> down the entire Cassandra ring due to the ripple effect. The following
> document proposes a generic purpose comprehensive rate limiter that works
> considering system signals, such as CPU, and internal signals, such as
> thread pools. The rate limiter will have knobs to filter out internal
> traffic, system traffic, replication traffic, and furthermore based on the
> types of queries.
>
> More design details to this doc: [OSS] Cassandra Generic Purpose Rate
> Limiter - Google Docs
> <https://docs.google.com/document/d/1w-A3fnoeBS6tS1ffBda_R0QR90olzFoMqLE7znFEUrQ/edit>
>
> Please let me know your thoughts.
>
> Jaydeep
>
>