Best Practices for Managing Concurrent Client Connections in Cassandra

[Naman kaushik <na...@gmail.com>]

Hello Cassandra Community,

We've been experiencing occasional spikes in the number of client
connections to our Cassandra cluster, particularly during high-volume API
request periods. We're using persistent connections, and we've noticed that
the number of connections can increase significantly during these spikes.

We're considering using the following Cassandra parameters to manage
concurrent client connections:

*native_transport_max_concurrent_connections*: This parameter sets the
maximum number of concurrent client connections allowed by the native
transport protocol. Currently, it's set to -1, indicating no limit.

*native_transport_max_concurrent_connections_per_ip*: This parameter sets
the maximum number of concurrent client connections allowed per source IP
address. Like the previous parameter, it's also set to -1.

We're thinking of using these parameters to limit the maximum number of
connections from a single IP address, especially to prevent overwhelming
the database during spikes in API requests that should be handled by our
SOA team exclusively.

Are these parameters suitable for production use, and would implementing
restrictions on concurrent connections per IP be considered a good practice
in managing Cassandra clusters?

Any insights or recommendations would be greatly appreciated.

Thank you!

Naman

Re: Best Practices for Managing Concurrent Client Connections in Cassandra

[Andrew Weaver <an...@gmail.com>]

We've used these settings in production with no issues.

What has been more valuable to us though is limiting the rate of client
connections via iptables. Often times users configure an aggressive
reconnection policy that floods the cluster with connections in certain
circumstances like a node restart or a network glitch.

Are you sure it's the number of connections causing problems or is it the
rate of connections being made? I've seen nodes handle over 40k
connections.

On Thu, Feb 29, 2024, 4:51 AM Naman kaushik <na...@gmail.com> wrote:

> Hello Cassandra Community,
>
> We've been experiencing occasional spikes in the number of client
> connections to our Cassandra cluster, particularly during high-volume API
> request periods. We're using persistent connections, and we've noticed that
> the number of connections can increase significantly during these spikes.
>
> We're considering using the following Cassandra parameters to manage
> concurrent client connections:
>
> *native_transport_max_concurrent_connections*: This parameter sets the
> maximum number of concurrent client connections allowed by the native
> transport protocol. Currently, it's set to -1, indicating no limit.
>
> *native_transport_max_concurrent_connections_per_ip*: This parameter sets
> the maximum number of concurrent client connections allowed per source IP
> address. Like the previous parameter, it's also set to -1.
>
> We're thinking of using these parameters to limit the maximum number of
> connections from a single IP address, especially to prevent overwhelming
> the database during spikes in API requests that should be handled by our
> SOA team exclusively.
>
> Are these parameters suitable for production use, and would implementing
> restrictions on concurrent connections per IP be considered a good practice
> in managing Cassandra clusters?
>
> Any insights or recommendations would be greatly appreciated.
>
> Thank you!
>
> Naman
>

Re: Best Practices for Managing Concurrent Client Connections in Cassandra

[Bowen Song via user <us...@cassandra.apache.org>]

They are suitable for production use for protecting your Cassandra 
server, not the clients. The clients likely will experience an error 
when the limit is reached, and it needs to handle that error appropriately.

What you really want to do probably are:

1. change the client's behaviour, limit the number of servers it 
connects to concurrently. The client can close connections not in use, 
and/or only connect to a subset of servers (note: affects token-aware 
routing).

2. after made the above change, if the number of connections is still an 
issue, horizontally scale up your Cassandra cluster to handle the peak 
number of connections. More nodes means less connections to each node.


On 29/02/2024 10:50, Naman kaushik wrote:
>
> Hello Cassandra Community,
>
> We've been experiencing occasional spikes in the number of client 
> connections to our Cassandra cluster, particularly during high-volume 
> API request periods. We're using persistent connections, and we've 
> noticed that the number of connections can increase significantly 
> during these spikes.
>
> We're considering using the following Cassandra parameters to manage 
> concurrent client connections:
>
> *native_transport_max_concurrent_connections*: This parameter sets the 
> maximum number of concurrent client connections allowed by the native 
> transport protocol. Currently, it's set to -1, indicating no limit.
>
> *native_transport_max_concurrent_connections_per_ip*: This parameter 
> sets the maximum number of concurrent client connections allowed per 
> source IP address. Like the previous parameter, it's also set to -1.
>
> We're thinking of using these parameters to limit the maximum number 
> of connections from a single IP address, especially to prevent 
> overwhelming the database during spikes in API requests that should be 
> handled by our SOA team exclusively.
>
> Are these parameters suitable for production use, and would 
> implementing restrictions on concurrent connections per IP be 
> considered a good practice in managing Cassandra clusters?
>
> Any insights or recommendations would be greatly appreciated.
>
> Thank you!
>
> Naman
>