Cluster CPU usage limit

[onmstester onmstester <on...@zoho.com>]

IMHO, Cassandra write is more of a CPU bound task, so while determining cluster write throughput, what CPU usage percent (avg among all cluster nodes) should be determined as limit?   Rephrase: what's the normal CPU usage in Cassandra cluster (while no compaction, streaming or heavy-read running) ? For a cluster with 10 nodes, i got 700K write per seconds for my data model, average cpu load is about 40%, i'm going to increase number of native threads (now is 256) and native queue (1024) to increase throughput (and CPU usage subsequently). Sent using Zoho Mail

Fwd: Re: Cluster CPU usage limit

[onmstester onmstester <on...@zoho.com>]

Actually its 256 Native Transport threads, the number of concurrent threads on each node is 32. My main concern is, What amount of CPU capacity should i keep free for tasks other than write that includes compaction and read? Sent using Zoho Mail ============ Forwarded message ============ From : Elliott Sims <el...@backblaze.com> To : <us...@cassandra.apache.org> Date : Fri, 07 Sep 2018 08:05:27 +0430 Subject : Re: Cluster CPU usage limit ============ Forwarded message ============ It's interesting and a bit surprising that 256 write threads isn't enough.  Even with a lot of cores, I'd expect you to be able to saturate CPU with that many threads.  I'd make sure you don't have other bottlenecks, like GC, IOPs, network, or "microbursts" where your load is actually fluctuating between 20-100% CPU. Admittedly, I actually did get best results with 256 threads (and haven't tested higher, but lower is definitely not enough), but every advice I've seen is for a lower write thread count being optimal for most cases. On Thu, Sep 6, 2018 at 5:51 AM, onmstester onmstester <on...@zoho.com> wrote: IMHO, Cassandra write is more of a CPU bound task, so while determining cluster write throughput, what CPU usage percent (avg among all cluster nodes) should be determined as limit?   Rephrase: what's the normal CPU usage in Cassandra cluster (while no compaction, streaming or heavy-read running) ? For a cluster with 10 nodes, i got 700K write per seconds for my data model, average cpu load is about 40%, i'm going to increase number of native threads (now is 256) and native queue (1024) to increase throughput (and CPU usage subsequently). Sent using Zoho Mail

Re: Cluster CPU usage limit

[Elliott Sims <el...@backblaze.com>]

It's interesting and a bit surprising that 256 write threads isn't enough.
Even with a lot of cores, I'd expect you to be able to saturate CPU with
that many threads.  I'd make sure you don't have other bottlenecks, like
GC, IOPs, network, or "microbursts" where your load is actually fluctuating
between 20-100% CPU.
Admittedly, I actually did get best results with 256 threads (and haven't
tested higher, but lower is definitely not enough), but every advice I've
seen is for a lower write thread count being optimal for most cases.

On Thu, Sep 6, 2018 at 5:51 AM, onmstester onmstester <on...@zoho.com>
wrote:

> IMHO, Cassandra write is more of a CPU bound task, so while determining
> cluster write throughput, what CPU usage percent (avg among all cluster
> nodes) should be determined as limit?
> Rephrase: what's the normal CPU usage in Cassandra cluster (while no
> compaction, streaming or heavy-read running) ?
> For a cluster with 10 nodes, i got 700K write per seconds for my data
> model, average cpu load is about 40%, i'm going to increase number of
> native threads (now is 256) and native queue (1024) to increase throughput
> (and CPU usage subsequently).
>
> Sent using Zoho Mail <https://www.zoho.com/mail/>
>
>
>