You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by onmstester onmstester <on...@zoho.com> on 2021/11/15 05:55:51 UTC
Separating storage and processing
Hi,
In our Cassandra cluster, because of big rows in input data/data model with TTL of several months, we ended up using almost 80% of storage (5TB per node), but having less than 20% of CPU usage which almost all of it would be writing rows to memtables and compacting sstables, so a lot of CPU capacity wasted.
I wonder if there is anything we can do to solve this problem using Cassandra or should migrate from Cassandra to something that separates storage and processing (currently i'm not aware of anything as satble as cassandra)?
Sent using https://www.zoho.com/mail/
Re: Separating storage and processing
Posted by Devopam Mittra <de...@gmail.com>.
Hi there,
Please don't be worried about under utilization of CPU/RAM as it will
increase with active usage of the data in due course.
However, as pointed out already by other members , you may want to relook
at the hardware itself wherein 5TB per node storage with higher CPU+RAM can
rather be reconfigured into a larger number of lower capacity nodes ( less
CPU+RAM, Storage). It will give you more distribution resulting in faster
query response and faster recovery from node failure.
It will be a massive exercise to reorganize but the benefits are surely
more than the effort put in.
Moving to another platform/solution just for this reason isn't perhaps the
best option as you will lose the benefits of cassandra in that platform and
will have to remodel your data from scratch again.
regards
Dev
On Mon, Nov 15, 2021 at 11:30 AM onmstester onmstester <on...@zoho.com>
wrote:
> Hi,
> In our Cassandra cluster, because of big rows in input data/data model
> with TTL of several months, we ended up using almost 80% of storage (5TB
> per node), but having less than 20% of CPU usage which almost all of it
> would be writing rows to memtables and compacting sstables, so a lot of CPU
> capacity wasted.
> I wonder if there is anything we can do to solve this problem using
> Cassandra or should migrate from Cassandra to something that separates
> storage and processing (currently i'm not aware of anything as satble as
> cassandra)?
>
> Sent using Zoho Mail <https://www.zoho.com/mail/>
>
>
>
RE: Separating storage and processing
Posted by "Durity, Sean R" <SE...@homedepot.com>.
We have apps like this, also. For straight Cassandra, I think it is just the nature of how it works. DataStax provides some interesting solutions in different directions: BigNode (for handling 10-20 TB nodes) or Astra (cloud-based/container-driven solution that DOES separate read, write, and storage into separately scaled aspects of Cassandra). I suppose that you could do some similar work on your own with k8cassandra and StarGate.
Sean Durity – Staff Systems Engineer, Cassandra
From: onmstester onmstester <on...@zoho.com>
Sent: Monday, November 15, 2021 12:56 AM
To: user <us...@cassandra.apache.org>
Subject: [EXTERNAL] Separating storage and processing
Hi,
In our Cassandra cluster, because of big rows in input data/data model with TTL of several months, we ended up using almost 80% of storage (5TB per node), but having less than 20% of CPU usage which almost all of it would be writing rows to memtables and compacting sstables, so a lot of CPU capacity wasted.
I wonder if there is anything we can do to solve this problem using Cassandra or should migrate from Cassandra to something that separates storage and processing (currently i'm not aware of anything as satble as cassandra)?
Sent using Zoho Mail [zoho.com]<https://urldefense.com/v3/__https:/www.zoho.com/mail/__;!!M-nmYVHPHQ!aA8YLrua2y_MKLoE-rwgqJ2lmRpWZqA2OVmfzExaUCPBJcsoxCWf6EMMerjdE-RL2sa7MIM$>
INTERNAL USE
Re: Separating storage and processing
Posted by ab...@aber.io.
Another option to consider is changing your SSTable compression. The default is LZ4, which is fast for reads and writes but compressed files have a larger disk footprint. A better alternative might be Zstd, which optimizes for disk footprint. Here’s the full documentation: https://cassandra.apache.org/doc/latest/cassandra/operating/compression.html
This won’t totally alleviate your problem, but it would help, and it might be useful for getting out of a tight situation in the future.
—
Abe
> On Nov 15, 2021, at 10:41 PM, onmstester onmstester <on...@zoho.com> wrote:
>
> Thank You
>
> Sent using Zoho Mail <https://www.zoho.com/mail/>
>
>
> ---- On Tue, 16 Nov 2021 10:00:19 +0330 <ab...@aber.io> wrote ----
>
> > I can, but i thought with 5TB per node already violated best practices (1-2 TB per node) and won't be a good idea to 2X or 3X that?
>
> The main downside of larger disks is that it takes longer to replace a host that goes down, since there’s less network capacity to move data from surviving instances to the new, replacement instances. The longer it takes to replace a host, the longer the time window when further failure may cause unavailability (for example: if you’re running in a 3-instance cluster, one node goes down and requires replacement, any additional nodes going down will cause downtime for reads that require a quorum).
>
> These are some of the main factors to consider here. You can always bump the disk capacity for one instance, measure replacement times, then decide whether to increase disk capacity across the cluster.
>
>
Re: Separating storage and processing
Posted by onmstester onmstester <on...@zoho.com>.
Thank You
Sent using https://www.zoho.com/mail/
---- On Tue, 16 Nov 2021 10:00:19 +0330 <ab...@aber.io> wrote ----
> I can, but i thought with 5TB per node already violated best practices (1-2 TB per node) and won't be a good idea to 2X or 3X that?
The main downside of larger disks is that it takes longer to replace a host that goes down, since there’s less network capacity to move data from surviving instances to the new, replacement instances. The longer it takes to replace a host, the longer the time window when further failure may cause unavailability (for example: if you’re running in a 3-instance cluster, one node goes down and requires replacement, any additional nodes going down will cause downtime for reads that require a quorum).
These are some of the main factors to consider here. You can always bump the disk capacity for one instance, measure replacement times, then decide whether to increase disk capacity across the cluster.
Re: Separating storage and processing
Posted by ab...@aber.io.
> I can, but i thought with 5TB per node already violated best practices (1-2 TB per node) and won't be a good idea to 2X or 3X that?
The main downside of larger disks is that it takes longer to replace a host that goes down, since there’s less network capacity to move data from surviving instances to the new, replacement instances. The longer it takes to replace a host, the longer the time window when further failure may cause unavailability (for example: if you’re running in a 3-instance cluster, one node goes down and requires replacement, any additional nodes going down will cause downtime for reads that require a quorum).
These are some of the main factors to consider here. You can always bump the disk capacity for one instance, measure replacement times, then decide whether to increase disk capacity across the cluster.
Re: Separating storage and processing
Posted by onmstester onmstester <on...@zoho.com>.
I can, but i thought with 5TB per node already violated best practices (1-2 TB per node) and won't be a good idea to 2X or 3X that?
Sent using https://www.zoho.com/mail/
---- On Mon, 15 Nov 2021 20:55:53 +0330 <ab...@aber.io> wrote ----
It sounds like you can downsize your cluster but increase your drive capacity. Depending on how your cluster is deployed, it’s very possible that disks larger than 5TB per node are available. Could you reduce the number of nodes and increase your disk sizes?
—
Abe
Re: Separating storage and processing
Posted by ab...@aber.io.
It sounds like you can downsize your cluster but increase your drive capacity. Depending on how your cluster is deployed, it’s very possible that disks larger than 5TB per node are available. Could you reduce the number of nodes and increase your disk sizes?
—
Abe