Cassandra JBOD disk configuration

[Hannu Kröger <hk...@gmail.com>]

Hi,

Let's imagine that I have one keyspace with one big table configured with
size tiered compaction strategy and nothing else. The disk configuration
would to have 10x 500GB disks, each mounted to separate directory.

Each directory would then be configured as a separate entry in
cassandra.yaml.

Over time data accumulates and I have at some point 4x 300GB sstables that
the cassandra would like to compact to one 1,2 TB sstable.

Since each directory has max 500GB disk space, that would not work. Right?

Is JBOD with more than 2 disks really usable with STCS? Probably LCS would
the only way to go in this case?

Cheers,
Hannu

Re: Cassandra JBOD disk configuration

[James Briggs <ja...@yahoo.com>]

I've used JBOD before and here's the operational problems I noticed:

1) each volume/disk fills at a different rate, so the min might be 100 GB data,
and the max might be 200 GB.That means you cannot use anywhere near your
real hard disk capacity. (Then on top of that compaction requires space.)

2) when a disk dies you lose that node immediately, whereas with RAID you get
some warning.

Those issues made JBOD unusable for us, but if you're just using Cassandra
as a cache, or your operations team doesn't mind rebuilding nodes all the time
with no advance notice, or your data size is small compared to the disk size,
then it might be ok for you.

Thanks, James Briggs.

________________________________
 From: Chris Lohfink <cl...@blackbirdit.com>
To: user@cassandra.apache.org 
Sent: Tuesday, September 9, 2014 12:14 PM
Subject: Re: Cassandra JBOD disk configuration
 

It can get really unbalanced with STCS.  Whats more is even if there was a disk that could fit the 600gb sstable it doesn't pay attention to space (first) so may pick the 75% full one over the 10% one.  Its a better idea to use LCS with it unless data model really needs it in which case monitor it carefully.  If you want to more completely utilize your disks you will probably just want to use RAID.  I imagine you would get far better performance out of JBOD though... "It Depends"

Chris

On Sep 4, 2014, at 4:48 AM, Hannu Kröger <hk...@gmail.com> wrote:

> Hi,
> 
> Let's imagine that I have one keyspace with one big table configured with size tiered compaction strategy and nothing else. The disk configuration would to have 10x 500GB disks, each mounted to separate directory.
> 
> Each directory would then be configured as a separate entry in cassandra.yaml.
> 
> Over time data accumulates and I have at some point 4x 300GB sstables that the cassandra would like to compact to one 1,2 TB sstable.
> 
> Since each directory has max 500GB disk space, that would not work. Right?
> 
> Is JBOD with more than 2 disks really usable with STCS? Probably LCS would the only way to go in this case?
> 
> Cheers,
> Hannu 

Re: Cassandra JBOD disk configuration

[Chris Lohfink <cl...@blackbirdit.com>]

It can get really unbalanced with STCS.  Whats more is even if there was a disk that could fit the 600gb sstable it doesn't pay attention to space (first) so may pick the 75% full one over the 10% one.  Its a better idea to use LCS with it unless data model really needs it in which case monitor it carefully.  If you want to more completely utilize your disks you will probably just want to use RAID.  I imagine you would get far better performance out of JBOD though... "It Depends"

Chris

On Sep 4, 2014, at 4:48 AM, Hannu Kröger <hk...@gmail.com> wrote:

> Hi,
> 
> Let's imagine that I have one keyspace with one big table configured with size tiered compaction strategy and nothing else. The disk configuration would to have 10x 500GB disks, each mounted to separate directory.
> 
> Each directory would then be configured as a separate entry in cassandra.yaml.
> 
> Over time data accumulates and I have at some point 4x 300GB sstables that the cassandra would like to compact to one 1,2 TB sstable.
> 
> Since each directory has max 500GB disk space, that would not work. Right?
> 
> Is JBOD with more than 2 disks really usable with STCS? Probably LCS would the only way to go in this case?
> 
> Cheers,
> Hannu