You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Mark <st...@gmail.com> on 2010/12/09 18:19:24 UTC
Cassandra and disk space
I recently ran into a problem during a repair operation where my nodes
completely ran out of space and my whole cluster was... well,
clusterfucked.
I want to make sure how to prevent this problem in the future.
Should I make sure that at all times every node is under 50% of its disk
space? Are there any normal day-to-day operations that would cause the
any one node to double in size that I should be aware of? If on or more
nodes to surpass the 50% mark, what should I plan to do?
Thanks for any advice
Re: Cassandra and disk space
Posted by Scott Dworkis <sv...@mylife.com>.
i recently finished a practice expansion of 4 nodes to 5 nodes, a series
of "nodetool move", "nodetool cleanup" and jmx gc steps. i found that in
some of the steps, disk usage actually grew to 2.5x the base data size on
one of the nodes. i'm using 0.6.4.
-scott
On Thu, 9 Dec 2010, Rustam Aliyev wrote:
> Is there any plans to improve this in future?
>
> For big data clusters this could be very expensive. Based on your comment, I will need 200TB of storage for 100TB of data to keep Cassandra running.
>
> --
> Rustam.
>
> On 09/12/2010 17:56, Tyler Hobbs wrote:
> If you are on 0.6, repair is particularly dangerous with respect to disk space usage. If your replica is sufficiently out of sync, you can
> triple your disk usage pretty easily. This has been improved in 0.7, so repairs should use about half as much disk space, on average.
>
> In general, yes, keep your nodes under 50% disk usage at all times. Any of: compaction, cleanup, snapshotting, repair, or bootstrapping (the
> latter two are improved in 0.7) can double your disk usage temporarily.
>
> You should plan to add more disk space or add nodes when you get close to this limit. Once you go over 50%, it's more difficult to add nodes,
> at least in 0.6.
>
> - Tyler
>
> On Thu, Dec 9, 2010 at 11:19 AM, Mark <st...@gmail.com> wrote:
> I recently ran into a problem during a repair operation where my nodes completely ran out of space and my whole cluster was...
> well, clusterfucked.
>
> I want to make sure how to prevent this problem in the future.
>
> Should I make sure that at all times every node is under 50% of its disk space? Are there any normal day-to-day operations that
> would cause the any one node to double in size that I should be aware of? If on or more nodes to surpass the 50% mark, what should
> I plan to do?
>
> Thanks for any advice
>
>
>
>
Re: Cassandra and disk space
Posted by Bill de hÓra <bi...@dehora.net>.
This is true, but for larger installations I end up needing more
servers to hold the disks, more racks to hold the servers the point
where the overall cost per GB climbs (granted the cost per IOP is
probably still good). AIUI, a chunk of that 50% is replicated data such
that the truly available space in the cluster is lower than 50% when
capacity planning? If so, for some workloads where it's just data
pouring in with very few updates, would have me thinking I'd want a
tiered model, archiving cold data onto a filer/hdfs.
Bill
On Thu, 2010-12-09 at 13:26 -0600, Tyler Hobbs wrote:
> That depends on your scenario. In the worst case of one big CF,
> there's not much that can be easily done for the disk usage of
> compaction and cleanup (which is essentially compaction).
>
> If, instead, you have several column families and no single CF makes
> up the majority of your data, you can push your disk usage a bit
> higher.
>
> A fundamental idea behind Cassandra's architecture is that disk space
> is cheap (which, indeed, it is). If you are particularly sensitive to
> this, Cassandra might not be the best solution to your problem. Also
> keep in mind that Cassandra performs well with average disks, so you
> don't need to spend a lot there. Additionally, most people find that
> the replication protects their data enough to allow them to use RAID 0
> instead of 1, 10, 5, or 6.
>
> - Tyler
>
>
> On Thu, Dec 9, 2010 at 12:20 PM, Rustam Aliyev <ru...@code.az> wrote:
>
> Is there any plans to improve this in future?
>
> For big data clusters this could be very expensive. Based on
> your comment, I will need 200TB of storage for 100TB of data
> to keep Cassandra running.
>
> --
> Rustam.
>
>
>
> On 09/12/2010 17:56, Tyler Hobbs wrote:
>
> > If you are on 0.6, repair is particularly dangerous with
> > respect to disk space usage. If your replica is
> > sufficiently out of sync, you can triple your disk usage
> > pretty easily. This has been improved in 0.7, so repairs
> > should use about half as much disk space, on average.
> >
> > In general, yes, keep your nodes under 50% disk usage at all
> > times. Any of: compaction, cleanup, snapshotting, repair,
> > or bootstrapping (the latter two are improved in 0.7) can
> > double your disk usage temporarily.
> >
> > You should plan to add more disk space or add nodes when you
> > get close to this limit. Once you go over 50%, it's more
> > difficult to add nodes, at least in 0.6.
> >
> > - Tyler
> >
> >
> > On Thu, Dec 9, 2010 at 11:19 AM, Mark
> > <st...@gmail.com> wrote:
> >
> > I recently ran into a problem during a repair
> > operation where my nodes completely ran out of space
> > and my whole cluster was... well, clusterfucked.
> >
> > I want to make sure how to prevent this problem in
> > the future.
> >
> > Should I make sure that at all times every node is
> > under 50% of its disk space? Are there any normal
> > day-to-day operations that would cause the any one
> > node to double in size that I should be aware of? If
> > on or more nodes to surpass the 50% mark, what
> > should I plan to do?
> >
> > Thanks for any advice
> >
> >
>
>
Re: Cassandra and disk space
Posted by Nick Bailey <ni...@riptano.com>.
Additionally, cleanup will fail to run when the disk is more than 50% full.
Another reason to stay below 50%.
On Thu, Dec 9, 2010 at 6:03 PM, Tyler Hobbs <ty...@riptano.com> wrote:
> Yes, that's correct, but I wouldn't push it too far. You'll become much
> more sensitive to disk usage changes; in particular, rebalancing your
> cluster will particularly difficult, and repair will also become dangerous.
> Disk performance also tends to drop when a disk nears capacity.
>
> There's no recommended maximum size -- it all depends on your access
> rates. Anywhere from 10 GB to 1TB is typical.
>
> - Tyler
>
>
> On Thu, Dec 9, 2010 at 5:52 PM, Rustam Aliyev <ru...@code.az> wrote:
>
>>
>> That depends on your scenario. In the worst case of one big CF, there's
>> not much that can be easily done for the disk usage of compaction and
>> cleanup (which is essentially compaction).
>>
>> If, instead, you have several column families and no single CF makes up
>> the majority of your data, you can push your disk usage a bit higher.
>>
>>
>> Is there any formula to calculate this? Let's say I have 500GB in single
>> CF. So I need at least 500GB of free space for compaction. If I partition
>> this CF and split it into 10 proportional CFs each 50GB, does it mean that I
>> will need only 50GB of free space?
>>
>> Also, is there recommended maximum of data size per node?
>>
>> Thanks.
>>
>>
>> A fundamental idea behind Cassandra's architecture is that disk space is
>> cheap (which, indeed, it is). If you are particularly sensitive to this,
>> Cassandra might not be the best solution to your problem. Also keep in mind
>> that Cassandra performs well with average disks, so you don't need to spend
>> a lot there. Additionally, most people find that the replication protects
>> their data enough to allow them to use RAID 0 instead of 1, 10, 5, or 6.
>>
>> - Tyler
>>
>> On Thu, Dec 9, 2010 at 12:20 PM, Rustam Aliyev <ru...@code.az> wrote:
>>
>>> Is there any plans to improve this in future?
>>>
>>> For big data clusters this could be very expensive. Based on your
>>> comment, I will need 200TB of storage for 100TB of data to keep Cassandra
>>> running.
>>>
>>> --
>>> Rustam.
>>>
>>> On 09/12/2010 17:56, Tyler Hobbs wrote:
>>>
>>> If you are on 0.6, repair is particularly dangerous with respect to disk
>>> space usage. If your replica is sufficiently out of sync, you can triple
>>> your disk usage pretty easily. This has been improved in 0.7, so repairs
>>> should use about half as much disk space, on average.
>>>
>>> In general, yes, keep your nodes under 50% disk usage at all times. Any
>>> of: compaction, cleanup, snapshotting, repair, or bootstrapping (the latter
>>> two are improved in 0.7) can double your disk usage temporarily.
>>>
>>> You should plan to add more disk space or add nodes when you get close to
>>> this limit. Once you go over 50%, it's more difficult to add nodes, at
>>> least in 0.6.
>>>
>>> - Tyler
>>>
>>> On Thu, Dec 9, 2010 at 11:19 AM, Mark <st...@gmail.com> wrote:
>>>
>>>> I recently ran into a problem during a repair operation where my nodes
>>>> completely ran out of space and my whole cluster was... well, clusterfucked.
>>>>
>>>> I want to make sure how to prevent this problem in the future.
>>>>
>>>> Should I make sure that at all times every node is under 50% of its disk
>>>> space? Are there any normal day-to-day operations that would cause the any
>>>> one node to double in size that I should be aware of? If on or more nodes to
>>>> surpass the 50% mark, what should I plan to do?
>>>>
>>>> Thanks for any advice
>>>>
>>>
>>>
>>
>
Re: Cassandra and disk space
Posted by Robert Coli <rc...@digg.com>.
On Thu, Dec 9, 2010 at 4:20 PM, Rustam Aliyev <ru...@code.az> wrote:
> Thanks Tyler, this is really useful.
> [ RAID0 vs JBOD question ]
> In other words, does splitting data folder into smaller ones bring any
> performance or stability advantages?
This is getting to be a FAQ, so here's my stock answer :
There are non-zero production deployments which have experienced fail
from multiple data directories in cassandra.
There are zero production deployments which have experienced win from
multiple data directories in cassandra.
YMMV, of course!
=Rob
PS - Maybe we should remove the multiple data directory stuff, so
people don't keep getting tempted to use it?
Re: Cassandra and disk space
Posted by Brandon Williams <dr...@gmail.com>.
On Thu, Dec 9, 2010 at 6:20 PM, Rustam Aliyev <ru...@code.az> wrote:
> Also, I noticed that you can specify multiple data file directories located
> on different disks. Let's say if I have machine with 4 x 500GB drives, what
> would be the difference between following 2 setups:
>
> 1. each drive mounted separately and has data file dirs on it (so 4x
> data file dirs)
> 2. disks are in RAID0 and mounted as one drive with one data folder on
> it
>
> In other words, does splitting data folder into smaller ones bring any
> performance or stability advantages?
>
It brings disadvantages. Your largest CF will be limited to the size of
your smallest drive, and you won't be using them in parallel when
compacting. RAID0 is the better option.
-Brandon
Re: Cassandra and disk space
Posted by Rustam Aliyev <ru...@code.az>.
Thanks Tyler, this is really useful.
Also, I noticed that you can specify multiple data file directories
located on different disks. Let's say if I have machine with 4 x 500GB
drives, what would be the difference between following 2 setups:
1. each drive mounted separately and has data file dirs on it (so 4x
data file dirs)
2. disks are in RAID0 and mounted as one drive with one data folder on it
In other words, does splitting data folder into smaller ones bring any
performance or stability advantages?
On 10/12/2010 00:03, Tyler Hobbs wrote:
> Yes, that's correct, but I wouldn't push it too far. You'll become
> much more sensitive to disk usage changes; in particular, rebalancing
> your cluster will particularly difficult, and repair will also become
> dangerous. Disk performance also tends to drop when a disk nears
> capacity.
>
> There's no recommended maximum size -- it all depends on your access
> rates. Anywhere from 10 GB to 1TB is typical.
>
> - Tyler
>
> On Thu, Dec 9, 2010 at 5:52 PM, Rustam Aliyev <rustam@code.az
> <ma...@code.az>> wrote:
>
>
>> That depends on your scenario. In the worst case of one big CF,
>> there's not much that can be easily done for the disk usage of
>> compaction and cleanup (which is essentially compaction).
>>
>> If, instead, you have several column families and no single CF
>> makes up the majority of your data, you can push your disk usage
>> a bit higher.
>>
>
> Is there any formula to calculate this? Let's say I have 500GB in
> single CF. So I need at least 500GB of free space for compaction.
> If I partition this CF and split it into 10 proportional CFs each
> 50GB, does it mean that I will need only 50GB of free space?
>
> Also, is there recommended maximum of data size per node?
>
> Thanks.
>
>
>> A fundamental idea behind Cassandra's architecture is that disk
>> space is cheap (which, indeed, it is). If you are particularly
>> sensitive to this, Cassandra might not be the best solution to
>> your problem. Also keep in mind that Cassandra performs well
>> with average disks, so you don't need to spend a lot there.
>> Additionally, most people find that the replication protects
>> their data enough to allow them to use RAID 0 instead of 1, 10,
>> 5, or 6.
>>
>> - Tyler
>>
>> On Thu, Dec 9, 2010 at 12:20 PM, Rustam Aliyev <rustam@code.az
>> <ma...@code.az>> wrote:
>>
>> Is there any plans to improve this in future?
>>
>> For big data clusters this could be very expensive. Based on
>> your comment, I will need 200TB of storage for 100TB of data
>> to keep Cassandra running.
>>
>> --
>> Rustam.
>>
>> On 09/12/2010 17:56, Tyler Hobbs wrote:
>>> If you are on 0.6, repair is particularly dangerous with
>>> respect to disk space usage. If your replica is
>>> sufficiently out of sync, you can triple your disk usage
>>> pretty easily. This has been improved in 0.7, so repairs
>>> should use about half as much disk space, on average.
>>>
>>> In general, yes, keep your nodes under 50% disk usage at all
>>> times. Any of: compaction, cleanup, snapshotting, repair,
>>> or bootstrapping (the latter two are improved in 0.7) can
>>> double your disk usage temporarily.
>>>
>>> You should plan to add more disk space or add nodes when you
>>> get close to this limit. Once you go over 50%, it's more
>>> difficult to add nodes, at least in 0.6.
>>>
>>> - Tyler
>>>
>>> On Thu, Dec 9, 2010 at 11:19 AM, Mark
>>> <static.void.dev@gmail.com
>>> <ma...@gmail.com>> wrote:
>>>
>>> I recently ran into a problem during a repair operation
>>> where my nodes completely ran out of space and my whole
>>> cluster was... well, clusterfucked.
>>>
>>> I want to make sure how to prevent this problem in the
>>> future.
>>>
>>> Should I make sure that at all times every node is under
>>> 50% of its disk space? Are there any normal day-to-day
>>> operations that would cause the any one node to double
>>> in size that I should be aware of? If on or more nodes
>>> to surpass the 50% mark, what should I plan to do?
>>>
>>> Thanks for any advice
>>>
>>>
>>
>
Re: Cassandra and disk space
Posted by Tyler Hobbs <ty...@riptano.com>.
Yes, that's correct, but I wouldn't push it too far. You'll become much
more sensitive to disk usage changes; in particular, rebalancing your
cluster will particularly difficult, and repair will also become dangerous.
Disk performance also tends to drop when a disk nears capacity.
There's no recommended maximum size -- it all depends on your access rates.
Anywhere from 10 GB to 1TB is typical.
- Tyler
On Thu, Dec 9, 2010 at 5:52 PM, Rustam Aliyev <ru...@code.az> wrote:
>
> That depends on your scenario. In the worst case of one big CF, there's
> not much that can be easily done for the disk usage of compaction and
> cleanup (which is essentially compaction).
>
> If, instead, you have several column families and no single CF makes up the
> majority of your data, you can push your disk usage a bit higher.
>
>
> Is there any formula to calculate this? Let's say I have 500GB in single
> CF. So I need at least 500GB of free space for compaction. If I partition
> this CF and split it into 10 proportional CFs each 50GB, does it mean that I
> will need only 50GB of free space?
>
> Also, is there recommended maximum of data size per node?
>
> Thanks.
>
>
> A fundamental idea behind Cassandra's architecture is that disk space is
> cheap (which, indeed, it is). If you are particularly sensitive to this,
> Cassandra might not be the best solution to your problem. Also keep in mind
> that Cassandra performs well with average disks, so you don't need to spend
> a lot there. Additionally, most people find that the replication protects
> their data enough to allow them to use RAID 0 instead of 1, 10, 5, or 6.
>
> - Tyler
>
> On Thu, Dec 9, 2010 at 12:20 PM, Rustam Aliyev <ru...@code.az> wrote:
>
>> Is there any plans to improve this in future?
>>
>> For big data clusters this could be very expensive. Based on your comment,
>> I will need 200TB of storage for 100TB of data to keep Cassandra running.
>>
>> --
>> Rustam.
>>
>> On 09/12/2010 17:56, Tyler Hobbs wrote:
>>
>> If you are on 0.6, repair is particularly dangerous with respect to disk
>> space usage. If your replica is sufficiently out of sync, you can triple
>> your disk usage pretty easily. This has been improved in 0.7, so repairs
>> should use about half as much disk space, on average.
>>
>> In general, yes, keep your nodes under 50% disk usage at all times. Any
>> of: compaction, cleanup, snapshotting, repair, or bootstrapping (the latter
>> two are improved in 0.7) can double your disk usage temporarily.
>>
>> You should plan to add more disk space or add nodes when you get close to
>> this limit. Once you go over 50%, it's more difficult to add nodes, at
>> least in 0.6.
>>
>> - Tyler
>>
>> On Thu, Dec 9, 2010 at 11:19 AM, Mark <st...@gmail.com> wrote:
>>
>>> I recently ran into a problem during a repair operation where my nodes
>>> completely ran out of space and my whole cluster was... well, clusterfucked.
>>>
>>> I want to make sure how to prevent this problem in the future.
>>>
>>> Should I make sure that at all times every node is under 50% of its disk
>>> space? Are there any normal day-to-day operations that would cause the any
>>> one node to double in size that I should be aware of? If on or more nodes to
>>> surpass the 50% mark, what should I plan to do?
>>>
>>> Thanks for any advice
>>>
>>
>>
>
Re: Cassandra and disk space
Posted by Rustam Aliyev <ru...@code.az>.
> That depends on your scenario. In the worst case of one big CF,
> there's not much that can be easily done for the disk usage of
> compaction and cleanup (which is essentially compaction).
>
> If, instead, you have several column families and no single CF makes
> up the majority of your data, you can push your disk usage a bit higher.
>
Is there any formula to calculate this? Let's say I have 500GB in single
CF. So I need at least 500GB of free space for compaction. If I
partition this CF and split it into 10 proportional CFs each 50GB, does
it mean that I will need only 50GB of free space?
Also, is there recommended maximum of data size per node?
Thanks.
> A fundamental idea behind Cassandra's architecture is that disk space
> is cheap (which, indeed, it is). If you are particularly sensitive to
> this, Cassandra might not be the best solution to your problem. Also
> keep in mind that Cassandra performs well with average disks, so you
> don't need to spend a lot there. Additionally, most people find that
> the replication protects their data enough to allow them to use RAID 0
> instead of 1, 10, 5, or 6.
>
> - Tyler
>
> On Thu, Dec 9, 2010 at 12:20 PM, Rustam Aliyev <rustam@code.az
> <ma...@code.az>> wrote:
>
> Is there any plans to improve this in future?
>
> For big data clusters this could be very expensive. Based on your
> comment, I will need 200TB of storage for 100TB of data to keep
> Cassandra running.
>
> --
> Rustam.
>
> On 09/12/2010 17:56, Tyler Hobbs wrote:
>> If you are on 0.6, repair is particularly dangerous with respect
>> to disk space usage. If your replica is sufficiently out of
>> sync, you can triple your disk usage pretty easily. This has
>> been improved in 0.7, so repairs should use about half as much
>> disk space, on average.
>>
>> In general, yes, keep your nodes under 50% disk usage at all
>> times. Any of: compaction, cleanup, snapshotting, repair, or
>> bootstrapping (the latter two are improved in 0.7) can double
>> your disk usage temporarily.
>>
>> You should plan to add more disk space or add nodes when you get
>> close to this limit. Once you go over 50%, it's more difficult
>> to add nodes, at least in 0.6.
>>
>> - Tyler
>>
>> On Thu, Dec 9, 2010 at 11:19 AM, Mark <static.void.dev@gmail.com
>> <ma...@gmail.com>> wrote:
>>
>> I recently ran into a problem during a repair operation where
>> my nodes completely ran out of space and my whole cluster
>> was... well, clusterfucked.
>>
>> I want to make sure how to prevent this problem in the future.
>>
>> Should I make sure that at all times every node is under 50%
>> of its disk space? Are there any normal day-to-day operations
>> that would cause the any one node to double in size that I
>> should be aware of? If on or more nodes to surpass the 50%
>> mark, what should I plan to do?
>>
>> Thanks for any advice
>>
>>
>
Re: Cassandra and disk space
Posted by Tyler Hobbs <ty...@riptano.com>.
That depends on your scenario. In the worst case of one big CF, there's not
much that can be easily done for the disk usage of compaction and cleanup
(which is essentially compaction).
If, instead, you have several column families and no single CF makes up the
majority of your data, you can push your disk usage a bit higher.
A fundamental idea behind Cassandra's architecture is that disk space is
cheap (which, indeed, it is). If you are particularly sensitive to this,
Cassandra might not be the best solution to your problem. Also keep in mind
that Cassandra performs well with average disks, so you don't need to spend
a lot there. Additionally, most people find that the replication protects
their data enough to allow them to use RAID 0 instead of 1, 10, 5, or 6.
- Tyler
On Thu, Dec 9, 2010 at 12:20 PM, Rustam Aliyev <ru...@code.az> wrote:
> Is there any plans to improve this in future?
>
> For big data clusters this could be very expensive. Based on your comment,
> I will need 200TB of storage for 100TB of data to keep Cassandra running.
>
> --
> Rustam.
>
> On 09/12/2010 17:56, Tyler Hobbs wrote:
>
> If you are on 0.6, repair is particularly dangerous with respect to disk
> space usage. If your replica is sufficiently out of sync, you can triple
> your disk usage pretty easily. This has been improved in 0.7, so repairs
> should use about half as much disk space, on average.
>
> In general, yes, keep your nodes under 50% disk usage at all times. Any
> of: compaction, cleanup, snapshotting, repair, or bootstrapping (the latter
> two are improved in 0.7) can double your disk usage temporarily.
>
> You should plan to add more disk space or add nodes when you get close to
> this limit. Once you go over 50%, it's more difficult to add nodes, at
> least in 0.6.
>
> - Tyler
>
> On Thu, Dec 9, 2010 at 11:19 AM, Mark <st...@gmail.com> wrote:
>
>> I recently ran into a problem during a repair operation where my nodes
>> completely ran out of space and my whole cluster was... well, clusterfucked.
>>
>> I want to make sure how to prevent this problem in the future.
>>
>> Should I make sure that at all times every node is under 50% of its disk
>> space? Are there any normal day-to-day operations that would cause the any
>> one node to double in size that I should be aware of? If on or more nodes to
>> surpass the 50% mark, what should I plan to do?
>>
>> Thanks for any advice
>>
>
>
Re: Cassandra and disk space
Posted by Rustam Aliyev <ru...@code.az>.
Is there any plans to improve this in future?
For big data clusters this could be very expensive. Based on your
comment, I will need 200TB of storage for 100TB of data to keep
Cassandra running.
--
Rustam.
On 09/12/2010 17:56, Tyler Hobbs wrote:
> If you are on 0.6, repair is particularly dangerous with respect to
> disk space usage. If your replica is sufficiently out of sync, you
> can triple your disk usage pretty easily. This has been improved in
> 0.7, so repairs should use about half as much disk space, on average.
>
> In general, yes, keep your nodes under 50% disk usage at all times.
> Any of: compaction, cleanup, snapshotting, repair, or bootstrapping
> (the latter two are improved in 0.7) can double your disk usage
> temporarily.
>
> You should plan to add more disk space or add nodes when you get close
> to this limit. Once you go over 50%, it's more difficult to add
> nodes, at least in 0.6.
>
> - Tyler
>
> On Thu, Dec 9, 2010 at 11:19 AM, Mark <static.void.dev@gmail.com
> <ma...@gmail.com>> wrote:
>
> I recently ran into a problem during a repair operation where my
> nodes completely ran out of space and my whole cluster was...
> well, clusterfucked.
>
> I want to make sure how to prevent this problem in the future.
>
> Should I make sure that at all times every node is under 50% of
> its disk space? Are there any normal day-to-day operations that
> would cause the any one node to double in size that I should be
> aware of? If on or more nodes to surpass the 50% mark, what should
> I plan to do?
>
> Thanks for any advice
>
>
Re: Cassandra and disk space
Posted by Tyler Hobbs <ty...@riptano.com>.
If you are on 0.6, repair is particularly dangerous with respect to disk
space usage. If your replica is sufficiently out of sync, you can triple
your disk usage pretty easily. This has been improved in 0.7, so repairs
should use about half as much disk space, on average.
In general, yes, keep your nodes under 50% disk usage at all times. Any of:
compaction, cleanup, snapshotting, repair, or bootstrapping (the latter two
are improved in 0.7) can double your disk usage temporarily.
You should plan to add more disk space or add nodes when you get close to
this limit. Once you go over 50%, it's more difficult to add nodes, at
least in 0.6.
- Tyler
On Thu, Dec 9, 2010 at 11:19 AM, Mark <st...@gmail.com> wrote:
> I recently ran into a problem during a repair operation where my nodes
> completely ran out of space and my whole cluster was... well, clusterfucked.
>
> I want to make sure how to prevent this problem in the future.
>
> Should I make sure that at all times every node is under 50% of its disk
> space? Are there any normal day-to-day operations that would cause the any
> one node to double in size that I should be aware of? If on or more nodes to
> surpass the 50% mark, what should I plan to do?
>
> Thanks for any advice
>
Re: Cassandra and disk space
Posted by Peter Schuller <pe...@infidyne.com>.
> I recently ran into a problem during a repair operation where my nodes
> completely ran out of space and my whole cluster was... well, clusterfucked.
>
> I want to make sure how to prevent this problem in the future.
Depending on which version you're on, you may be seeing this:
https://issues.apache.org/jira/browse/CASSANDRA-1674
But regardless, disk space variations is a fact of life with
cassandra. Off the top of my head I'm not ready to say what the
expectations are with respect to repair under all circumstances.
Anyone?
> Should I make sure that at all times every node is under 50% of its disk
> space? Are there any normal day-to-day operations that would cause the any
> one node to double in size that I should be aware of? If on or more nodes to
> surpass the 50% mark, what should I plan to do?
Major compactions can potentially double the amount of disk, if you
have a single large column family that contributes almost all disk
space. For such clusters, regular background compaction can indeed
cause a doubling when the compaction happened to be a major one (i.e.,
happened to include all sstables).
--
/ Peter Schuller