You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Jonathan Colby <jo...@gmail.com> on 2011/03/21 21:50:25 UTC
cassandra nodes with mixed hard disk sizes
This is a two part question ...
1. If you have cassandra nodes with different sized hard disks, how do you deal with assigning the token ring such that the nodes with larger disks get more data? In other words, given equally distributed token ranges, when the smaller disk nodes run out of space, the larger disk nodes with still have unused capacity. Or is installing a mixed hardware cluster a no-no?
2. What happens when a cassandra node runs out of disk space for its data files? Does it continue serving the data while not accepting new data? Or does the node break and require manual intervention?
This info has alluded me elsewhere.
Jon
Re: cassandra nodes with mixed hard disk sizes
Posted by Aaron Morton <aa...@thelastpickle.com>.
Should be: not that I know of without making code changes.
Aaron
On 22/03/2011, at 11:05 PM, Aaron Morton <aa...@thelastpickle.com> wrote:
> Not that I know of.
> Aaron
>
> On 22/03/2011, at 10:45 PM, buddhasystem <po...@bnl.gov> wrote:
>
>>
>> aaron morton wrote:
>>>
>>>
>>> Also a node is be responsible for storing it's token range and acting as a
>>> replica for other token ranges. So reducing the token range may not have a
>>> dramatic affect on the storage requirements.
>>>
>>
>> Aaron,
>>
>> is there a way to configure wimpy nodes such that the replicas are
>> elsewhere?
>>
>>
>> --
>> View this message in context: http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/cassandra-nodes-with-mixed-hard-disk-sizes-tp6194071p6195543.html
>> Sent from the cassandra-user@incubator.apache.org mailing list archive at Nabble.com.
Re: cassandra nodes with mixed hard disk sizes
Posted by Aaron Morton <aa...@thelastpickle.com>.
Not that I know of.
Aaron
On 22/03/2011, at 10:45 PM, buddhasystem <po...@bnl.gov> wrote:
>
> aaron morton wrote:
>>
>>
>> Also a node is be responsible for storing it's token range and acting as a
>> replica for other token ranges. So reducing the token range may not have a
>> dramatic affect on the storage requirements.
>>
>
> Aaron,
>
> is there a way to configure wimpy nodes such that the replicas are
> elsewhere?
>
>
> --
> View this message in context: http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/cassandra-nodes-with-mixed-hard-disk-sizes-tp6194071p6195543.html
> Sent from the cassandra-user@incubator.apache.org mailing list archive at Nabble.com.
Re: cassandra nodes with mixed hard disk sizes
Posted by buddhasystem <po...@bnl.gov>.
aaron morton wrote:
>
>
> Also a node is be responsible for storing it's token range and acting as a
> replica for other token ranges. So reducing the token range may not have a
> dramatic affect on the storage requirements.
>
Aaron,
is there a way to configure wimpy nodes such that the replicas are
elsewhere?
--
View this message in context: http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/cassandra-nodes-with-mixed-hard-disk-sizes-tp6194071p6195543.html
Sent from the cassandra-user@incubator.apache.org mailing list archive at Nabble.com.
Re: cassandra nodes with mixed hard disk sizes
Posted by aaron morton <aa...@thelastpickle.com>.
I probably could have saved myself some time by saying (as Peter and Edward pointed out) "if you use nodes with different capabilities you will need treat all nodes as having the lowest spec and that could be a waste." :)
Aaron
On 23 Mar 2011, at 07:26, Peter Schuller wrote:
>> Wait! maybe this is a quadruple-whammy since we have to account for
>> the data being replicated to other nodes. At replication factor 3 only
>> 1/3rd of the data on the node actually belongs in that TokenRange, So
>> it is not as simple as having small nodes with smaller ranges, you
>> also have to consider nodes around it and somehow balance them out to.
>> (I am not convinced it can be done)
>
> This is what I was talking about.
>
> However I forgot about the memtable settings etc actually being global
> in that sense as you reiterated (this was presumably what Aaron meant
> from the start - I mis-interpreted). Solving that in a way that
> doesn't make schema management much more complex might be an
> interesting problem. Maybe having a per-node scaling factor for some
> of these things would help.
>
> But that and the RF issue seem like the major concerns.
>
> You mentioned difficulty w.r.t. not only balancing request amount but
> also differing costs per request depending on data sizes - yes, but
> that's just a fundamental problem of balancing systems like this. Just
> because some node is "twice as fast" using some particular metric
> doesn't mean that metric is the only thing of concern for your access
> pattern. I don't think Cassandra exacerbates that particularly.
>
> Virtual nodes or some other method of dispersing data across a ring in
> a more flexible way may mitigate or eliminate the RF induced problem
> (along with having other nice effects).
>
> Anyways, I agree that it is advisable to avoid mixing slim and fat
> nodes in a cluster.
>
> --
> / Peter Schuller
Re: cassandra nodes with mixed hard disk sizes
Posted by Peter Schuller <pe...@infidyne.com>.
> Wait! maybe this is a quadruple-whammy since we have to account for
> the data being replicated to other nodes. At replication factor 3 only
> 1/3rd of the data on the node actually belongs in that TokenRange, So
> it is not as simple as having small nodes with smaller ranges, you
> also have to consider nodes around it and somehow balance them out to.
> (I am not convinced it can be done)
This is what I was talking about.
However I forgot about the memtable settings etc actually being global
in that sense as you reiterated (this was presumably what Aaron meant
from the start - I mis-interpreted). Solving that in a way that
doesn't make schema management much more complex might be an
interesting problem. Maybe having a per-node scaling factor for some
of these things would help.
But that and the RF issue seem like the major concerns.
You mentioned difficulty w.r.t. not only balancing request amount but
also differing costs per request depending on data sizes - yes, but
that's just a fundamental problem of balancing systems like this. Just
because some node is "twice as fast" using some particular metric
doesn't mean that metric is the only thing of concern for your access
pattern. I don't think Cassandra exacerbates that particularly.
Virtual nodes or some other method of dispersing data across a ring in
a more flexible way may mitigate or eliminate the RF induced problem
(along with having other nice effects).
Anyways, I agree that it is advisable to avoid mixing slim and fat
nodes in a cluster.
--
/ Peter Schuller
Re: cassandra nodes with mixed hard disk sizes
Posted by Edward Capriolo <ed...@gmail.com>.
On Tue, Mar 22, 2011 at 12:23 PM, Peter Schuller
<pe...@infidyne.com> wrote:
>> I may be wrong on this, so anyone else feel free to jump in. Here are some issues to consider...
>>
>> - keyspace memory requirements are global, all nodes must have enough memory to support the CFs.
>> - During node moves, additions or deletions the token range may increase, nodes with less total space than others would make this more complicated.
>> - during a write the mutation is sent to all replicas, a weak node that is a replica for a strong and busy node will be asked to store data from the strong node.
>> - read repair reads from all replicas
>> - when strong nodes that replicate to a weak node are compacting or repairing the dynamic snitch may order them lower than the weak node. Potentially increasing read requests on the weak one.
>> - down time for a strong node (or cluster partition) may result in increased read traffic to a weak node if all up replicas are needed to achieve the CL.
>> - nodes store their token range and the token range for RF-1 other nodes.
>
> The idea is to layout your ring to account for differences. However
> the kink is that this only works exactly as you would want for RF=1
> where you can directly control the capacity of each node by assigning
> an appropriately sized ring. For RF > 1 you start having to consider
> how replicas are chosen, and that a small node with a large "neighbor"
> (neighbor in the sense of replica selection; direct neighbor in the
> ring in the simplest case) contributes to the load on your small node.
>
> So there are definitely concerns with mixing arbitrarily performing
> nodes, but it's not like you must have identically sized nodes.
> Probably a reasonable way to mix nodes is to have as few classes of
> nodes as possible, and have them adjacent to each other on the ring.
> So e.g., 15 fat nodes followed by 20 slim nodes. The fat nodes near
> the fat/slim barrier would probably not be fully utilized because it
> would spill over (due to RF > 1) on the slimmer nodes.
>
> But yes, ring management and interaction with the chosen replication
> strategy becomes more complex. Keeping in mind though that at worst
> you have to treat some slightly better nodes as if they weren't. So it
> only becomes an issue where the node capacity is sufficiently
> different that you start caring about actually utilizing them fully.
>
> I'd be interested to hear what people end up doing about this in
> production, assuming people have any clusters that have survived long
> enough on an evolving ring of hardware to actually have this problem
> yet :)
>
> --
> / Peter Schuller
>
The problem is that laying out the Tokens is very hard. For example
one machine may have 40GB another machine may have 60GB. This creates
a technical term coming...double-whammy.
First it will receive proportionally more requests. (dynamic snitch helps here)
Second these requests will have to Random Read through more data.
Actually it is more like a tripple-whammy since every configuration
like key cache size,memtable flush,etc is now in CFMetaData and shared
across the cluster. What setting do you pick for keycache? If you go
small you do not get the most out of hardware. If you go large your
smaller nodes may have GC issues.
Wait! maybe this is a quadruple-whammy since we have to account for
the data being replicated to other nodes. At replication factor 3 only
1/3rd of the data on the node actually belongs in that TokenRange, So
it is not as simple as having small nodes with smaller ranges, you
also have to consider nodes around it and somehow balance them out to.
(I am not convinced it can be done)
These "whammy" conditions may not be a big deal since your cluster
should have extra capacity. As long as you are operating with some
overhead they may not be noticeable, However as you start approaching
capacity your smaller nodes will start showing (usually IO ) problems
first.
I would say mixed hardware is a no-no if the hardware. By mixed I mean
drastically different hardware. As to how this translates long
term...hopefully you have a 3 or 5 year hardware replacement policy.
Before then disk and memory upgrades should serve you well.
Re: cassandra nodes with mixed hard disk sizes
Posted by Peter Schuller <pe...@infidyne.com>.
> I may be wrong on this, so anyone else feel free to jump in. Here are some issues to consider...
>
> - keyspace memory requirements are global, all nodes must have enough memory to support the CFs.
> - During node moves, additions or deletions the token range may increase, nodes with less total space than others would make this more complicated.
> - during a write the mutation is sent to all replicas, a weak node that is a replica for a strong and busy node will be asked to store data from the strong node.
> - read repair reads from all replicas
> - when strong nodes that replicate to a weak node are compacting or repairing the dynamic snitch may order them lower than the weak node. Potentially increasing read requests on the weak one.
> - down time for a strong node (or cluster partition) may result in increased read traffic to a weak node if all up replicas are needed to achieve the CL.
> - nodes store their token range and the token range for RF-1 other nodes.
The idea is to layout your ring to account for differences. However
the kink is that this only works exactly as you would want for RF=1
where you can directly control the capacity of each node by assigning
an appropriately sized ring. For RF > 1 you start having to consider
how replicas are chosen, and that a small node with a large "neighbor"
(neighbor in the sense of replica selection; direct neighbor in the
ring in the simplest case) contributes to the load on your small node.
So there are definitely concerns with mixing arbitrarily performing
nodes, but it's not like you must have identically sized nodes.
Probably a reasonable way to mix nodes is to have as few classes of
nodes as possible, and have them adjacent to each other on the ring.
So e.g., 15 fat nodes followed by 20 slim nodes. The fat nodes near
the fat/slim barrier would probably not be fully utilized because it
would spill over (due to RF > 1) on the slimmer nodes.
But yes, ring management and interaction with the chosen replication
strategy becomes more complex. Keeping in mind though that at worst
you have to treat some slightly better nodes as if they weren't. So it
only becomes an issue where the node capacity is sufficiently
different that you start caring about actually utilizing them fully.
I'd be interested to hear what people end up doing about this in
production, assuming people have any clusters that have survived long
enough on an evolving ring of hardware to actually have this problem
yet :)
--
/ Peter Schuller
Re: cassandra nodes with mixed hard disk sizes
Posted by Aaron Morton <aa...@thelastpickle.com>.
My assumption is from not seeing anything in the code to explicitly support nodes of different specs (also think I saw it somewhere ages ago). AFAIK the dynamic snitch is there to detect nodes with a temporarily reduced throughput and try to reduce the read load on them.
I may be wrong on this, so anyone else feel free to jump in. Here are some issues to consider...
- keyspace memory requirements are global, all nodes must have enough memory to support the CFs.
- During node moves, additions or deletions the token range may increase, nodes with less total space than others would make this more complicated.
- during a write the mutation is sent to all replicas, a weak node that is a replica for a strong and busy node will be asked to store data from the strong node.
- read repair reads from all replicas
- when strong nodes that replicate to a weak node are compacting or repairing the dynamic snitch may order them lower than the weak node. Potentially increasing read requests on the weak one.
- down time for a strong node (or cluster partition) may result in increased read traffic to a weak node if all up replicas are needed to achieve the CL.
- nodes store their token range and the token range for RF-1 other nodes.
Overall when a node goes down other nodes need to be able to handle the potential extra load (connections, reads, storing HH). If you have some weak and some strong nodes there is a chance of the weak nodes been overwhelmed which may reduce the availability of your cluster.
Hope that helps.
Aaron
On 22/03/2011, at 10:54 PM, Daniel Doubleday <da...@gmx.net> wrote:
>
> On Mar 22, 2011, at 5:09 AM, aaron morton wrote:
>> 1) You should use nodes with the same capacity (CPU, RAM, HDD), cassandra assumes they are all equal.
>
> Care to elaborate? While equal node will certainly make life easier I would have thought that dynamic snitch would take care of performance differences and manual assignment of token ranges can yield to any data distribution. Obviously if a node has twices as much data will probably get twice the load. But if that is no problem ...
>
> Where does cassandra assume that all are equal?
>
> Cheers Daniel
>
>
>>
>> 2) Not sure what exactly would happen. Am guessing either the node would shutdown or writes would eventually block, probably the former. If the node was up read performance may suffer (if there were more writes been sent in). If you really want to know more let me know and I may find time to dig into it.
>>
>> Also a node is be responsible for storing it's token range and acting as a replica for other token ranges. So reducing the token range may not have a dramatic affect on the storage requirements.
>>
>> Hope that helps.
>> Aaron
>>
>> On 22 Mar 2011, at 09:50, Jonathan Colby wrote:
>>
>>>
>>> This is a two part question ...
>>>
>>> 1. If you have cassandra nodes with different sized hard disks, how do you deal with assigning the token ring such that the nodes with larger disks get more data? In other words, given equally distributed token ranges, when the smaller disk nodes run out of space, the larger disk nodes with still have unused capacity. Or is installing a mixed hardware cluster a no-no?
>>>
>>> 2. What happens when a cassandra node runs out of disk space for its data files? Does it continue serving the data while not accepting new data? Or does the node break and require manual intervention?
>>>
>>> This info has alluded me elsewhere.
>>> Jon
>>
>
Re: cassandra nodes with mixed hard disk sizes
Posted by Daniel Doubleday <da...@gmx.net>.
On Mar 22, 2011, at 5:09 AM, aaron morton wrote:
> 1) You should use nodes with the same capacity (CPU, RAM, HDD), cassandra assumes they are all equal.
Care to elaborate? While equal node will certainly make life easier I would have thought that dynamic snitch would take care of performance differences and manual assignment of token ranges can yield to any data distribution. Obviously if a node has twice as much data will probably get twice the load. But if that is no problem ...
Where does cassandra assume that all are equal?
Cheers Daniel
>
> 2) Not sure what exactly would happen. Am guessing either the node would shutdown or writes would eventually block, probably the former. If the node was up read performance may suffer (if there were more writes been sent in). If you really want to know more let me know and I may find time to dig into it.
>
> Also a node is be responsible for storing it's token range and acting as a replica for other token ranges. So reducing the token range may not have a dramatic affect on the storage requirements.
>
> Hope that helps.
> Aaron
>
> On 22 Mar 2011, at 09:50, Jonathan Colby wrote:
>
>>
>> This is a two part question ...
>>
>> 1. If you have cassandra nodes with different sized hard disks, how do you deal with assigning the token ring such that the nodes with larger disks get more data? In other words, given equally distributed token ranges, when the smaller disk nodes run out of space, the larger disk nodes with still have unused capacity. Or is installing a mixed hardware cluster a no-no?
>>
>> 2. What happens when a cassandra node runs out of disk space for its data files? Does it continue serving the data while not accepting new data? Or does the node break and require manual intervention?
>>
>> This info has alluded me elsewhere.
>> Jon
>
Re: cassandra nodes with mixed hard disk sizes
Posted by aaron morton <aa...@thelastpickle.com>.
1) You should use nodes with the same capacity (CPU, RAM, HDD), cassandra assumes they are all equal.
2) Not sure what exactly would happen. Am guessing either the node would shutdown or writes would eventually block, probably the former. If the node was up read performance may suffer (if there were more writes been sent in). If you really want to know more let me know and I may find time to dig into it.
Also a node is be responsible for storing it's token range and acting as a replica for other token ranges. So reducing the token range may not have a dramatic affect on the storage requirements.
Hope that helps.
Aaron
On 22 Mar 2011, at 09:50, Jonathan Colby wrote:
>
> This is a two part question ...
>
> 1. If you have cassandra nodes with different sized hard disks, how do you deal with assigning the token ring such that the nodes with larger disks get more data? In other words, given equally distributed token ranges, when the smaller disk nodes run out of space, the larger disk nodes with still have unused capacity. Or is installing a mixed hardware cluster a no-no?
>
> 2. What happens when a cassandra node runs out of disk space for its data files? Does it continue serving the data while not accepting new data? Or does the node break and require manual intervention?
>
> This info has alluded me elsewhere.
> Jon