You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Erik Forsberg <fo...@opera.com> on 2011/06/07 13:34:06 UTC
Re: Multiple large disks in server - setup considerations
On Tue, 31 May 2011 13:23:36 -0500
Jonathan Ellis <jb...@gmail.com> wrote:
> Have you read http://wiki.apache.org/cassandra/CassandraHardware ?
I had, but it was a while ago so I guess I kind of deserved an RTFM! :-)
After re-reading it, I still want to know:
* If we disregard the performance hit caused by having the commitlog on
the same physical device as parts of the data, are there any other
grave effects on Cassandra's functionality with a setup like that?
* How does Cassandra handle a case where one of the disks in a striped
RAID0 partition goes bad and is replaced? Is the only option to wipe
everything from that node and reinit the node, or will it handle
corrupt files? I.e, what's the recommended thing to do from an
operations point of view when a disk dies on one of the nodes in a
RAID0 Cassandra setup? What will cause the least risk for data loss?
What will be the fastest way to get the node up to speed with the
rest of the cluster?
Thanks,
\EF
>
> On Tue, May 31, 2011 at 7:47 AM, Erik Forsberg <fo...@opera.com>
> wrote:
> > Hi!
> >
> > I'm considering setting up a small (4-6 nodes) Cassandra cluster on
> > machines that each have 3x2TB disks. There's no hardware RAID in the
> > machine, and if there were, it could only stripe single disks
> > together, not parts of disks.
> >
> > I'm planning RF=2 (or higher).
> >
> > I'm pondering what the best disk configuration is. Two alternatives:
> >
> > 1) Make small partition on first disk for Linux installation and
> > commit log. Use Linux' software RAID0 to stripe the remaining space
> > on disk1
> > + the two remaining disks into one large XFS partition.
> >
> > 2) Make small partition on first disk for Linux installation and
> > commit log. Mount rest of disk 1 as /var/cassandra1, then disk2
> > as /var/cassandra2 and disk3 as /var/cassandra3.
> >
> > Is it unwise to put the commit log on the same physical disk as
> > some of the data? I guess it could impact write performance, but
> > maybe it's bad from a data consistency point of view?
> >
> > How does Cassandra handle replacement of a bad disk in the two
> > alternatives? With option 1) I guess there's risk of files being
> > corrupt. With option 2) they will simply be missing after replacing
> > the disk with a new one.
> >
> > With option 2) I guess I'm limiting the size of the total amount of
> > data in the largest CF at compaction to, hmm.. the free space on the
> > disk with most free space, correct?
> >
> > Comments welcome!
> >
> > Thanks,
> > \EF
> > --
> > Erik Forsberg <fo...@opera.com>
> > Developer, Opera Software - http://www.opera.com/
> >
>
>
>
--
Erik Forsberg <fo...@opera.com>
Developer, Opera Software - http://www.opera.com/
Re: Multiple large disks in server - setup considerations
Posted by Edward Capriolo <ed...@gmail.com>.
On Wed, Jun 8, 2011 at 12:19 AM, AJ <aj...@dude.podzone.net> wrote:
> On 6/7/2011 9:32 PM, Edward Capriolo wrote:
> <snip>
>
>>
>> I do not like large disk set-ups. I think they end up not being
>> economical. Most low latency use cases want high RAM to DISK ratio. Two
>> machines with 32GB RAM is usually less expensive then one machine with 64GB
>> ram.
>>
>> For a machine with 1TB drives (or multiple 1TB drives) it is going to be
>> difficult to get enough RAM to help with random read patterns.
>>
>> Also cluster operations like joining, decommissioning, or repair can take
>> a *VERY* long time maybe a day. More smaller servers like blade style or
>> more agile.
>>
>>
> Is there some rule-of-thumb as to how much RAM is needed per GB of data? I
> know it probably "depends", but if you could try to explain the best you can
> that would be great! I too am projecting "big data" requirements.
>
The way this is normally explained is active-set. IE you have 100,000,000
users but at any given time only 1,000,000 are active thus you need enough
RAM to keep these users cached.
No there is no rule of thumb it depends on access patterns. In the most
extreme case you are using cassandra for an ETL workload. In this case your
data will far exceed your RAM and since most operations will be like a "full
table scan" caching is almost hopeless and useless. On the other side there
are those that want every lookup to be predicatable low latency and totally
random read and those might want to maintain a 1-1 ratio.
I would track these things over time:
reads/writes to c*
disk utilization
size of CF on disk
cache hit rate
latency
And eventually you find what your ratio is. IE.
last month:
i had 30 reads/sec
my disk was 40% utilized
my column family was 40 GB
my cache hit was 70%
my latency was 1ms
this month:
i had 45 reads/sec
my disk was 95% utilized
my column family was 40 GB
my cache hit was 30%
my latency was 5ms
Conclusion:
my disk maxed and my cache hit/rate is dropping. I probably need more
nodes|or more RAM.
Re: Multiple large disks in server - setup considerations
Posted by AJ <aj...@dude.podzone.net>.
On 6/7/2011 9:32 PM, Edward Capriolo wrote:
<snip>
>
> I do not like large disk set-ups. I think they end up not being
> economical. Most low latency use cases want high RAM to DISK ratio.
> Two machines with 32GB RAM is usually less expensive then one machine
> with 64GB ram.
>
> For a machine with 1TB drives (or multiple 1TB drives) it is going to
> be difficult to get enough RAM to help with random read patterns.
>
> Also cluster operations like joining, decommissioning, or repair can
> take a *VERY* long time maybe a day. More smaller servers like blade
> style or more agile.
>
Is there some rule-of-thumb as to how much RAM is needed per GB of
data? I know it probably "depends", but if you could try to explain the
best you can that would be great! I too am projecting "big data"
requirements.
Re: Multiple large disks in server - setup considerations
Posted by Edward Capriolo <ed...@gmail.com>.
On Tue, Jun 7, 2011 at 12:43 PM, Ryan King <ry...@twitter.com> wrote:
> On Tue, Jun 7, 2011 at 4:34 AM, Erik Forsberg <fo...@opera.com> wrote:
> > On Tue, 31 May 2011 13:23:36 -0500
> > Jonathan Ellis <jb...@gmail.com> wrote:
> >
> >> Have you read http://wiki.apache.org/cassandra/CassandraHardware ?
> >
> > I had, but it was a while ago so I guess I kind of deserved an RTFM! :-)
> >
> > After re-reading it, I still want to know:
> >
> > * If we disregard the performance hit caused by having the commitlog on
> > the same physical device as parts of the data, are there any other
> > grave effects on Cassandra's functionality with a setup like that?
>
> You'll take a performance hit if you hare a high write load. I'd
> recommend doing your own benchmarks (with an existing benchmark
> framework like YCSB) against the configuration you'd like to use.
>
> > * How does Cassandra handle a case where one of the disks in a striped
> > RAID0 partition goes bad and is replaced? Is the only option to wipe
> > everything from that node and reinit the node, or will it handle
> > corrupt files?
>
> Don't plan on being able to recover any date on that node.
>
> > I.e, what's the recommended thing to do from an
> > operations point of view when a disk dies on one of the nodes in a
> > RAID0 Cassandra setup? What will cause the least risk for data loss?
> > What will be the fastest way to get the node up to speed with the
> > rest of the cluster?
>
> Decommission (or removetoken) on the dead node, replace the drive and
> rebootstrap.
>
> -ryan
>
I do not like large disk set-ups. I think they end up not being economical.
Most low latency use cases want high RAM to DISK ratio. Two machines with
32GB RAM is usually less expensive then one machine with 64GB ram.
For a machine with 1TB drives (or multiple 1TB drives) it is going to be
difficult to get enough RAM to help with random read patterns.
Also cluster operations like joining, decommissioning, or repair can take a
*VERY* long time maybe a day. More smaller servers like blade style or more
agile.
Re: Multiple large disks in server - setup considerations
Posted by Ryan King <ry...@twitter.com>.
On Tue, Jun 7, 2011 at 4:34 AM, Erik Forsberg <fo...@opera.com> wrote:
> On Tue, 31 May 2011 13:23:36 -0500
> Jonathan Ellis <jb...@gmail.com> wrote:
>
>> Have you read http://wiki.apache.org/cassandra/CassandraHardware ?
>
> I had, but it was a while ago so I guess I kind of deserved an RTFM! :-)
>
> After re-reading it, I still want to know:
>
> * If we disregard the performance hit caused by having the commitlog on
> the same physical device as parts of the data, are there any other
> grave effects on Cassandra's functionality with a setup like that?
You'll take a performance hit if you hare a high write load. I'd
recommend doing your own benchmarks (with an existing benchmark
framework like YCSB) against the configuration you'd like to use.
> * How does Cassandra handle a case where one of the disks in a striped
> RAID0 partition goes bad and is replaced? Is the only option to wipe
> everything from that node and reinit the node, or will it handle
> corrupt files?
Don't plan on being able to recover any date on that node.
> I.e, what's the recommended thing to do from an
> operations point of view when a disk dies on one of the nodes in a
> RAID0 Cassandra setup? What will cause the least risk for data loss?
> What will be the fastest way to get the node up to speed with the
> rest of the cluster?
Decommission (or removetoken) on the dead node, replace the drive and
rebootstrap.
-ryan