You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Eric Rosenberry <ep...@gmail.com> on 2010/10/25 05:16:53 UTC
Experiences with Cassandra hardware planning
All-
Over the past nine months I have been working to tune our hardware
configuration to optimally balance CPU/RAM/disk/iops/network per node for
our Cassandra workload. Thanks much to those here who have provided helpful
advice.
I wanted to share back to the community some of the learnings we have come
across including the hardware configuration we have been successful with
(YMMV). This is still a work in progress naturally.
I have written up a detailed blog post about this here:
http://www.bitplumber.net/2010/10/a-cassandra-hardware-stack-dell-c1100s-ocz-vertex-2-ssds-with-sandforce-arista-7048s/
Here are the highlights:
- Dell C1100 "cloud series" servers with 10x 2.5 inch drive bays
- OCZ Technology Vertex 2 MLC SSD's with the Sandforce 1200 series
controllers
- Arista Networks 7048 1U Top of Rack switches running MLAG with LACP to
the hosts
Let me know if you have any questions!
-Eric
Re: Experiences with Cassandra hardware planning
Posted by Jonathan Ellis <jb...@gmail.com>.
On Mon, Oct 25, 2010 at 10:25 AM, Edward Capriolo <ed...@gmail.com> wrote:
>> 2. We gave up on using Cassandra's row cache as loading any reasonable
>> amount of data into the cache would take days/weeks with our tiny row size.
>> We instead are using file system cache.
I don't follow the reasoning there. Row cache or fs cache, it will be
hot after reading it once, the difference is that doing a read to the
cached data is much faster from row cache.
>> 4. When Cassandra is set to replication factor of three and the read replica
>> count is one, data still gets read (for read repair) on all three nodes that
>> have a copy of the data.
FWIW, you don't have to wait for 0.7 to turn this off (see
http://www.riptano.com/blog/whats-new-cassandra-066). However, to pin
affinity to a single replica you will need SimpleStrategy from 0.7.
--
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of Riptano, the source for professional Cassandra support
http://riptano.com
Re: Experiences with Cassandra hardware planning
Posted by Edward Capriolo <ed...@gmail.com>.
On Mon, Oct 25, 2010 at 11:21 AM, Eric Rosenberry <er...@rosenberry.org> wrote:
> Hey Chris-
> That is tough to say as we started out with no data and have been
> continuously loading data into the cluster. Initially we had less data than
> the amount of RAM in each node (48 gigs) but we have eventually exceeded
> that and now have many times more data on each node than in the entire
> cluster.
> Some key points though:
> 1. Upon cold start of the cluster (i.e. nothing in file system cache) disk
> i/o was massive even when the total dataset was less than the RAM in one
> system (this same thing holds true in RDBMS systems of course, though many
> of them are smart about pre-loading data)
> 2. We gave up on using Cassandra's row cache as loading any reasonable
> amount of data into the cache would take days/weeks with our tiny row size.
> We instead are using file system cache.
> 3. After switching to SSD's we thought we might be able to get away with les
> RAM (as we were relying on the SSD's to be fast rather than RAM cache) but
> dropping them to 24 gigs cut the clusters read capacity by 75%.
> 4. When Cassandra is set to replication factor of three and the read replica
> count is one, data still gets read (for read repair) on all three nodes that
> have a copy of the data. This brings that data into memory on those
> machines so the amount of total cluster memory available to cache actual
> data is not 192 gigs in my example of four nodes, but only 64 gigs minus OS
> and Cassandra overhead (I divided 192 by three since three copies are stored
> in RAM across the cluster).
> -Eric
>
> On Mon, Oct 25, 2010 at 7:41 AM, Chris Burroughs <ch...@gmail.com>
> wrote:
>>
>> You mention that you consistently found your boxes IO bound despite the
>> large amount of RAM available for caching. Could you state roughly what
>> the ratio of RAM to on disk data was?
>
>
If reading properly it looks like you used Linux Software Raid on top
of the SSD devices. Can you talk about this? I would think that even
with a simple RAID this would drive you CPU high. But it seems you may
not have other options since SSD RAID cards probably do not exist.
Re: Experiences with Cassandra hardware planning
Posted by Eric Rosenberry <er...@rosenberry.org>.
Hey Chris-
That is tough to say as we started out with no data and have been
continuously loading data into the cluster. Initially we had less data than
the amount of RAM in each node (48 gigs) but we have eventually exceeded
that and now have many times more data on each node than in the entire
cluster.
Some key points though:
1. Upon cold start of the cluster (i.e. nothing in file system cache) disk
i/o was massive even when the total dataset was less than the RAM in one
system (this same thing holds true in RDBMS systems of course, though many
of them are smart about pre-loading data)
2. We gave up on using Cassandra's row cache as loading any reasonable
amount of data into the cache would take days/weeks with our tiny row size.
We instead are using file system cache.
3. After switching to SSD's we thought we might be able to get away with les
RAM (as we were relying on the SSD's to be fast rather than RAM cache) but
dropping them to 24 gigs cut the clusters read capacity by 75%.
4. When Cassandra is set to replication factor of three and the read replica
count is one, data still gets read (for read repair) on all three nodes that
have a copy of the data. This brings that data into memory on those
machines so the amount of total cluster memory available to cache actual
data is not 192 gigs in my example of four nodes, but only 64 gigs minus OS
and Cassandra overhead (I divided 192 by three since three copies are stored
in RAM across the cluster).
-Eric
On Mon, Oct 25, 2010 at 7:41 AM, Chris Burroughs
<ch...@gmail.com>wrote:
>
>
> You mention that you consistently found your boxes IO bound despite the
> large amount of RAM available for caching. Could you state roughly what
> the ratio of RAM to on disk data was?
>
Re: Experiences with Cassandra hardware planning
Posted by Chris Burroughs <ch...@gmail.com>.
On 10/24/2010 11:16 PM, Eric Rosenberry wrote:
> I wanted to share back to the community some of the learnings we have come
> across including the hardware configuration we have been successful with
> (YMMV). This is still a work in progress naturally.
>
> I have written up a detailed blog post about this here:
> http://www.bitplumber.net/2010/10/a-cassandra-hardware-stack-dell-c1100s-ocz-vertex-2-ssds-with-sandforce-arista-7048s/
Eric, thank you for the details writeup!
You mention that you consistently found your boxes IO bound despite the
large amount of RAM available for caching. Could you state roughly what
the ratio of RAM to on disk data was?
Re: Experiences with Cassandra hardware planning
Posted by Jonathan Ellis <jb...@gmail.com>.
On Mon, Oct 25, 2010 at 3:10 PM, Eric Rosenberry <er...@rosenberry.org> wrote:
>> I don't follow the reasoning there. Row cache or fs cache, it will be
>> hot after reading it once, the difference is that doing a read to the
>> cached data is much faster from row cache.
>
> Yeah, I would have thought the same. Benjamin Black actually recommended we
> go this route as with our dataset (we have huge numbers of super-tiny rows)
> it would take weeks of running for the row cache to become useful.
Sounds like the assumption is that the FS cache's guessing that if one
row from a data page is hot, the rest of the page will too, will be
helpful. Depending on your workload it may help, or it may be
counterproductive. :)
--
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of Riptano, the source for professional Cassandra support
http://riptano.com
Re: Experiences with Cassandra hardware planning
Posted by Eric Rosenberry <er...@rosenberry.org>.
I am going to respond to multiple questions in one email to keep down the
thread insanity:
On Mon, Oct 25, 2010 at 12:39 AM, David Dabbs <dm...@gmail.com> wrote:
> Sorry, Eric I’m not following you. You’ve set the JVM’s processor
> affinity so it only runs on one of the processors?
>
My understanding is that Linux will launch a given process on one "node"
(processor in this case) or another and then attempt to allocate memory only
from that node for that process. If free memory is unavailable on that node
it will assign memory from the other node. The process scheduler will try
and schedule the process on that node as well.
My knowledge is very limited here, and in fact, most of what I know comes
from this article:
http://jcole.us/blog/archives/2010/09/28/mysql-swap-insanity-and-the-numa-architecture/
On Mon, Oct 25, 2010 at 8:25 AM, Edward Capriolo <ed...@gmail.com>
wrote:
> If reading properly it looks like you used Linux Software Raid on top
> of the SSD devices. Can you talk about this? I would think that even
> with a simple RAID this would drive you CPU high. But it seems you may
> not have other options since SSD RAID cards probably do not exist.
>
Yes, we are running Linux kernel raid (not LVM). This is mostly because our
first batch of machines had the SSD's hooked directly to the onboard Intel
ICH10 SATA controller rather than any add in RAID card. We are only doing
RAID 0 here so I would not expect this to take any CPU to speak of since
it's just doing a mod operator (or something simple) to figure out which
disk the data goes on. With RAID 0 there is no parity calculation. Even if
there was more work to be done, there are 8 cores (and 16 virtual processors
when you consider hyperthreading) for that operation to be scheduled on. We
don't seem to be CPU bound.
That being said, we really should try out the LSI 2008's RAID 0 capability,
but we have not had a chance yet.
On Mon, Oct 25, 2010 at 9:07 AM, Jonathan Ellis <jb...@gmail.com> wrote:
> On Mon, Oct 25, 2010 at 10:25 AM, Edward Capriolo <ed...@gmail.com>
> wrote:
> >> 2. We gave up on using Cassandra's row cache as loading any reasonable
> >> amount of data into the cache would take days/weeks with our tiny row
> size.
> >> We instead are using file system cache.
>
> I don't follow the reasoning there. Row cache or fs cache, it will be
> hot after reading it once, the difference is that doing a read to the
> cached data is much faster from row cache.
Yeah, I would have thought the same. Benjamin Black actually recommended we
go this route as with our dataset (we have huge numbers of super-tiny rows)
it would take weeks of running for the row cache to become useful.
-Eric
RE: Experiences with Cassandra hardware planning
Posted by David Dabbs <dm...@gmail.com>.
Sorry, Eric I'm not following you. You've set the JVM's processor affinity
so it only runs on one of the processors?
From: eprosenx@gmail.com [mailto:eprosenx@gmail.com] On Behalf Of Eric
Rosenberry
Sent: Monday, October 25, 2010 12:49 AM
To: user@cassandra.apache.org
Subject: Re: Experiences with Cassandra hardware planning
We have not started our JVM's with numactl.
I am not sure what (if any) benefit there has been to turning on NUMA in the
BIOS. Turning it on could have in fact reduced performance. I suspect that
Java is only using memory from one of the processors (since less than half
of the physical memory is assigned to the JVM) and the other processors
memory is being used for file system cache.
Clearly there is probably some room for improvement here. We have not
invested much time in this as of yet.
If anyone else has knowledge in this area please chime in!
-Eric
On Sun, Oct 24, 2010 at 10:36 PM, David Dabbs <dm...@gmail.com> wrote:
Eric,
Thanks for the detailed post! Did you need to start your JVMs with numactl
in order to take advantage of NUMA?
I know the board, OS and JVM must be configured properly, but it's not
clear if the JVMs must be started with numactl.
Thanks,
David
From: Eric Rosenberry [mailto:eprosenx@gmail.com]
Sent: Sunday, October 24, 2010 10:17 PM
To: user@cassandra.apache.org
Subject: Experiences with Cassandra hardware planning
All-
Over the past nine months I have been working to tune our hardware
configuration to optimally balance CPU/RAM/disk/iops/network per node for
our Cassandra workload. Thanks much to those here who have provided helpful
advice.
I wanted to share back to the community some of the learnings we have come
across including the hardware configuration we have been successful with
(YMMV). This is still a work in progress naturally.
I have written up a detailed blog post about this here:
http://www.bitplumber.net/2010/10/a-cassandra-hardware-stack-dell-c1100s-ocz
<http://www.bitplumber.net/2010/10/a-cassandra-hardware-stack-dell-c1100s-oc
z%0d%0a-vertex-2-ssds-with-sandforce-arista-7048s/>
-vertex-2-ssds-with-sandforce-arista-7048s/
Here are the highlights:
. Dell C1100 "cloud series" servers with 10x 2.5 inch drive bays
. OCZ Technology Vertex 2 MLC SSD's with the Sandforce 1200 series
controllers
. Arista Networks 7048 1U Top of Rack switches running MLAG with LACP to the
hosts
Let me know if you have any questions!
-Eric
Re: Experiences with Cassandra hardware planning
Posted by Eric Rosenberry <er...@rosenberry.org>.
We have not started our JVM's with numactl.
I am not sure what (if any) benefit there has been to turning on NUMA in the
BIOS. Turning it on could have in fact reduced performance. I suspect that
Java is only using memory from one of the processors (since less than half
of the physical memory is assigned to the JVM) and the other processors
memory is being used for file system cache.
Clearly there is probably some room for improvement here. We have not
invested much time in this as of yet.
If anyone else has knowledge in this area please chime in!
-Eric
On Sun, Oct 24, 2010 at 10:36 PM, David Dabbs <dm...@gmail.com> wrote:
> Eric,
>
> Thanks for the detailed post! Did you need to start your JVMs with numactl
> in order to take advantage of NUMA?
> I know the board, OS and JVM must be configured properly, but it's not
> clear if the JVMs must be started with numactl.
>
>
>
> Thanks,
>
> David
>
>
>
> From: Eric Rosenberry [mailto:eprosenx@gmail.com]
> Sent: Sunday, October 24, 2010 10:17 PM
> To: user@cassandra.apache.org
> Subject: Experiences with Cassandra hardware planning
>
> All-
>
> Over the past nine months I have been working to tune our hardware
> configuration to optimally balance CPU/RAM/disk/iops/network per node for
> our Cassandra workload. Thanks much to those here who have provided
> helpful
> advice.
>
> I wanted to share back to the community some of the learnings we have come
> across including the hardware configuration we have been successful with
> (YMMV). This is still a work in progress naturally.
>
> I have written up a detailed blog post about this here:
>
> http://www.bitplumber.net/2010/10/a-cassandra-hardware-stack-dell-c1100s-ocz
> -vertex-2-ssds-with-sandforce-arista-7048s/
>
> Here are the highlights:
> • Dell C1100 "cloud series" servers with 10x 2.5 inch drive bays
> • OCZ Technology Vertex 2 MLC SSD's with the Sandforce 1200 series
> controllers
> • Arista Networks 7048 1U Top of Rack switches running MLAG with LACP to
> the
> hosts
> Let me know if you have any questions!
>
> -Eric
>
>
RE: Experiences with Cassandra hardware planning
Posted by David Dabbs <dm...@gmail.com>.
Eric,
Thanks for the detailed post! Did you need to start your JVMs with numactl
in order to take advantage of NUMA?
I know the board, OS and JVM must be configured properly, but it's not
clear if the JVMs must be started with numactl.
Thanks,
David
From: Eric Rosenberry [mailto:eprosenx@gmail.com]
Sent: Sunday, October 24, 2010 10:17 PM
To: user@cassandra.apache.org
Subject: Experiences with Cassandra hardware planning
All-
Over the past nine months I have been working to tune our hardware
configuration to optimally balance CPU/RAM/disk/iops/network per node for
our Cassandra workload. Thanks much to those here who have provided helpful
advice.
I wanted to share back to the community some of the learnings we have come
across including the hardware configuration we have been successful with
(YMMV). This is still a work in progress naturally.
I have written up a detailed blog post about this here:
http://www.bitplumber.net/2010/10/a-cassandra-hardware-stack-dell-c1100s-ocz
-vertex-2-ssds-with-sandforce-arista-7048s/
Here are the highlights:
Dell C1100 "cloud series" servers with 10x 2.5 inch drive bays
OCZ Technology Vertex 2 MLC SSD's with the Sandforce 1200 series
controllers
Arista Networks 7048 1U Top of Rack switches running MLAG with LACP to the
hosts
Let me know if you have any questions!
-Eric