You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by "Rosenberry, Eric" <er...@iovation.com> on 2010/03/06 08:12:56 UTC
Cassandra hardware - balancing CPU/memory/iops/disk space
I am looking for advice from others that are further along in deploying Cassandra in production environments than we are. I want to know what you are finding your bottlenecks to be. I would feel silly purchasing dual processor quad core 2.93ghz Nehalem machines with 192 gigs of RAM just to find out that the two local SATA disks kept all that CPU and RAM from being useful (clearly that example would be a dumb).
I need to spec out hardware for an "optimal" Cassandra node (though our read/write characteristics are not yet fully defined so let's go with an "average" configuration).
My main concern is finding the right balance of:
* Available CPU
* RAM amount
* RAM speed (think Nehalem architecture where memory comes in a few speeds, though I doubt this is much of a concern as it is mainly dictated by which processor you buy and how many slots you populate)
* Total iops available (i.e. number of disks)
* Total disk space available (depending on the ratio of iops/space deciding on SAS vs. SATA and various rotational speeds)
My current thinking is 1U boxes with four 3.5 inch disks since that seems to be a readily available config. One big question is should I go with a single processor Nehalem system to go with those four disks, or would two CPU's be useful, and also, how much RAM is appropriate to match? I am making the assumption that Cassandra nodes are going to be disk bound as they must do a random read to answer any given query (i.e. indexes in RAM, but all data lives on disk?).
The other big decision is what type of hard disks others are finding to provide the optimal ratio of iops to available space? SAS or SATA? And what rotational speed?
Let me throw out here an actual hardware config and feel free to tell me the error of my ways:
* A SuperMicro SuperServer 6016T-NTRF configured as follows:
o 2.26 ghz E5520 dual processor quad core hyperthreaded Nehalem architecture (this proc provides a lot of bang for the buck, faster procs get more expensive quickly)
o Qty 12, 4 gig 1066mhz DIMMS for a total of 48 gigs RAM (the 4 gig DIMMS seem to be the price sweet spot)
o Dual on board 1 gigabit NIC's (perhaps one for client connections and the other for cluster communication?)
o Dual power supplies (I don't want to lose half my cluster due to a failure on one power leg)
o 4x 1TB SATA disks (this is a complete SWAG)
o No RAID controller (all just single individual disks presented to the OS) - Though is there any down side to using a RAID controller with RAID 0 (perhaps one single disk for the log for sequential io's, and 3x disks in a stripe for the random io's)
o The on-board IPMI based OOB controller (so we can kick the boxes remotely if need be)
* http://www.supermicro.com/products/system/1U/6016/SYS-6016T-NTRF.cfm
I can't help but think the above config has way too much RAM and CPU and not enough iops capacity. My understanding is that Cassandra does not cache much in RAM though?
Any thoughts are appreciated. Thanks.
-Eric
_______________________________________________________________
Eric Rosenberry
Sr. Infrastructure Architect | Chief Bit Plumber
iovation
111 SW Fifth Avenue
Suite 3200
Portland, OR 97204
www.iovation.com<http://www.iovation.com/>
The information contained in this email message may be privileged, confidential and protected from disclosure. If you are not the intended recipient, any dissemination, distribution or copying is strictly prohibited. If you think that you have received this email message in error, please notify the sender by reply email and delete the message and any attachments.
Re: Cassandra hardware - balancing CPU/memory/iops/disk space
Posted by Mark Robson <ma...@gmail.com>.
On 7 March 2010 19:05, Jonathan Ellis <jb...@gmail.com> wrote:
> Yes, but I would guess 90% of workloads are better served with
> spending the extra money on more machines w/ cheap sata disks and lots
> of ram.
>
I'm not an expert, but I imagine that in many cases, capex is not the
limiting factor. In data centres, power appears to be the main limitations -
modern x86 servers use so much power that many older data centres cannot be
stocked at very high density (because the don't have enough cooling for that
much heat).
So the number of boxes you can put in is directly affected by the amount of
power they use. For this reason, the most efficient way is to find the ideal
combination of CPU, memory and disc to do what you want with the least
power.
I guess ram is pretty cheap on power - I mean, it uses hardly any.
Mark
Re: High CPU Usage since 0.6.2
Posted by Lu Ming <xl...@live.com>.
Does the following code in IncomingStreamReader.read cause 100% CPU???
while (bytesRead < pendingFile.getExpectedBytes()) {
bytesRead += fc.transferFrom(socketChannel, bytesRead,
FileStreamTask.CHUNK_SIZE);
pendingFile.update(bytesRead);
}
BTW: our cassandra cluster is delpoyed in two datacenters.
--------------------------------------------------
From: "Lu Ming" <xl...@live.com>
Sent: Friday, June 04, 2010 7:01 PM
To: <us...@cassandra.apache.org>
Subject: Re: High CPU Usage since 0.6.2
> I do the Thread Dump on each cassandra node, and count the thread with
> call stack string "at
> org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:62)atorg.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.jav
> a:66)" in "thread-xxx"
>
> then I find an interesting things.
>
> Server Name/Thread Count/Average CPU Usage
> A 0 <20%
> B 8 760%
> C 0 <20%
> D 3 300%
> E 4 220%
> F 0 <20%
> G 2 200%
> H 7 700%
> I 2 200%
> J 3 300%
>
> It seems that a thread calling
> "org.apache.cassandra.net.IncomingTcpConnection.run" occupies the 100% CPU
> So I guess the code in IncomingTcpConnection.java:66 cause high CPU usage.
>
> if (isStream)
> {
> new IncomingStreamReader(socket.getChannel()).read();
> }
>
>
>
> --------------------------------------------------
> From: "Lu Ming" <xl...@live.com>
> Sent: Friday, June 04, 2010 12:55 PM
> To: <ca...@incubator.apache.org>
> Subject: High CPU Usage since 0.6.2
>
>>
>> I have ten 0.5.1 Cassandra nodes in my cluster, and I update them to
>> cassandra to 0.6.2 yesterday.
>> But today I find six cassandra nodes have high CPU usage more than 400%
>> in my 8-core CPU sever.
>> The worst one is more than 760%. It is very serious.
>>
>> I use jvisualvm to watch the worst node, and I found that there are many
>> running threads named "thread-xxx"
>> the status of other threads is waiting and sleeping.
>>
>> "Thread-130" - Thread t@240
>> java.lang.Thread.State: RUNNABLE
>> at sun.misc.Unsafe.setMemory(Native Method)
>> at sun.nio.ch.Util.erase(Util.java:202)
>> at
>> sun.nio.ch.FileChannelImpl.transferFromArbitraryChannel(FileChannelImpl.java:560)
>> at sun.nio.ch.FileChannelImpl.transferFrom(FileChannelImpl.java:603)
>> at
>> org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:62)
>> at
>> org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:66)
>> Locked ownable synchronizers:
>> - None
>>
>> "Thread-126" - Thread t@236
>> java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.FileDispatcher.read0(Native Method)
>> at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:21)
>> at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:233)
>> at sun.nio.ch.IOUtil.read(IOUtil.java:200)
>> at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:236)
>> - locked java.lang.Object@10808561
>> at
>> sun.nio.ch.FileChannelImpl.transferFromArbitraryChannel(FileChannelImpl.java:565)
>> at sun.nio.ch.FileChannelImpl.transferFrom(FileChannelImpl.java:603)
>> at
>> org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:62)
>> at
>> org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:66)
>>
>> Locked ownable synchronizers:
>> - None
>>
>> "Thread-119" - Thread t@229
>> java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.NativeThread.current(Native Method)
>> at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:182)
>> - locked java.lang.Object@65b4abbd
>> - locked java.lang.Object@38773975
>> at
>> sun.nio.ch.FileChannelImpl.transferFromArbitraryChannel(FileChannelImpl.java:565)
>> at sun.nio.ch.FileChannelImpl.transferFrom(FileChannelImpl.java:603)
>> at
>> org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:62)
>> at
>> org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:66)
>>
>> Locked ownable synchronizers:
>> - None
>>
>>
>>
>>
>
Re: High CPU Usage since 0.6.2
Posted by Lu Ming <xl...@live.com>.
I do the Thread Dump on each cassandra node, and count the thread with call
stack string "at
org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:62)atorg.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.jav
a:66)" in "thread-xxx"
then I find an interesting things.
Server Name/Thread Count/Average CPU Usage
A 0 <20%
B 8 760%
C 0 <20%
D 3 300%
E 4 220%
F 0 <20%
G 2 200%
H 7 700%
I 2 200%
J 3 300%
It seems that a thread calling
"org.apache.cassandra.net.IncomingTcpConnection.run" occupies the 100% CPU
So I guess the code in IncomingTcpConnection.java:66 cause high CPU usage.
if (isStream)
{
new IncomingStreamReader(socket.getChannel()).read();
}
--------------------------------------------------
From: "Lu Ming" <xl...@live.com>
Sent: Friday, June 04, 2010 12:55 PM
To: <ca...@incubator.apache.org>
Subject: High CPU Usage since 0.6.2
>
> I have ten 0.5.1 Cassandra nodes in my cluster, and I update them to
> cassandra to 0.6.2 yesterday.
> But today I find six cassandra nodes have high CPU usage more than 400% in
> my 8-core CPU sever.
> The worst one is more than 760%. It is very serious.
>
> I use jvisualvm to watch the worst node, and I found that there are many
> running threads named "thread-xxx"
> the status of other threads is waiting and sleeping.
>
> "Thread-130" - Thread t@240
> java.lang.Thread.State: RUNNABLE
> at sun.misc.Unsafe.setMemory(Native Method)
> at sun.nio.ch.Util.erase(Util.java:202)
> at
> sun.nio.ch.FileChannelImpl.transferFromArbitraryChannel(FileChannelImpl.java:560)
> at sun.nio.ch.FileChannelImpl.transferFrom(FileChannelImpl.java:603)
> at
> org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:62)
> at
> org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:66)
> Locked ownable synchronizers:
> - None
>
> "Thread-126" - Thread t@236
> java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.FileDispatcher.read0(Native Method)
> at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:21)
> at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:233)
> at sun.nio.ch.IOUtil.read(IOUtil.java:200)
> at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:236)
> - locked java.lang.Object@10808561
> at
> sun.nio.ch.FileChannelImpl.transferFromArbitraryChannel(FileChannelImpl.java:565)
> at sun.nio.ch.FileChannelImpl.transferFrom(FileChannelImpl.java:603)
> at
> org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:62)
> at
> org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:66)
>
> Locked ownable synchronizers:
> - None
>
> "Thread-119" - Thread t@229
> java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.NativeThread.current(Native Method)
> at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:182)
> - locked java.lang.Object@65b4abbd
> - locked java.lang.Object@38773975
> at
> sun.nio.ch.FileChannelImpl.transferFromArbitraryChannel(FileChannelImpl.java:565)
> at sun.nio.ch.FileChannelImpl.transferFrom(FileChannelImpl.java:603)
> at
> org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:62)
> at
> org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:66)
>
> Locked ownable synchronizers:
> - None
>
>
>
>
Re: High CPU Usage since 0.6.2
Posted by Gary Dusbabek <gd...@gmail.com>.
Chris,
Can you get me a stack dump of one of the busy nodes (kill -3)?
Gary
On Thu, Jun 3, 2010 at 22:50, Chris Goffinet <go...@digg.com> wrote:
> We're seeing this as well. We were testing with a 40+ node cluster on the latest 0.6 branch from few days ago.
>
> -Chris
>
> On Jun 3, 2010, at 9:55 PM, Lu Ming wrote:
>
>>
>> I have ten 0.5.1 Cassandra nodes in my cluster, and I update them to cassandra to 0.6.2 yesterday.
>> But today I find six cassandra nodes have high CPU usage more than 400% in my 8-core CPU sever.
>> The worst one is more than 760%. It is very serious.
>>
>> I use jvisualvm to watch the worst node, and I found that there are many running threads named "thread-xxx"
>> the status of other threads is waiting and sleeping.
>>
>> "Thread-130" - Thread t@240
>> java.lang.Thread.State: RUNNABLE
>> at sun.misc.Unsafe.setMemory(Native Method)
>> at sun.nio.ch.Util.erase(Util.java:202)
>> at sun.nio.ch.FileChannelImpl.transferFromArbitraryChannel(FileChannelImpl.java:560)
>> at sun.nio.ch.FileChannelImpl.transferFrom(FileChannelImpl.java:603)
>> at org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:62)
>> at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:66)
>> Locked ownable synchronizers:
>> - None
>>
>> "Thread-126" - Thread t@236
>> java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.FileDispatcher.read0(Native Method)
>> at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:21)
>> at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:233)
>> at sun.nio.ch.IOUtil.read(IOUtil.java:200)
>> at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:236)
>> - locked java.lang.Object@10808561
>> at sun.nio.ch.FileChannelImpl.transferFromArbitraryChannel(FileChannelImpl.java:565)
>> at sun.nio.ch.FileChannelImpl.transferFrom(FileChannelImpl.java:603)
>> at org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:62)
>> at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:66)
>>
>> Locked ownable synchronizers:
>> - None
>>
>> "Thread-119" - Thread t@229
>> java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.NativeThread.current(Native Method)
>> at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:182)
>> - locked java.lang.Object@65b4abbd
>> - locked java.lang.Object@38773975
>> at sun.nio.ch.FileChannelImpl.transferFromArbitraryChannel(FileChannelImpl.java:565)
>> at sun.nio.ch.FileChannelImpl.transferFrom(FileChannelImpl.java:603)
>> at org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:62)
>> at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:66)
>>
>> Locked ownable synchronizers:
>> - None
>>
>>
>
>
Re: High CPU Usage since 0.6.2
Posted by Lu Ming <xl...@live.com>.
I notice that: there are more than 100 "CLOSE_WAIT" incomming connections
on storage port 7000
In my two cassandra node:
126 of 146 storage connections is "CLOSE_WAIT"
196 of 217 storage connections is "CLOSE_WAIT"
Is it normal?
--------------------------------------------------
From: "Chris Goffinet" <go...@digg.com>
Sent: Friday, June 04, 2010 1:50 PM
To: <us...@cassandra.apache.org>
Cc: <ca...@incubator.apache.org>
Subject: Re: High CPU Usage since 0.6.2
> We're seeing this as well. We were testing with a 40+ node cluster on the
> latest 0.6 branch from few days ago.
>
> -Chris
>
> On Jun 3, 2010, at 9:55 PM, Lu Ming wrote:
>
>>
>> I have ten 0.5.1 Cassandra nodes in my cluster, and I update them to
>> cassandra to 0.6.2 yesterday.
>> But today I find six cassandra nodes have high CPU usage more than 400%
>> in my 8-core CPU sever.
>> The worst one is more than 760%. It is very serious.
>>
>> I use jvisualvm to watch the worst node, and I found that there are many
>> running threads named "thread-xxx"
>> the status of other threads is waiting and sleeping.
>>
>> "Thread-130" - Thread t@240
>> java.lang.Thread.State: RUNNABLE
>> at sun.misc.Unsafe.setMemory(Native Method)
>> at sun.nio.ch.Util.erase(Util.java:202)
>> at
>> sun.nio.ch.FileChannelImpl.transferFromArbitraryChannel(FileChannelImpl.java:560)
>> at sun.nio.ch.FileChannelImpl.transferFrom(FileChannelImpl.java:603)
>> at
>> org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:62)
>> at
>> org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:66)
>> Locked ownable synchronizers:
>> - None
>>
>> "Thread-126" - Thread t@236
>> java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.FileDispatcher.read0(Native Method)
>> at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:21)
>> at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:233)
>> at sun.nio.ch.IOUtil.read(IOUtil.java:200)
>> at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:236)
>> - locked java.lang.Object@10808561
>> at
>> sun.nio.ch.FileChannelImpl.transferFromArbitraryChannel(FileChannelImpl.java:565)
>> at sun.nio.ch.FileChannelImpl.transferFrom(FileChannelImpl.java:603)
>> at
>> org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:62)
>> at
>> org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:66)
>>
>> Locked ownable synchronizers:
>> - None
>>
>> "Thread-119" - Thread t@229
>> java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.NativeThread.current(Native Method)
>> at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:182)
>> - locked java.lang.Object@65b4abbd
>> - locked java.lang.Object@38773975
>> at
>> sun.nio.ch.FileChannelImpl.transferFromArbitraryChannel(FileChannelImpl.java:565)
>> at sun.nio.ch.FileChannelImpl.transferFrom(FileChannelImpl.java:603)
>> at
>> org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:62)
>> at
>> org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:66)
>>
>> Locked ownable synchronizers:
>> - None
>>
>>
>
>
Re: High CPU Usage since 0.6.2
Posted by Chris Goffinet <go...@digg.com>.
We're seeing this as well. We were testing with a 40+ node cluster on the latest 0.6 branch from few days ago.
-Chris
On Jun 3, 2010, at 9:55 PM, Lu Ming wrote:
>
> I have ten 0.5.1 Cassandra nodes in my cluster, and I update them to cassandra to 0.6.2 yesterday.
> But today I find six cassandra nodes have high CPU usage more than 400% in my 8-core CPU sever.
> The worst one is more than 760%. It is very serious.
>
> I use jvisualvm to watch the worst node, and I found that there are many running threads named "thread-xxx"
> the status of other threads is waiting and sleeping.
>
> "Thread-130" - Thread t@240
> java.lang.Thread.State: RUNNABLE
> at sun.misc.Unsafe.setMemory(Native Method)
> at sun.nio.ch.Util.erase(Util.java:202)
> at sun.nio.ch.FileChannelImpl.transferFromArbitraryChannel(FileChannelImpl.java:560)
> at sun.nio.ch.FileChannelImpl.transferFrom(FileChannelImpl.java:603)
> at org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:62)
> at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:66)
> Locked ownable synchronizers:
> - None
>
> "Thread-126" - Thread t@236
> java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.FileDispatcher.read0(Native Method)
> at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:21)
> at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:233)
> at sun.nio.ch.IOUtil.read(IOUtil.java:200)
> at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:236)
> - locked java.lang.Object@10808561
> at sun.nio.ch.FileChannelImpl.transferFromArbitraryChannel(FileChannelImpl.java:565)
> at sun.nio.ch.FileChannelImpl.transferFrom(FileChannelImpl.java:603)
> at org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:62)
> at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:66)
>
> Locked ownable synchronizers:
> - None
>
> "Thread-119" - Thread t@229
> java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.NativeThread.current(Native Method)
> at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:182)
> - locked java.lang.Object@65b4abbd
> - locked java.lang.Object@38773975
> at sun.nio.ch.FileChannelImpl.transferFromArbitraryChannel(FileChannelImpl.java:565)
> at sun.nio.ch.FileChannelImpl.transferFrom(FileChannelImpl.java:603)
> at org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:62)
> at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:66)
>
> Locked ownable synchronizers:
> - None
>
>
High CPU Usage since 0.6.2
Posted by Lu Ming <xl...@live.com>.
I have ten 0.5.1 Cassandra nodes in my cluster, and I update them to
cassandra to 0.6.2 yesterday.
But today I find six cassandra nodes have high CPU usage more than 400% in
my 8-core CPU sever.
The worst one is more than 760%. It is very serious.
I use jvisualvm to watch the worst node, and I found that there are many
running threads named "thread-xxx"
the status of other threads is waiting and sleeping.
"Thread-130" - Thread t@240
java.lang.Thread.State: RUNNABLE
at sun.misc.Unsafe.setMemory(Native Method)
at sun.nio.ch.Util.erase(Util.java:202)
at
sun.nio.ch.FileChannelImpl.transferFromArbitraryChannel(FileChannelImpl.java:560)
at sun.nio.ch.FileChannelImpl.transferFrom(FileChannelImpl.java:603)
at
org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:62)
at
org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:66)
Locked ownable synchronizers:
- None
"Thread-126" - Thread t@236
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.FileDispatcher.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:21)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:233)
at sun.nio.ch.IOUtil.read(IOUtil.java:200)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:236)
- locked java.lang.Object@10808561
at
sun.nio.ch.FileChannelImpl.transferFromArbitraryChannel(FileChannelImpl.java:565)
at sun.nio.ch.FileChannelImpl.transferFrom(FileChannelImpl.java:603)
at
org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:62)
at
org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:66)
Locked ownable synchronizers:
- None
"Thread-119" - Thread t@229
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.NativeThread.current(Native Method)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:182)
- locked java.lang.Object@65b4abbd
- locked java.lang.Object@38773975
at
sun.nio.ch.FileChannelImpl.transferFromArbitraryChannel(FileChannelImpl.java:565)
at sun.nio.ch.FileChannelImpl.transferFrom(FileChannelImpl.java:603)
at
org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:62)
at
org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:66)
Locked ownable synchronizers:
- None
Re: Cassandra hardware - balancing CPU/memory/iops/disk space
Posted by Jesse McConnell <je...@gmail.com>.
let us know how the SSD's pan out, I am curious about that as well
cheers,
jesse
--
jesse mcconnell
jesse.mcconnell@gmail.com
On Tue, Mar 9, 2010 at 12:08, B. Todd Burruss <bb...@real.com> wrote:
> our dataset is too big to fit into cache, so we are hitting disk. not a
> problem for normal operation, but when a node is restored, hinted handoff,
> load balanced, or if reads/write simply build up we see a problem. the
> nodes can't seem to catch up. this seems to be centered around drive seek
> time, not cassandra per se.
>
> to combat we are doing the following:
>
> - add more smaller drives per machine in RAID 0 to combat drive seek time.
> - scale horizontally - add more machines to cluster to spread the load
> - we also plan to try out SSDs as well.
>
>
> Jonathan Ellis wrote:
>>
>> Yes, but I would guess 90% of workloads are better served with
>> spending the extra money on more machines w/ cheap sata disks and lots
>> of ram.
>>
>> -Jonathan
>>
>> On Sun, Mar 7, 2010 at 1:00 PM, Boris Shulman <sh...@gmail.com> wrote:
>>
>>>
>>> Do you think having SAS disks will give better performance?
>>>
>>> On Sat, Mar 6, 2010 at 5:47 PM, Jonathan Ellis <jb...@gmail.com> wrote:
>>>
>>>>
>>>> I think http://wiki.apache.org/cassandra/CassandraHardware answers
>>>> most of your questions.
>>>>
>>>> If possible, it's definitely useful to try out a small fraction of
>>>> your anticipated workload against a test cluster, even a single node,
>>>> before finalizing your production hardware purchase.
>>>>
>>>> On Sat, Mar 6, 2010 at 1:12 AM, Rosenberry, Eric
>>>> <er...@iovation.com> wrote:
>>>>
>>>>>
>>>>> I am looking for advice from others that are further along in deploying
>>>>> Cassandra in production environments than we are. I want to know what
>>>>> you
>>>>> are finding your bottlenecks to be. I would feel silly purchasing dual
>>>>> processor quad core 2.93ghz Nehalem machines with 192 gigs of RAM just
>>>>> to
>>>>> find out that the two local SATA disks kept all that CPU and RAM from
>>>>> being
>>>>> useful (clearly that example would be a dumb).
>>>>>
>>>>>
>>>>>
>>>>> I need to spec out hardware for an “optimal” Cassandra node (though our
>>>>> read/write characteristics are not yet fully defined so let’s go with
>>>>> an
>>>>> “average” configuration).
>>>>>
>>>>>
>>>>>
>>>>> My main concern is finding the right balance of:
>>>>>
>>>>> · Available CPU
>>>>>
>>>>> · RAM amount
>>>>>
>>>>> · RAM speed (think Nehalem architecture where memory comes in a
>>>>> few
>>>>> speeds, though I doubt this is much of a concern as it is mainly
>>>>> dictated by
>>>>> which processor you buy and how many slots you populate)
>>>>>
>>>>> · Total iops available (i.e. number of disks)
>>>>>
>>>>> · Total disk space available (depending on the ratio of
>>>>> iops/space
>>>>> deciding on SAS vs. SATA and various rotational speeds)
>>>>>
>>>>>
>>>>>
>>>>> My current thinking is 1U boxes with four 3.5 inch disks since that
>>>>> seems to
>>>>> be a readily available config. One big question is should I go with a
>>>>> single processor Nehalem system to go with those four disks, or would
>>>>> two
>>>>> CPU’s be useful, and also, how much RAM is appropriate to match? I am
>>>>> making the assumption that Cassandra nodes are going to be disk bound
>>>>> as
>>>>> they must do a random read to answer any given query (i.e. indexes in
>>>>> RAM,
>>>>> but all data lives on disk?).
>>>>>
>>>>>
>>>>>
>>>>> The other big decision is what type of hard disks others are finding to
>>>>> provide the optimal ratio of iops to available space? SAS or SATA?
>>>>> And
>>>>> what rotational speed?
>>>>>
>>>>>
>>>>>
>>>>> Let me throw out here an actual hardware config and feel free to tell
>>>>> me the
>>>>> error of my ways:
>>>>>
>>>>> · A SuperMicro SuperServer 6016T-NTRF configured as follows:
>>>>>
>>>>> o 2.26 ghz E5520 dual processor quad core hyperthreaded Nehalem
>>>>> architecture (this proc provides a lot of bang for the buck, faster
>>>>> procs
>>>>> get more expensive quickly)
>>>>>
>>>>> o Qty 12, 4 gig 1066mhz DIMMS for a total of 48 gigs RAM (the 4 gig
>>>>> DIMMS
>>>>> seem to be the price sweet spot)
>>>>>
>>>>> o Dual on board 1 gigabit NIC’s (perhaps one for client connections
>>>>> and
>>>>> the other for cluster communication?)
>>>>>
>>>>> o Dual power supplies (I don’t want to lose half my cluster due to a
>>>>> failure on one power leg)
>>>>>
>>>>> o 4x 1TB SATA disks (this is a complete SWAG)
>>>>>
>>>>> o No RAID controller (all just single individual disks presented to
>>>>> the
>>>>> OS) – Though is there any down side to using a RAID controller with
>>>>> RAID 0
>>>>> (perhaps one single disk for the log for sequential io’s, and 3x disks
>>>>> in a
>>>>> stripe for the random io’s)
>>>>>
>>>>> o The on-board IPMI based OOB controller (so we can kick the boxes
>>>>> remotely if need be)
>>>>>
>>>>> ·
>>>>> http://www.supermicro.com/products/system/1U/6016/SYS-6016T-NTRF.cfm
>>>>>
>>>>>
>>>>>
>>>>> I can’t help but think the above config has way too much RAM and CPU
>>>>> and not
>>>>> enough iops capacity. My understanding is that Cassandra does not
>>>>> cache
>>>>> much in RAM though?
>>>>>
>>>>>
>>>>>
>>>>> Any thoughts are appreciated. Thanks.
>>>>>
>>>>>
>>>>>
>>>>> -Eric
>>>>>
>>>>> _______________________________________________________________
>>>>> Eric Rosenberry
>>>>> Sr. Infrastructure Architect | Chief Bit Plumber
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> iovation
>>>>> 111 SW Fifth Avenue
>>>>> Suite 3200
>>>>> Portland, OR 97204
>>>>> www.iovation.com
>>>>>
>>>>> The information contained in this email message may be privileged,
>>>>> confidential and protected from disclosure. If you are not the intended
>>>>> recipient, any dissemination, distribution or copying is strictly
>>>>> prohibited. If you think that you have received this email message in
>>>>> error,
>>>>> please notify the sender by reply email and delete the message and any
>>>>> attachments.
>>>>>
>
Re: Cassandra hardware - balancing CPU/memory/iops/disk space
Posted by "B. Todd Burruss" <bb...@real.com>.
our dataset is too big to fit into cache, so we are hitting disk. not a
problem for normal operation, but when a node is restored, hinted
handoff, load balanced, or if reads/write simply build up we see a
problem. the nodes can't seem to catch up. this seems to be centered
around drive seek time, not cassandra per se.
to combat we are doing the following:
- add more smaller drives per machine in RAID 0 to combat drive seek time.
- scale horizontally - add more machines to cluster to spread the load
- we also plan to try out SSDs as well.
Jonathan Ellis wrote:
> Yes, but I would guess 90% of workloads are better served with
> spending the extra money on more machines w/ cheap sata disks and lots
> of ram.
>
> -Jonathan
>
> On Sun, Mar 7, 2010 at 1:00 PM, Boris Shulman <sh...@gmail.com> wrote:
>
>> Do you think having SAS disks will give better performance?
>>
>> On Sat, Mar 6, 2010 at 5:47 PM, Jonathan Ellis <jb...@gmail.com> wrote:
>>
>>> I think http://wiki.apache.org/cassandra/CassandraHardware answers
>>> most of your questions.
>>>
>>> If possible, it's definitely useful to try out a small fraction of
>>> your anticipated workload against a test cluster, even a single node,
>>> before finalizing your production hardware purchase.
>>>
>>> On Sat, Mar 6, 2010 at 1:12 AM, Rosenberry, Eric
>>> <er...@iovation.com> wrote:
>>>
>>>> I am looking for advice from others that are further along in deploying
>>>> Cassandra in production environments than we are. I want to know what you
>>>> are finding your bottlenecks to be. I would feel silly purchasing dual
>>>> processor quad core 2.93ghz Nehalem machines with 192 gigs of RAM just to
>>>> find out that the two local SATA disks kept all that CPU and RAM from being
>>>> useful (clearly that example would be a dumb).
>>>>
>>>>
>>>>
>>>> I need to spec out hardware for an “optimal” Cassandra node (though our
>>>> read/write characteristics are not yet fully defined so let’s go with an
>>>> “average” configuration).
>>>>
>>>>
>>>>
>>>> My main concern is finding the right balance of:
>>>>
>>>> · Available CPU
>>>>
>>>> · RAM amount
>>>>
>>>> · RAM speed (think Nehalem architecture where memory comes in a few
>>>> speeds, though I doubt this is much of a concern as it is mainly dictated by
>>>> which processor you buy and how many slots you populate)
>>>>
>>>> · Total iops available (i.e. number of disks)
>>>>
>>>> · Total disk space available (depending on the ratio of iops/space
>>>> deciding on SAS vs. SATA and various rotational speeds)
>>>>
>>>>
>>>>
>>>> My current thinking is 1U boxes with four 3.5 inch disks since that seems to
>>>> be a readily available config. One big question is should I go with a
>>>> single processor Nehalem system to go with those four disks, or would two
>>>> CPU’s be useful, and also, how much RAM is appropriate to match? I am
>>>> making the assumption that Cassandra nodes are going to be disk bound as
>>>> they must do a random read to answer any given query (i.e. indexes in RAM,
>>>> but all data lives on disk?).
>>>>
>>>>
>>>>
>>>> The other big decision is what type of hard disks others are finding to
>>>> provide the optimal ratio of iops to available space? SAS or SATA? And
>>>> what rotational speed?
>>>>
>>>>
>>>>
>>>> Let me throw out here an actual hardware config and feel free to tell me the
>>>> error of my ways:
>>>>
>>>> · A SuperMicro SuperServer 6016T-NTRF configured as follows:
>>>>
>>>> o 2.26 ghz E5520 dual processor quad core hyperthreaded Nehalem
>>>> architecture (this proc provides a lot of bang for the buck, faster procs
>>>> get more expensive quickly)
>>>>
>>>> o Qty 12, 4 gig 1066mhz DIMMS for a total of 48 gigs RAM (the 4 gig DIMMS
>>>> seem to be the price sweet spot)
>>>>
>>>> o Dual on board 1 gigabit NIC’s (perhaps one for client connections and
>>>> the other for cluster communication?)
>>>>
>>>> o Dual power supplies (I don’t want to lose half my cluster due to a
>>>> failure on one power leg)
>>>>
>>>> o 4x 1TB SATA disks (this is a complete SWAG)
>>>>
>>>> o No RAID controller (all just single individual disks presented to the
>>>> OS) – Though is there any down side to using a RAID controller with RAID 0
>>>> (perhaps one single disk for the log for sequential io’s, and 3x disks in a
>>>> stripe for the random io’s)
>>>>
>>>> o The on-board IPMI based OOB controller (so we can kick the boxes
>>>> remotely if need be)
>>>>
>>>> ·
>>>> http://www.supermicro.com/products/system/1U/6016/SYS-6016T-NTRF.cfm
>>>>
>>>>
>>>>
>>>> I can’t help but think the above config has way too much RAM and CPU and not
>>>> enough iops capacity. My understanding is that Cassandra does not cache
>>>> much in RAM though?
>>>>
>>>>
>>>>
>>>> Any thoughts are appreciated. Thanks.
>>>>
>>>>
>>>>
>>>> -Eric
>>>>
>>>> _______________________________________________________________
>>>> Eric Rosenberry
>>>> Sr. Infrastructure Architect | Chief Bit Plumber
>>>>
>>>>
>>>>
>>>>
>>>> iovation
>>>> 111 SW Fifth Avenue
>>>> Suite 3200
>>>> Portland, OR 97204
>>>> www.iovation.com
>>>>
>>>> The information contained in this email message may be privileged,
>>>> confidential and protected from disclosure. If you are not the intended
>>>> recipient, any dissemination, distribution or copying is strictly
>>>> prohibited. If you think that you have received this email message in error,
>>>> please notify the sender by reply email and delete the message and any
>>>> attachments.
>>>>
Re: Cassandra hardware - balancing CPU/memory/iops/disk space
Posted by Jonathan Ellis <jb...@gmail.com>.
Yes, but I would guess 90% of workloads are better served with
spending the extra money on more machines w/ cheap sata disks and lots
of ram.
-Jonathan
On Sun, Mar 7, 2010 at 1:00 PM, Boris Shulman <sh...@gmail.com> wrote:
> Do you think having SAS disks will give better performance?
>
> On Sat, Mar 6, 2010 at 5:47 PM, Jonathan Ellis <jb...@gmail.com> wrote:
>> I think http://wiki.apache.org/cassandra/CassandraHardware answers
>> most of your questions.
>>
>> If possible, it's definitely useful to try out a small fraction of
>> your anticipated workload against a test cluster, even a single node,
>> before finalizing your production hardware purchase.
>>
>> On Sat, Mar 6, 2010 at 1:12 AM, Rosenberry, Eric
>> <er...@iovation.com> wrote:
>>> I am looking for advice from others that are further along in deploying
>>> Cassandra in production environments than we are. I want to know what you
>>> are finding your bottlenecks to be. I would feel silly purchasing dual
>>> processor quad core 2.93ghz Nehalem machines with 192 gigs of RAM just to
>>> find out that the two local SATA disks kept all that CPU and RAM from being
>>> useful (clearly that example would be a dumb).
>>>
>>>
>>>
>>> I need to spec out hardware for an “optimal” Cassandra node (though our
>>> read/write characteristics are not yet fully defined so let’s go with an
>>> “average” configuration).
>>>
>>>
>>>
>>> My main concern is finding the right balance of:
>>>
>>> · Available CPU
>>>
>>> · RAM amount
>>>
>>> · RAM speed (think Nehalem architecture where memory comes in a few
>>> speeds, though I doubt this is much of a concern as it is mainly dictated by
>>> which processor you buy and how many slots you populate)
>>>
>>> · Total iops available (i.e. number of disks)
>>>
>>> · Total disk space available (depending on the ratio of iops/space
>>> deciding on SAS vs. SATA and various rotational speeds)
>>>
>>>
>>>
>>> My current thinking is 1U boxes with four 3.5 inch disks since that seems to
>>> be a readily available config. One big question is should I go with a
>>> single processor Nehalem system to go with those four disks, or would two
>>> CPU’s be useful, and also, how much RAM is appropriate to match? I am
>>> making the assumption that Cassandra nodes are going to be disk bound as
>>> they must do a random read to answer any given query (i.e. indexes in RAM,
>>> but all data lives on disk?).
>>>
>>>
>>>
>>> The other big decision is what type of hard disks others are finding to
>>> provide the optimal ratio of iops to available space? SAS or SATA? And
>>> what rotational speed?
>>>
>>>
>>>
>>> Let me throw out here an actual hardware config and feel free to tell me the
>>> error of my ways:
>>>
>>> · A SuperMicro SuperServer 6016T-NTRF configured as follows:
>>>
>>> o 2.26 ghz E5520 dual processor quad core hyperthreaded Nehalem
>>> architecture (this proc provides a lot of bang for the buck, faster procs
>>> get more expensive quickly)
>>>
>>> o Qty 12, 4 gig 1066mhz DIMMS for a total of 48 gigs RAM (the 4 gig DIMMS
>>> seem to be the price sweet spot)
>>>
>>> o Dual on board 1 gigabit NIC’s (perhaps one for client connections and
>>> the other for cluster communication?)
>>>
>>> o Dual power supplies (I don’t want to lose half my cluster due to a
>>> failure on one power leg)
>>>
>>> o 4x 1TB SATA disks (this is a complete SWAG)
>>>
>>> o No RAID controller (all just single individual disks presented to the
>>> OS) – Though is there any down side to using a RAID controller with RAID 0
>>> (perhaps one single disk for the log for sequential io’s, and 3x disks in a
>>> stripe for the random io’s)
>>>
>>> o The on-board IPMI based OOB controller (so we can kick the boxes
>>> remotely if need be)
>>>
>>> ·
>>> http://www.supermicro.com/products/system/1U/6016/SYS-6016T-NTRF.cfm
>>>
>>>
>>>
>>> I can’t help but think the above config has way too much RAM and CPU and not
>>> enough iops capacity. My understanding is that Cassandra does not cache
>>> much in RAM though?
>>>
>>>
>>>
>>> Any thoughts are appreciated. Thanks.
>>>
>>>
>>>
>>> -Eric
>>>
>>> _______________________________________________________________
>>> Eric Rosenberry
>>> Sr. Infrastructure Architect | Chief Bit Plumber
>>>
>>>
>>>
>>>
>>> iovation
>>> 111 SW Fifth Avenue
>>> Suite 3200
>>> Portland, OR 97204
>>> www.iovation.com
>>>
>>> The information contained in this email message may be privileged,
>>> confidential and protected from disclosure. If you are not the intended
>>> recipient, any dissemination, distribution or copying is strictly
>>> prohibited. If you think that you have received this email message in error,
>>> please notify the sender by reply email and delete the message and any
>>> attachments.
>>
>
Re: Cassandra hardware - balancing CPU/memory/iops/disk space
Posted by Boris Shulman <sh...@gmail.com>.
Do you think having SAS disks will give better performance?
On Sat, Mar 6, 2010 at 5:47 PM, Jonathan Ellis <jb...@gmail.com> wrote:
> I think http://wiki.apache.org/cassandra/CassandraHardware answers
> most of your questions.
>
> If possible, it's definitely useful to try out a small fraction of
> your anticipated workload against a test cluster, even a single node,
> before finalizing your production hardware purchase.
>
> On Sat, Mar 6, 2010 at 1:12 AM, Rosenberry, Eric
> <er...@iovation.com> wrote:
>> I am looking for advice from others that are further along in deploying
>> Cassandra in production environments than we are. I want to know what you
>> are finding your bottlenecks to be. I would feel silly purchasing dual
>> processor quad core 2.93ghz Nehalem machines with 192 gigs of RAM just to
>> find out that the two local SATA disks kept all that CPU and RAM from being
>> useful (clearly that example would be a dumb).
>>
>>
>>
>> I need to spec out hardware for an “optimal” Cassandra node (though our
>> read/write characteristics are not yet fully defined so let’s go with an
>> “average” configuration).
>>
>>
>>
>> My main concern is finding the right balance of:
>>
>> · Available CPU
>>
>> · RAM amount
>>
>> · RAM speed (think Nehalem architecture where memory comes in a few
>> speeds, though I doubt this is much of a concern as it is mainly dictated by
>> which processor you buy and how many slots you populate)
>>
>> · Total iops available (i.e. number of disks)
>>
>> · Total disk space available (depending on the ratio of iops/space
>> deciding on SAS vs. SATA and various rotational speeds)
>>
>>
>>
>> My current thinking is 1U boxes with four 3.5 inch disks since that seems to
>> be a readily available config. One big question is should I go with a
>> single processor Nehalem system to go with those four disks, or would two
>> CPU’s be useful, and also, how much RAM is appropriate to match? I am
>> making the assumption that Cassandra nodes are going to be disk bound as
>> they must do a random read to answer any given query (i.e. indexes in RAM,
>> but all data lives on disk?).
>>
>>
>>
>> The other big decision is what type of hard disks others are finding to
>> provide the optimal ratio of iops to available space? SAS or SATA? And
>> what rotational speed?
>>
>>
>>
>> Let me throw out here an actual hardware config and feel free to tell me the
>> error of my ways:
>>
>> · A SuperMicro SuperServer 6016T-NTRF configured as follows:
>>
>> o 2.26 ghz E5520 dual processor quad core hyperthreaded Nehalem
>> architecture (this proc provides a lot of bang for the buck, faster procs
>> get more expensive quickly)
>>
>> o Qty 12, 4 gig 1066mhz DIMMS for a total of 48 gigs RAM (the 4 gig DIMMS
>> seem to be the price sweet spot)
>>
>> o Dual on board 1 gigabit NIC’s (perhaps one for client connections and
>> the other for cluster communication?)
>>
>> o Dual power supplies (I don’t want to lose half my cluster due to a
>> failure on one power leg)
>>
>> o 4x 1TB SATA disks (this is a complete SWAG)
>>
>> o No RAID controller (all just single individual disks presented to the
>> OS) – Though is there any down side to using a RAID controller with RAID 0
>> (perhaps one single disk for the log for sequential io’s, and 3x disks in a
>> stripe for the random io’s)
>>
>> o The on-board IPMI based OOB controller (so we can kick the boxes
>> remotely if need be)
>>
>> ·
>> http://www.supermicro.com/products/system/1U/6016/SYS-6016T-NTRF.cfm
>>
>>
>>
>> I can’t help but think the above config has way too much RAM and CPU and not
>> enough iops capacity. My understanding is that Cassandra does not cache
>> much in RAM though?
>>
>>
>>
>> Any thoughts are appreciated. Thanks.
>>
>>
>>
>> -Eric
>>
>> _______________________________________________________________
>> Eric Rosenberry
>> Sr. Infrastructure Architect | Chief Bit Plumber
>>
>>
>>
>>
>> iovation
>> 111 SW Fifth Avenue
>> Suite 3200
>> Portland, OR 97204
>> www.iovation.com
>>
>> The information contained in this email message may be privileged,
>> confidential and protected from disclosure. If you are not the intended
>> recipient, any dissemination, distribution or copying is strictly
>> prohibited. If you think that you have received this email message in error,
>> please notify the sender by reply email and delete the message and any
>> attachments.
>
Re: Cassandra hardware - balancing CPU/memory/iops/disk space
Posted by Jonathan Ellis <jb...@gmail.com>.
I think http://wiki.apache.org/cassandra/CassandraHardware answers
most of your questions.
If possible, it's definitely useful to try out a small fraction of
your anticipated workload against a test cluster, even a single node,
before finalizing your production hardware purchase.
On Sat, Mar 6, 2010 at 1:12 AM, Rosenberry, Eric
<er...@iovation.com> wrote:
> I am looking for advice from others that are further along in deploying
> Cassandra in production environments than we are. I want to know what you
> are finding your bottlenecks to be. I would feel silly purchasing dual
> processor quad core 2.93ghz Nehalem machines with 192 gigs of RAM just to
> find out that the two local SATA disks kept all that CPU and RAM from being
> useful (clearly that example would be a dumb).
>
>
>
> I need to spec out hardware for an “optimal” Cassandra node (though our
> read/write characteristics are not yet fully defined so let’s go with an
> “average” configuration).
>
>
>
> My main concern is finding the right balance of:
>
> · Available CPU
>
> · RAM amount
>
> · RAM speed (think Nehalem architecture where memory comes in a few
> speeds, though I doubt this is much of a concern as it is mainly dictated by
> which processor you buy and how many slots you populate)
>
> · Total iops available (i.e. number of disks)
>
> · Total disk space available (depending on the ratio of iops/space
> deciding on SAS vs. SATA and various rotational speeds)
>
>
>
> My current thinking is 1U boxes with four 3.5 inch disks since that seems to
> be a readily available config. One big question is should I go with a
> single processor Nehalem system to go with those four disks, or would two
> CPU’s be useful, and also, how much RAM is appropriate to match? I am
> making the assumption that Cassandra nodes are going to be disk bound as
> they must do a random read to answer any given query (i.e. indexes in RAM,
> but all data lives on disk?).
>
>
>
> The other big decision is what type of hard disks others are finding to
> provide the optimal ratio of iops to available space? SAS or SATA? And
> what rotational speed?
>
>
>
> Let me throw out here an actual hardware config and feel free to tell me the
> error of my ways:
>
> · A SuperMicro SuperServer 6016T-NTRF configured as follows:
>
> o 2.26 ghz E5520 dual processor quad core hyperthreaded Nehalem
> architecture (this proc provides a lot of bang for the buck, faster procs
> get more expensive quickly)
>
> o Qty 12, 4 gig 1066mhz DIMMS for a total of 48 gigs RAM (the 4 gig DIMMS
> seem to be the price sweet spot)
>
> o Dual on board 1 gigabit NIC’s (perhaps one for client connections and
> the other for cluster communication?)
>
> o Dual power supplies (I don’t want to lose half my cluster due to a
> failure on one power leg)
>
> o 4x 1TB SATA disks (this is a complete SWAG)
>
> o No RAID controller (all just single individual disks presented to the
> OS) – Though is there any down side to using a RAID controller with RAID 0
> (perhaps one single disk for the log for sequential io’s, and 3x disks in a
> stripe for the random io’s)
>
> o The on-board IPMI based OOB controller (so we can kick the boxes
> remotely if need be)
>
> ·
> http://www.supermicro.com/products/system/1U/6016/SYS-6016T-NTRF.cfm
>
>
>
> I can’t help but think the above config has way too much RAM and CPU and not
> enough iops capacity. My understanding is that Cassandra does not cache
> much in RAM though?
>
>
>
> Any thoughts are appreciated. Thanks.
>
>
>
> -Eric
>
> _______________________________________________________________
> Eric Rosenberry
> Sr. Infrastructure Architect | Chief Bit Plumber
>
>
>
>
> iovation
> 111 SW Fifth Avenue
> Suite 3200
> Portland, OR 97204
> www.iovation.com
>
> The information contained in this email message may be privileged,
> confidential and protected from disclosure. If you are not the intended
> recipient, any dissemination, distribution or copying is strictly
> prohibited. If you think that you have received this email message in error,
> please notify the sender by reply email and delete the message and any
> attachments.
Re: Cassandra hardware - balancing CPU/memory/iops/disk space
Posted by Krishna Sankar <ks...@gmail.com>.
Eric,
Couple of thoughts:
1. Hardware
Definitely dual quad core
12 X 4 DIMMS. This is the sweet spot for memory. I have many
machines with this config and some with the 12 X 2 configs
I haven¹t found the need for SATA and the higher price
Make sure you get good NICs
Are you using any virtualization layer ? I assume these are bare
metal with Ubuntu or RedHat.
2. Scaling
Naturally you should look at horizontal scaling than vertical.
An estimate of the application characteristics and data
properties would be helpful to get a first estimate
I think eventually you will end up with multiple boxes anyway,
so my philosophy has been to buy multiple optimal boxes
We are working on scaling characteristics (memory, network and
storage), unfortunately way too early to make any inferences
HTH.
Cheers
<k/>
On 3/5/10 Fri Mar 5, 10, "Rosenberry, Eric" <er...@iovation.com>
wrote:
> I am looking for advice from others that are further along in deploying
> Cassandra in production environments than we are. I want to know what you are
> finding your bottlenecks to be. I would feel silly purchasing dual processor
> quad core 2.93ghz Nehalem machines with 192 gigs of RAM just to find out that
> the two local SATA disks kept all that CPU and RAM from being useful (clearly
> that example would be a dumb).
>
> I need to spec out hardware for an ³optimal² Cassandra node (though our
> read/write characteristics are not yet fully defined so let¹s go with an
> ³average² configuration).
>
> My main concern is finding the right balance of:
> · Available CPU
>
> · RAM amount
>
> · RAM speed (think Nehalem architecture where memory comes in a few
> speeds, though I doubt this is much of a concern as it is mainly dictated by
> which processor you buy and how many slots you populate)
>
> · Total iops available (i.e. number of disks)
>
> · Total disk space available (depending on the ratio of iops/space
> deciding on SAS vs. SATA and various rotational speeds)
>
>
> My current thinking is 1U boxes with four 3.5 inch disks since that seems to
> be a readily available config. One big question is should I go with a single
> processor Nehalem system to go with those four disks, or would two CPU¹s be
> useful, and also, how much RAM is appropriate to match? I am making the
> assumption that Cassandra nodes are going to be disk bound as they must do a
> random read to answer any given query (i.e. indexes in RAM, but all data lives
> on disk?).
>
> The other big decision is what type of hard disks others are finding to
> provide the optimal ratio of iops to available space? SAS or SATA? And what
> rotational speed?
>
> Let me throw out here an actual hardware config and feel free to tell me the
> error of my ways:
> · A SuperMicro SuperServer 6016T-NTRF configured as follows:
>
> o 2.26 ghz E5520 dual processor quad core hyperthreaded Nehalem architecture
> (this proc provides a lot of bang for the buck, faster procs get more
> expensive quickly)
>
> o Qty 12, 4 gig 1066mhz DIMMS for a total of 48 gigs RAM (the 4 gig DIMMS
> seem to be the price sweet spot)
>
> o Dual on board 1 gigabit NIC¹s (perhaps one for client connections and the
> other for cluster communication?)
>
> o Dual power supplies (I don¹t want to lose half my cluster due to a failure
> on one power leg)
>
> o 4x 1TB SATA disks (this is a complete SWAG)
>
> o No RAID controller (all just single individual disks presented to the OS)
> Though is there any down side to using a RAID controller with RAID 0 (perhaps
> one single disk for the log for sequential io¹s, and 3x disks in a stripe for
> the random io¹s)
>
> o The on-board IPMI based OOB controller (so we can kick the boxes remotely
> if need be)
>
> · http://www.supermicro.com/products/system/1U/6016/SYS-6016T-NTRF.cfm
>
>
> I can¹t help but think the above config has way too much RAM and CPU and not
> enough iops capacity. My understanding is that Cassandra does not cache much
> in RAM though?
>
> Any thoughts are appreciated. Thanks.
>
> -Eric
> _______________________________________________________________
> Eric Rosenberry
> Sr. Infrastructure Architect | Chief Bit Plumber
>
>
> iovation
> 111 SW Fifth Avenue
> Suite 3200
> Portland, OR 97204
> www.iovation.com <http://www.iovation.com/>
>
> The information contained in this email message may be privileged,
> confidential and protected from disclosure. If you are not the intended
> recipient, any dissemination, distribution or copying is strictly prohibited.
> If you think that you have received this email message in error, please notify
> the sender by reply email and delete the message and any attachments.
>