What does a cluster throttled by the network look like ?

[Philippe <wa...@gmail.com>]

Dear all,
I'm working with a 12-node, RF=3 cluster on low-end hardware (core i5 with
16GB of RAM & SATA disks).
I'm using a BOP and each node has a load between 50GB and 100GB (yes, I
apparently did not set my tokens right... I'll fix that later).

I'm hitting the cluster with a little over 100 concurrent threads during
reads & writes in counter columns & plain columns. When I peek at my
various servers & look at my munin graphs, it looks like everything is
great : iostat -dmx shows peeks of 5% utilization at worst and vmstat shows
peeks of 30% CPU utilization, no swapping and plenty of caching. Munin
shows me that the JVM's heap is prancing around 50% of its allocated amount
(and I have no row caching for now).

So my thinking has been that the cluster is under-utilized but when I
increase the number of threads hitting the cluster, I do not see the
throughput increase which has been puzzling. I do see an occasional
TimedOutException now that I've made my batches smaller (to dozens of
mutations rather than hundreds) and my slices smaller too.

What I do see is that each server in the cluster has about 60-80Mb/s
traffic in & out and I'm running on 100Mb ethernet. I seem to remember that
you can only get to a little over 80% of the available ethernet bandwidth
in real-life.
Could my performance be limited by the Network Adapters ?

Thanks

Re: What does a cluster throttled by the network look like ?

[David Jeske <da...@gmail.com>]

You are answering your own question here. If you are running at 80% of
network bandwidth, you are saturating your network.

AFAIK - most distributed databases are running on gigabit, not 100mb. I
recommend you upgrade your switch (and nics if necessary). Gigabit is
insanely cheap now. In the extremes, some distributed databases have
dedicated gigabit for internode traffic vs external traffic.

On Oct 30, 2011 3:15 PM, "Philippe" <wa...@gmail.com> wrote:
> What I do see is that each server in the cluster has about 60-80Mb/s
> traffic in & out and I'm running on 100Mb ethernet.