CPU usage is increased on node addition, even after bootstrap is finished

[Thanasis Naskos <an...@csd.auth.gr>]

Hi,

CLUSTER SETUP
I'm using Cassandra 2.2.3 running on small private cloud infrastructure 
(supported by ganeti+KVM).
I have an initial Cassandra cluster of 8 nodes and a keyspace with 
Simple Strategy and Replication Factor 3, which is loaded with 2GBs of 
data (2GBs * 3rep = ~6GBs data in total).
On every Cassandra node I'm running ganglia to collect measurements of 
various metrics like incoming load, throughput, response latency and CPU 
usage.
Every Cassandra node has 2 vCPUs, 80GB HDD (commitlog NOT in a separate 
disk) and 6GB RAM.

CLIENTS SETUP
I'm using YCSB benchmark to produce READ load, using CQL clients 
(v2.1.8) with asynchronous calls.
21 YCSB client threads produce ~780 read requests/second each (~16380 
req/sec in total).

EXPERIMENT 1
I'm keeping the load steady to ~16380 r/s and I'm adding 1 node 
periodically. After adding a new node and after the bootstrapping is 
over, I expect both the response latency and the CPU usage to decrease, 
however this is not the case. On every node addition CPU usage increases 
and resp. latency is either steady or increases too.

MEASUREMENT LOGS
# nodes
	LOAD (req/sec) 	THROUGHPUT (req/sec)
	LATENCY (ms)
	CPU (%)
8
	15668,81
	15679,81
	56,09
	86,67
9
	16177,45
	16185,05
	62,96
	88.61
10
	16353,36
	16343,27
	75,22
	89,48
11
	15723,14
	15682,06
	65,84
	90,0
12
	15348,97
	15327,27
	103,13
	90,40


I moved from
8 to 9 nodes after 10 minutes,
9 to 10 after 15 minutes,
10 to 11 after 25 minutes,
11 to 12 after 25 minutes.
The bootstrapping took about 1,5 minutes on every addition. I didn't run 
nodetool cleanup at all.
The measurements of the table are averages.

EXPERIMENT 2
I have also tried with 7 nodes (up to 12) and lower incoming load ~9600 
req/seq. Both latency and CPU usage are kept to the same level no matter 
the number of nodes (3-4ms latency and 75% CPU load). And I've also run 
nodetool cleanup, still no decrease.

I've read somewhere that the benefits of the node addition in Cassandra 
are linear, am I missing something?

Thanks a lot!
Thanasis