Need Feedback about cassandra-stress tests

[Bhuvan Rawal <bh...@gmail.com>]

Hi All,

I have been trying to set up a cluster for POC and ran cassandra-stress
tests today on an 8 Node cluster with 2 DC having 4 nodes each. The disk
used was SSD with I/O of greater than 500MB per sec, CPU- 2.3 Ghz ,4 Core
Xeon and 8Gigs of RAM in each node.

First of all I would like to thank *Alain Rodriguez* and *Sabastian Estevez*
for helping me to sort out an issue which was troubling me.

I used the schema and queries mentioned in the gist at this link -
https://gist.github.com/tjake/fb166a659e8fe4c8d4a3#file-query2-txt-L14. I
used Network Replication on the keyspace with RF of 3 for each DC, meaning
each key would be replicated to 6

Im not sure if I have managed to get the config right because the read
operations seem to be slow. I would need your comments on them.
Command - $ cassandra-stress user profile=./blogpost.yaml
ops\(insert=1\) -node 10.41.55.21
Insert Operation
type id total ops op/s pk/s row/s
threadCount 4 insert 90259 2994 2994 4088
threadCount 4 total 90259 2994 2994 4088
threadCount 8 insert 161754 5362 5362 7335
threadCount 8 total 161754 5362 5362 7335
threadCount 16 insert 261958 7796 7796 10653
threadCount 16 total 261958 7796 7796 10653
threadCount 24 insert 769757 8132 8132 11115
threadCount 24 total 769757 8132 8132 11115
threadCount 36 insert 571600 9762 9762 13362
threadCount 36 total 571600 9762 9762 13362
threadCount 54 insert 884351 10730 10730 14672
threadCount 54 total 884351 10730 10730 14672
threadCount 81 insert 623874 10919 10919 14931
threadCount 81 total 623874 10919 10919 14931
threadCount 121 insert 1234867 11736 11736 16053
threadCount 121 total 1234867 11736 11736 16053
threadCount 181 insert 2377008 10310 10310 14110
threadCount 181 total 2377008 10310 10310 14110
Command-$cassandra-stress user profile=./blogpost.yaml
ops\(singlepost=2,timeline=1,insert=1\) -node 10.41.55.21
Mixed – Read, Write
Query – singlepost=2,timeline=1,insert=1
type id total ops op/s pk/s row/s
threadCount 4 insert 27179 76 76 103
threadCount 4 singlepost 54118 151 151 151
threadCount 4 timeline 27343 76 76 576
threadCount 4 total 108640 303 303 831
threadCount 8 insert 8642 149 149 203
threadCount 8 singlepost 17838 307 307 307
threadCount 8 timeline 8529 147 147 1107
threadCount 8 total 35009 602 602 1617
threadCount 16 insert 18784 176 176 242
threadCount 16 singlepost 37960 356 356 356
threadCount 16 timeline 19058 179 179 1359
threadCount 16 total 75802 710 710 1957
threadCount 24 insert 21564 151 151 208
threadCount 24 singlepost 44545 313 313 313
threadCount 24 timeline 21553 151 151 1150
threadCount 24 total 87662 615 615 1670
threadCount 36 insert 38054 185 185 252
threadCount 36 singlepost 76495 372 372 372
threadCount 36 timeline 38783 189 189 1443
threadCount 36 total 153332 746 746 2068
threadCount 54 insert 92639 167 167 229
threadCount 54 singlepost 187085 338 338 338
threadCount 54 timeline 92679 167 167 1292
threadCount 54 total 372403 673 673 1859
Command - $ cassandra-stress user profile=./blogpost.yaml
ops\(singlepost=1\) -node 10.41.55.21
Read only load, single value, include blog text (which is 5000 chars)
– singlepost query
Query- select * from blogposts where domain = ? LIMIT 1
type id total ops op/s pk/s row/s
threadCount 4 singlepost 11334 368 368 368
threadCount 4 total 11334 368 368 368
threadCount 8 singlepost 18744 547 547 547
threadCount 8 total 18744 547 547 547
threadCount 16 singlepost 26038 607 607 607
threadCount 16 total 26038 607 607 607
threadCount 24 singlepost 35903 654 654 654
threadCount 24 total 35903 654 654 654
threadCount 36 singlepost 34293 682 682 682
threadCount 36 total 34293 682 682 682
threadCount 54 singlepost 45267 707 707 707
threadCount 54 total 45267 707 707 707
threadCount 81 singlepost 49626 766 766 766
threadCount 81 total 49626 766 766 766
threadCount 121 singlepost 58785 800 800 800
threadCount 121 total 58785 800 800 800
threadCount 181 singlepost 38556 795 795 795
threadCount 181 total 38556 795 795 795
threadCount 271 singlepost 47256 838 838 838
threadCount 271 total 47256 838 838 838
threadCount 406 singlepost 57278 901 901 901
threadCount 406 total 57278 901 901 901
threadCount 609 singlepost 73497 904 904 904
threadCount 609 total 73497 904 904 904
threadCount 913 singlepost 88913 929 929 929
threadCount 913 total 88913 929 929 929
Command - $ cassandra-stress user profile=./blogpost.yaml
ops\(timeline=1\) -node 10.41.55.21
Read only load – timeline query
Query - select url, title, published_date from blogposts where domain
= ? LIMIT 10
type id total ops op/s pk/s row/s
threadCount 4 timeline 10805 352 352 2727
threadCount 4 total 10805 352 352 2727
threadCount 8 timeline 14906 471 471 3648
threadCount 8 total 14906 471 471 3648
threadCount 16 timeline 34687 534 534 4148
threadCount 16 total 34687 534 534 4148
threadCount 24 timeline 35705 562 562 4368
threadCount 24 total 35705 562 562 4368
threadCount 36 timeline 46772 593 593 4601
threadCount 36 total 46772 593 593 4601
threadCount 54 timeline 46259 610 610 4738
threadCount 54 total 46259 610 610 4738
threadCount 81 timeline 53709 645 645 4996
threadCount 81 total 53709 645 645 4996
threadCount 121 timeline 46524 650 650 5037
threadCount 121 total 46524 650 650 5037
threadCount 181 timeline 36158 673 673 5224
threadCount 181 total 36158 673 673 5224
threadCount 271 timeline 43682 747 747 5800
threadCount 271 total 43682 747 747 5800
threadCount 406 timeline 55336 785 785 6094
threadCount 406 total 55336 785 785 6094
threadCount 609 timeline 69326 831 831 6449
threadCount 609 total 69326 831 831 6449
threadCount 913 timeline 94283 837 837 6482
threadCount 913 total 94283 837 837 6482

Am I missing something here?

Thanks & Regards,
Bhuvan Rawal

Re: Need Feedback about cassandra-stress tests

[daemeon reiydelle <da...@gmail.com>]

Might I suggest you START by using the default schema provided by
cassandra-stress. Using someone else's schema is great AFTER you use have
used a standard and generally well understood baseline.

>From that you can decide whether a 4 node x 2 cluster is right for you.
FYI, given your 6 way replication, the above do not seem unreasonable. You
can see where YOUR test hits a peak and falls back in throughput. But what
are your USER requirements?

With the default schema (you can try distinct as well as the default
keyspace1 shared keyspace), look very hard at your latencies. If you can
withstand occasional 10 second worst case latencies (10000msec) during
compaction, great. if your 90/99/99.5 latencies are important, you will be
able to see where your tiny cluster plays out. Only THEN can you try some
other no doubt fun schema.



*.......*






*“Life should not be a journey to the grave with the intention of arriving
safely in apretty and well preserved body, but rather to skid in broadside
in a cloud of smoke,thoroughly used up, totally worn out, and loudly
proclaiming “Wow! What a Ride!” - Hunter ThompsonDaemeon C.M. ReiydelleUSA
(+1) 415.501.0198London (+44) (0) 20 8144 9872*

On Sat, Jan 23, 2016 at 10:18 AM, Bhuvan Rawal <bh...@gmail.com> wrote:

> Hi All,
>
> I have been trying to set up a cluster for POC and ran cassandra-stress
> tests today on an 8 Node cluster with 2 DC having 4 nodes each. The disk
> used was SSD with I/O of greater than 500MB per sec, CPU- 2.3 Ghz ,4 Core
> Xeon and 8Gigs of RAM in each node.
>
> First of all I would like to thank *Alain Rodriguez* and *Sabastian
> Estevez* for helping me to sort out an issue which was troubling me.
>
> I used the schema and queries mentioned in the gist at this link -
> https://gist.github.com/tjake/fb166a659e8fe4c8d4a3#file-query2-txt-L14. I
> used Network Replication on the keyspace with RF of 3 for each DC, meaning
> each key would be replicated to 6
>
> Im not sure if I have managed to get the config right because the read
> operations seem to be slow. I would need your comments on them.
>
> Command - $ cassandra-stress user profile=./blogpost.yaml ops\(insert=1\) -node 10.41.55.21
> Insert Operation
> type id total ops op/s pk/s row/s
> threadCount 4 insert 90259 2994 2994 4088
> threadCount 4 total 90259 2994 2994 4088
> threadCount 8 insert 161754 5362 5362 7335
> threadCount 8 total 161754 5362 5362 7335
> threadCount 16 insert 261958 7796 7796 10653
> threadCount 16 total 261958 7796 7796 10653
> threadCount 24 insert 769757 8132 8132 11115
> threadCount 24 total 769757 8132 8132 11115
> threadCount 36 insert 571600 9762 9762 13362
> threadCount 36 total 571600 9762 9762 13362
> threadCount 54 insert 884351 10730 10730 14672
> threadCount 54 total 884351 10730 10730 14672
> threadCount 81 insert 623874 10919 10919 14931
> threadCount 81 total 623874 10919 10919 14931
> threadCount 121 insert 1234867 11736 11736 16053
> threadCount 121 total 1234867 11736 11736 16053
> threadCount 181 insert 2377008 10310 10310 14110
> threadCount 181 total 2377008 10310 10310 14110
>
> Command-$cassandra-stress user profile=./blogpost.yaml ops\(singlepost=2,timeline=1,insert=1\) -node 10.41.55.21
> Mixed – Read, Write
> Query – singlepost=2,timeline=1,insert=1
> type id total ops op/s pk/s row/s
> threadCount 4 insert 27179 76 76 103
> threadCount 4 singlepost 54118 151 151 151
> threadCount 4 timeline 27343 76 76 576
> threadCount 4 total 108640 303 303 831
> threadCount 8 insert 8642 149 149 203
> threadCount 8 singlepost 17838 307 307 307
> threadCount 8 timeline 8529 147 147 1107
> threadCount 8 total 35009 602 602 1617
> threadCount 16 insert 18784 176 176 242
> threadCount 16 singlepost 37960 356 356 356
> threadCount 16 timeline 19058 179 179 1359
> threadCount 16 total 75802 710 710 1957
> threadCount 24 insert 21564 151 151 208
> threadCount 24 singlepost 44545 313 313 313
> threadCount 24 timeline 21553 151 151 1150
> threadCount 24 total 87662 615 615 1670
> threadCount 36 insert 38054 185 185 252
> threadCount 36 singlepost 76495 372 372 372
> threadCount 36 timeline 38783 189 189 1443
> threadCount 36 total 153332 746 746 2068
> threadCount 54 insert 92639 167 167 229
> threadCount 54 singlepost 187085 338 338 338
> threadCount 54 timeline 92679 167 167 1292
> threadCount 54 total 372403 673 673 1859
>
> Command - $ cassandra-stress user profile=./blogpost.yaml ops\(singlepost=1\) -node 10.41.55.21
>
> Read only load, single value, include blog text (which is 5000 chars) – singlepost query
> Query- select * from blogposts where domain = ? LIMIT 1
> type id total ops op/s pk/s row/s
> threadCount 4 singlepost 11334 368 368 368
> threadCount 4 total 11334 368 368 368
> threadCount 8 singlepost 18744 547 547 547
> threadCount 8 total 18744 547 547 547
> threadCount 16 singlepost 26038 607 607 607
> threadCount 16 total 26038 607 607 607
> threadCount 24 singlepost 35903 654 654 654
> threadCount 24 total 35903 654 654 654
> threadCount 36 singlepost 34293 682 682 682
> threadCount 36 total 34293 682 682 682
> threadCount 54 singlepost 45267 707 707 707
> threadCount 54 total 45267 707 707 707
> threadCount 81 singlepost 49626 766 766 766
> threadCount 81 total 49626 766 766 766
> threadCount 121 singlepost 58785 800 800 800
> threadCount 121 total 58785 800 800 800
> threadCount 181 singlepost 38556 795 795 795
> threadCount 181 total 38556 795 795 795
> threadCount 271 singlepost 47256 838 838 838
> threadCount 271 total 47256 838 838 838
> threadCount 406 singlepost 57278 901 901 901
> threadCount 406 total 57278 901 901 901
> threadCount 609 singlepost 73497 904 904 904
> threadCount 609 total 73497 904 904 904
> threadCount 913 singlepost 88913 929 929 929
> threadCount 913 total 88913 929 929 929
>
> Command - $ cassandra-stress user profile=./blogpost.yaml ops\(timeline=1\) -node 10.41.55.21
> Read only load – timeline query
>
> Query - select url, title, published_date from blogposts where domain = ? LIMIT 10
> type id total ops op/s pk/s row/s
> threadCount 4 timeline 10805 352 352 2727
> threadCount 4 total 10805 352 352 2727
> threadCount 8 timeline 14906 471 471 3648
> threadCount 8 total 14906 471 471 3648
> threadCount 16 timeline 34687 534 534 4148
> threadCount 16 total 34687 534 534 4148
> threadCount 24 timeline 35705 562 562 4368
> threadCount 24 total 35705 562 562 4368
> threadCount 36 timeline 46772 593 593 4601
> threadCount 36 total 46772 593 593 4601
> threadCount 54 timeline 46259 610 610 4738
> threadCount 54 total 46259 610 610 4738
> threadCount 81 timeline 53709 645 645 4996
> threadCount 81 total 53709 645 645 4996
> threadCount 121 timeline 46524 650 650 5037
> threadCount 121 total 46524 650 650 5037
> threadCount 181 timeline 36158 673 673 5224
> threadCount 181 total 36158 673 673 5224
> threadCount 271 timeline 43682 747 747 5800
> threadCount 271 total 43682 747 747 5800
> threadCount 406 timeline 55336 785 785 6094
> threadCount 406 total 55336 785 785 6094
> threadCount 609 timeline 69326 831 831 6449
> threadCount 609 total 69326 831 831 6449
> threadCount 913 timeline 94283 837 837 6482
> threadCount 913 total 94283 837 837 6482
>
> Am I missing something here?
>
> Thanks & Regards,
> Bhuvan Rawal
>