You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Andrew Cobley <a....@dundee.ac.uk> on 2013/07/22 12:44:40 UTC
Strange cassandra-stress results with 2.0.0 beta1
I've been noticing some strange casandra-stress results with 2.0.0 beta 1. I've set up a single node on a Mac (4 gig ram, 2.8Ghz core 2 duo) and installed 2.0.0 beta1.
When I run ./cassandra-stress -d 134.36.36.218 I'm seeing the interval-op-rate drop from a peek of 11562 at the start to 0 -20 after about 358 seconds. Here's the output:
./cassandra-stress -d 134.36.36.218
Created keyspaces. Sleeping 1s for propagation.
total,interval_op_rate,interval_key_rate,latency,95th,99th,elapsed_time
13629,1362,1362,15.3,117.6,531.1,10
37545,2391,2391,9.4,74.0,438.4,20
105410,6786,6786,3.3,38.0,216.4,30
209014,10360,10360,2.7,26.2,216.5,41
320024,11101,11101,2.5,14.5,216.5,51
430216,11019,11019,2.4,10.5,215.4,61
531905,10168,10168,2.4,8.3,177.1,72
619672,8776,8776,2.4,6.2,145.8,82
653977,3430,3430,2.4,5.9,145.4,92
682648,2867,2867,2.4,6.1,145.4,102
692771,1012,1012,2.4,6.1,145.4,113
712163,1939,1939,2.4,6.0,4461.7,123
723761,1159,1159,2.4,6.0,4461.7,133
731115,735,735,2.4,6.1,4461.7,143
737627,651,651,2.4,6.2,4461.7,153
743853,622,622,2.4,7.0,5056.1,164
747345,349,349,2.4,7.0,5056.1,174
755501,815,815,2.5,6.9,6351.7,184
758692,319,319,2.5,6.9,5056.1,195
763960,526,526,2.5,7.3,5292.6,205
767966,400,400,2.5,7.4,5292.6,215
769514,154,154,2.5,7.4,5292.6,225
773492,397,397,2.5,7.2,6435.7,236
775374,188,188,2.5,7.4,5292.6,246
776035,66,66,2.6,7.5,5347.1,256
777896,186,186,2.6,7.9,6407.3,266
778791,89,89,2.6,8.0,6438.9,276
779646,85,85,2.6,8.0,6523.2,287
780785,113,113,2.6,11.1,6523.2,297
781336,55,55,2.6,11.7,6523.2,307
781684,34,34,2.6,13.3,16734.0,317
781818,13,13,2.7,13.9,16764.2,328
781952,13,13,2.7,13.9,16900.1,338
782195,24,24,2.7,15.3,16900.1,348
782294,9,9,2.7,15.3,16900.1,358
782362,6,6,2.7,15.5,21487.3,368
782589,22,22,2.7,20.1,21487.3,379
782775,18,18,2.7,28.8,21487.3,389
783012,23,23,2.8,29.4,21487.3,399
783248,23,23,2.8,68.2,21487.3,409
783270,2,2,2.8,68.2,21487.3,419
783283,1,1,2.8,68.2,21487.3,430
783283,0,0,2.8,68.2,21487.3,440
783337,5,5,2.8,198.6,42477.2,450
783492,15,15,2.9,5057.6,52687.3,460
783581,8,8,2.9,5349.0,59306.9,471
783807,22,22,3.0,6429.1,59306.9,481
783816,0,0,3.0,6438.3,59306.9,491
783945,12,12,3.2,11566.9,59306.9,501
783962,1,1,3.2,11612.0,59306.9,512
783962,0,0,3.2,11612.0,59306.9,522
I installed 1.2.6 for comparision and that gave expected results:
./cassandra-stress -d 134.36.36.218
Unable to create stress keyspace: Keyspace names must be case-insensitively unique ("Keyspace1" conflicts with "Keyspace1")
total,interval_op_rate,interval_key_rate,latency/95th/99th,elapsed_time
22158,2215,2215,0.8,8.9,549.0,10
64192,4203,4203,1.0,7.0,486.2,20
114236,5004,5004,1.3,6.5,411.9,30
178407,6417,6417,1.4,5.5,411.9,40
281714,10330,10330,2.2,5.1,409.6,51
371684,8997,8997,2.4,5.1,407.5,61
464401,9271,9271,2.5,5.0,236.7,72
562797,9839,9839,2.6,5.2,87.5,82
655755,9295,9295,2.6,5.2,87.5,92
751560,9580,9580,2.6,4.7,179.9,103
848022,9646,9646,2.6,4.5,826.0,113
914539,6651,6651,2.6,4.7,823.2,123
1000000,8546,8546,2.6,5.1,1048.5,133
Any ideas whats causing the slow down ?
Andy
The University of Dundee is a registered Scottish Charity, No: SC015096
Re: Strange cassandra-stress results with 2.0.0 beta1
Posted by Radim Kolar <hs...@filez.com>.
Dne 25.7.2013 20:03, Andrew Cobley napsal(a):
> Any idea on how I can go about pinpointing the problem to raise a JIRA issue ?
http://www.ehow.com/how_8705297_create-java-heap-dump.html
Re: Strange cassandra-stress results with 2.0.0 beta1
Posted by Andrew Cobley <a....@dundee.ac.uk>.
So is that in Cassandra somewhere ? Any idea on how I can go about pinpointing the problem to raise a JIRA issue ?
Andy
On 25 Jul 2013, at 17:50, Radim Kolar <hs...@filez.com> wrote:
>
>> I'm wondering if it's a GC issue ?
> yes it is:
>
> 1039280992 used; max is 1052770304
>
> most likely memory leak.
>
>
The University of Dundee is a registered Scottish Charity, No: SC015096
Re: Strange cassandra-stress results with 2.0.0 beta1
Posted by Radim Kolar <hs...@filez.com>.
> I'm wondering if it's a GC issue ?
yes it is:
1039280992 used; max is 1052770304
most likely memory leak.
Re: Strange cassandra-stress results with 2.0.0 beta1
Posted by Andrew Cobley <a....@dundee.ac.uk>.
I'm getting something similar for beta 2, you can see here it's dropping from 12000 to 400 quite quickly. The output from cassandra is available n dropbox here:
https://dl.dropboxusercontent.com/u/1638201/output.txt (184kBytes)
I'm wondering if it's a GC issue ?
./cassandra-stress -d 134.36.36.218 -n 30000000
Unable to create stress keyspace: Keyspace names must be case-insensitively unique ("Keyspace1" conflicts with "Keyspace1")
total,interval_op_rate,interval_key_rate,latency/95th/99th,elapsed_time
73761,7376,7376,2.1,7.6,181.0,10
199159,12539,12539,2.4,7.5,102.4,20
316978,11781,11781,2.4,7.8,172.7,30
434060,11708,11708,2.4,8.1,170.6,40
552650,11859,11859,2.5,7.3,170.8,50
602432,4978,4978,2.5,7.3,170.8,60
614364,1193,1193,2.5,7.5,5490.4,71
634199,1983,1983,2.5,8.1,5490.4,81
653293,1909,1909,2.5,8.9,5490.4,91
658398,510,510,2.5,8.7,5490.4,101
671805,1340,1340,2.6,8.7,5490.4,111
678677,687,687,2.6,9.1,5490.4,121
685247,657,657,2.6,10.2,5626.8,131
691519,627,627,2.6,10.7,5626.8,141
693682,216,216,2.6,10.7,5626.8,151
700082,640,640,2.7,11.9,6460.1,161
703952,387,387,2.7,12.5,6460.1,172
706007,205,205,2.7,12.5,6460.1,182
709988,398,398,2.7,12.7,6647.0,192
^Cmacaroon:bin administrator$
Andy C
On 25 Jul 2013, at 15:28, Sávio Teles <sa...@lupa.inf.ufg.br>> wrote:
Some bug was fixed in 2.0.0-beta2 by C* developers. Try it!
2013/7/22 Andrew Cobley <a....@dundee.ac.uk>>
I've been noticing some strange casandra-stress results with 2.0.0 beta 1. I've set up a single node on a Mac (4 gig ram, 2.8Ghz core 2 duo) and installed 2.0.0 beta1.
When I run ./cassandra-stress -d 134.36.36.218 I'm seeing the interval-op-rate drop from a peek of 11562 at the start to 0 -20 after about 358 seconds. Here's the output:
The University of Dundee is a registered Scottish Charity, No: SC015096
Re: Strange cassandra-stress results with 2.0.0 beta1
Posted by Sávio Teles <sa...@lupa.inf.ufg.br>.
Some bug was fixed in 2.0.0-beta2 by C* developers. Try it!
2013/7/22 Andrew Cobley <a....@dundee.ac.uk>
> I've been noticing some strange casandra-stress results with 2.0.0 beta
> 1. I've set up a single node on a Mac (4 gig ram, 2.8Ghz core 2 duo) and
> installed 2.0.0 beta1.
>
> When I run ./cassandra-stress -d 134.36.36.218 I'm seeing the
> interval-op-rate drop from a peek of 11562 at the start to 0 -20 after
> about 358 seconds. Here's the output:
>
> ./cassandra-stress -d 134.36.36.218
> Created keyspaces. Sleeping 1s for propagation.
> total,interval_op_rate,interval_key_rate,latency,95th,99th,elapsed_time
> 13629,1362,1362,15.3,117.6,531.1,10
> 37545,2391,2391,9.4,74.0,438.4,20
> 105410,6786,6786,3.3,38.0,216.4,30
> 209014,10360,10360,2.7,26.2,216.5,41
> 320024,11101,11101,2.5,14.5,216.5,51
> 430216,11019,11019,2.4,10.5,215.4,61
> 531905,10168,10168,2.4,8.3,177.1,72
> 619672,8776,8776,2.4,6.2,145.8,82
> 653977,3430,3430,2.4,5.9,145.4,92
> 682648,2867,2867,2.4,6.1,145.4,102
> 692771,1012,1012,2.4,6.1,145.4,113
> 712163,1939,1939,2.4,6.0,4461.7,123
> 723761,1159,1159,2.4,6.0,4461.7,133
> 731115,735,735,2.4,6.1,4461.7,143
> 737627,651,651,2.4,6.2,4461.7,153
> 743853,622,622,2.4,7.0,5056.1,164
> 747345,349,349,2.4,7.0,5056.1,174
> 755501,815,815,2.5,6.9,6351.7,184
> 758692,319,319,2.5,6.9,5056.1,195
> 763960,526,526,2.5,7.3,5292.6,205
> 767966,400,400,2.5,7.4,5292.6,215
> 769514,154,154,2.5,7.4,5292.6,225
> 773492,397,397,2.5,7.2,6435.7,236
> 775374,188,188,2.5,7.4,5292.6,246
> 776035,66,66,2.6,7.5,5347.1,256
> 777896,186,186,2.6,7.9,6407.3,266
> 778791,89,89,2.6,8.0,6438.9,276
> 779646,85,85,2.6,8.0,6523.2,287
> 780785,113,113,2.6,11.1,6523.2,297
> 781336,55,55,2.6,11.7,6523.2,307
> 781684,34,34,2.6,13.3,16734.0,317
> 781818,13,13,2.7,13.9,16764.2,328
> 781952,13,13,2.7,13.9,16900.1,338
> 782195,24,24,2.7,15.3,16900.1,348
> 782294,9,9,2.7,15.3,16900.1,358
> 782362,6,6,2.7,15.5,21487.3,368
> 782589,22,22,2.7,20.1,21487.3,379
> 782775,18,18,2.7,28.8,21487.3,389
> 783012,23,23,2.8,29.4,21487.3,399
> 783248,23,23,2.8,68.2,21487.3,409
> 783270,2,2,2.8,68.2,21487.3,419
> 783283,1,1,2.8,68.2,21487.3,430
> 783283,0,0,2.8,68.2,21487.3,440
> 783337,5,5,2.8,198.6,42477.2,450
> 783492,15,15,2.9,5057.6,52687.3,460
> 783581,8,8,2.9,5349.0,59306.9,471
> 783807,22,22,3.0,6429.1,59306.9,481
> 783816,0,0,3.0,6438.3,59306.9,491
> 783945,12,12,3.2,11566.9,59306.9,501
> 783962,1,1,3.2,11612.0,59306.9,512
> 783962,0,0,3.2,11612.0,59306.9,522
>
>
> I installed 1.2.6 for comparision and that gave expected results:
>
> ./cassandra-stress -d 134.36.36.218
> Unable to create stress keyspace: Keyspace names must be
> case-insensitively unique ("Keyspace1" conflicts with "Keyspace1")
> total,interval_op_rate,interval_key_rate,latency/95th/99th,elapsed_time
> 22158,2215,2215,0.8,8.9,549.0,10
> 64192,4203,4203,1.0,7.0,486.2,20
> 114236,5004,5004,1.3,6.5,411.9,30
> 178407,6417,6417,1.4,5.5,411.9,40
> 281714,10330,10330,2.2,5.1,409.6,51
> 371684,8997,8997,2.4,5.1,407.5,61
> 464401,9271,9271,2.5,5.0,236.7,72
> 562797,9839,9839,2.6,5.2,87.5,82
> 655755,9295,9295,2.6,5.2,87.5,92
> 751560,9580,9580,2.6,4.7,179.9,103
> 848022,9646,9646,2.6,4.5,826.0,113
> 914539,6651,6651,2.6,4.7,823.2,123
> 1000000,8546,8546,2.6,5.1,1048.5,133
>
> Any ideas whats causing the slow down ?
>
> Andy
>
>
> The University of Dundee is a registered Scottish Charity, No: SC015096
>
--
Atenciosamente,
Sávio S. Teles de Oliveira
voice: +55 62 9136 6996
http://br.linkedin.com/in/savioteles
Mestrando em Ciências da Computação - UFG
Arquiteto de Software
Laboratory for Ubiquitous and Pervasive Applications (LUPA) - UFG