You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Marcus Eriksson (JIRA)" <ji...@apache.org> on 2013/10/02 15:00:26 UTC
[jira] [Comment Edited] (CASSANDRA-4338) Experiment with direct
buffer in SequentialWriter
[ https://issues.apache.org/jira/browse/CASSANDRA-4338?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13783883#comment-13783883 ]
Marcus Eriksson edited comment on CASSANDRA-4338 at 10/2/13 12:58 PM:
----------------------------------------------------------------------
So, got the CompressedSequentialWriter working, code pushed to github: https://github.com/krummas/cassandra/commits/marcuse/4338. It uses snappys direct bytebuffer support, and a custom Adler32 made by me that can checksum direct byte buffers (code here: https://github.com/krummas/adler32 (probably only builds on linux, did not spend much time on it).
Micro benchmarks look great, almost no GC at all with the patched version (the benchmark is left in main(...) in CompressedSequentialWriter.java):
.h2 Trunk
!gc-trunk-me.png!
.h2 Patched
!gc-with-patch-me.png!
Proper single-node stress benchmarks look good as well:
h2. Trunk
{noformat}
total,interval_op_rate,interval_key_rate,latency,95th,99.9th,elapsed_time
394141,39414,39414,0.0,0.0,0.0,10
1078321,68418,68418,0.0,0.0,0.0,20
1726219,64789,64789,0.0,0.0,0.0,30
2327295,60107,60107,0.0,0.0,0.0,40
2928533,60123,60123,0.0,0.0,0.0,50
3533878,60534,60534,0.0,0.0,0.0,60
3602168,6829,6829,0.0,0.0,0.0,70
3967820,36565,36565,0.0,0.0,0.0,80
4647217,67939,67939,0.0,0.0,0.0,91
5248142,60092,60092,0.0,0.0,0.0,101
5930662,68252,68252,0.0,0.0,0.0,111
6417903,48724,48724,0.0,0.0,0.0,121
6952933,53503,53503,0.0,0.0,0.0,131
7221662,26872,26872,0.0,0.0,0.0,141
7221662,0,0,0.0,0.0,0.0,151
7221662,0,0,0.0,0.0,0.0,161
7221662,0,0,0.0,0.0,0.0,172
7221662,0,0,0.0,0.0,0.0,182
7221662,0,0,0.0,0.0,0.0,192
7509240,28757,28757,0.0,0.0,0.0,202
7780984,27174,27174,0.0,0.0,0.0,212
7780984,0,0,0.0,0.0,0.0,222
7780984,0,0,0.0,0.0,0.0,232
7780984,0,0,0.0,0.0,0.0,242
8414140,63315,63315,0.0,0.0,0.0,252
8968246,55410,55410,0.0,0.0,0.0,263
9669857,70161,70161,0.0,0.0,0.0,273
10236467,56661,56661,0.0,0.0,0.0,283
10774593,53812,53812,0.0,0.0,0.0,293
10824657,5006,5006,0.0,0.0,0.0,303
11165174,34051,34051,0.0,0.0,0.0,313
11165174,0,0,0.0,0.0,0.0,323
11165174,0,0,0.0,0.0,0.0,333
11165174,0,0,0.0,0.0,0.0,343
11304248,13907,13907,0.0,0.0,0.0,354
11927380,62313,62313,0.0,0.0,0.0,364
12526960,59958,59958,0.0,0.0,0.0,374
13234647,70768,70768,0.0,0.0,0.0,384
13792652,55800,55800,0.0,0.0,0.0,394
14329718,53706,53706,0.0,0.0,0.0,404
14512350,18263,18263,0.0,0.0,0.0,414
14512929,57,57,0.0,0.0,0.0,424
14710476,19754,19754,0.0,0.0,0.0,434
14710476,0,0,0.0,0.0,0.0,445
14710476,0,0,0.0,0.0,0.0,455
15061043,35056,35056,0.0,0.0,0.0,465
15760509,69946,69946,0.0,0.0,0.0,475
16461318,70080,70080,0.0,0.0,0.0,485
17126749,66543,66543,0.0,0.0,0.0,495
17708154,58140,58140,0.0,0.0,0.0,505
18226801,51864,51864,0.0,0.0,0.0,515
18226801,0,0,0.0,0.0,0.0,526
18227225,42,42,0.0,0.0,0.0,536
18858228,63100,63100,0.0,0.0,0.0,546
19459047,60081,60081,0.0,0.0,0.0,556
19988583,52953,52953,0.0,0.0,0.0,566
20000000,1141,1141,0.0,0.0,0.0,567
Averages from the middle 80% of values:
interval_op_rate : 34003
interval_key_rate : 34003
latency median : 0.0
latency 95th percentile : 0.0
latency 99.9th percentile : 0.0
Total operation time : 00:09:27
END
{noformat}
h2. Patched version
{noformat}
total,interval_op_rate,interval_key_rate,latency,95th,99.9th,elapsed_time
398380,39838,39838,0.0,0.0,0.0,10
1090332,69195,69195,0.0,0.0,0.0,20
1756859,66652,66652,0.0,0.0,0.0,30
2408330,65147,65147,0.0,0.0,0.0,40
3021314,61298,61298,0.0,0.0,0.0,50
3602221,58090,58090,0.0,0.0,0.0,60
3602221,0,0,0.0,0.0,0.0,70
4086404,48418,48418,0.0,0.0,0.0,80
4670997,58459,58459,0.0,0.0,0.0,91
5328657,65766,65766,0.0,0.0,0.0,101
5950535,62187,62187,0.0,0.0,0.0,111
6544475,59394,59394,0.0,0.0,0.0,121
7163644,61916,61916,0.0,0.0,0.0,131
7307634,14399,14399,0.0,0.0,0.0,141
7331684,2405,2405,0.0,0.0,0.0,151
7989707,65802,65802,0.0,0.0,0.0,161
8653302,66359,66359,0.0,0.0,0.0,172
9273188,61988,61988,0.0,0.0,0.0,182
9935986,66279,66279,0.0,0.0,0.0,192
10489010,55302,55302,0.0,0.0,0.0,202
10909996,42098,42098,0.0,0.0,0.0,212
10962871,5287,5287,0.0,0.0,0.0,222
11274293,31142,31142,0.0,0.0,0.0,232
11274293,0,0,0.0,0.0,0.0,242
11274293,0,0,0.0,0.0,0.0,252
11297105,2281,2281,0.0,0.0,0.0,263
11946842,64973,64973,0.0,0.0,0.0,273
12509283,56244,56244,0.0,0.0,0.0,283
13205933,69665,69665,0.0,0.0,0.0,293
13809534,60360,60360,0.0,0.0,0.0,303
14334735,52520,52520,0.0,0.0,0.0,313
14615255,28052,28052,0.0,0.0,0.0,323
14615958,70,70,0.0,0.0,0.0,333
14841997,22603,22603,0.0,0.0,0.0,343
14841997,0,0,0.0,0.0,0.0,354
14841997,0,0,0.0,0.0,0.0,364
15262968,42097,42097,0.0,0.0,0.0,374
15943731,68076,68076,0.0,0.0,0.0,384
16619205,67547,67547,0.0,0.0,0.0,394
17197417,57821,57821,0.0,0.0,0.0,404
17776353,57893,57893,0.0,0.0,0.0,414
18235461,45910,45910,0.0,0.0,0.0,424
18267460,3199,3199,0.0,0.0,0.0,434
18592152,32469,32469,0.0,0.0,0.0,445
18732480,14032,14032,0.0,0.0,0.0,455
19328150,59567,59567,0.0,0.0,0.0,465
19930114,60196,60196,0.0,0.0,0.0,475
20000000,6988,6988,0.0,0.0,0.0,479
Averages from the middle 80% of values:
interval_op_rate : 42156
interval_key_rate : 42156
latency median : 0.0
latency 95th percentile : 0.0
latency 99.9th percentile : 0.0
Total operation time : 00:07:59
END
{noformat}
Note that the trunk run was one of the good runs, many runs it just ended up in a GC loop not making any progress, that never happened on the patched version.
The code is a PoC, and will need more work and another (atleast) pair of eyes.
Maybe [~enigmacurry] can run a proper benchmark? Poke me on irc if you need help to get the adler32 stuff running.
was (Author: krummas):
So, got the CompressedSequentialWriter working, code pushed to github: https://github.com/krummas/cassandra/commits/marcuse/4338. It uses snappys direct bytebuffer support, and a custom Adler32 made by me that can checksum direct byte buffers (code here: https://github.com/krummas/adler32 (probably only builds on linux, did not spend much time on it).
Micro benchmarks look great, almost no GC at all with the patched version (the benchmark is left in main(...) in CompressedSequentialWriter.java):
.h2 Trunk
!gc-trunk-me.png!
.h2 Patched
!gc-with-patch-me.png!
Proper single-node stress benchmarks look good as well:
h2. Trunk
{noformat}
total,interval_op_rate,interval_key_rate,latency,95th,99.9th,elapsed_time
394141,39414,39414,0.0,0.0,0.0,10
1078321,68418,68418,0.0,0.0,0.0,20
1726219,64789,64789,0.0,0.0,0.0,30
2327295,60107,60107,0.0,0.0,0.0,40
2928533,60123,60123,0.0,0.0,0.0,50
3533878,60534,60534,0.0,0.0,0.0,60
3602168,6829,6829,0.0,0.0,0.0,70
3967820,36565,36565,0.0,0.0,0.0,80
4647217,67939,67939,0.0,0.0,0.0,91
5248142,60092,60092,0.0,0.0,0.0,101
5930662,68252,68252,0.0,0.0,0.0,111
6417903,48724,48724,0.0,0.0,0.0,121
6952933,53503,53503,0.0,0.0,0.0,131
7221662,26872,26872,0.0,0.0,0.0,141
7221662,0,0,0.0,0.0,0.0,151
7221662,0,0,0.0,0.0,0.0,161
7221662,0,0,0.0,0.0,0.0,172
7221662,0,0,0.0,0.0,0.0,182
7221662,0,0,0.0,0.0,0.0,192
7509240,28757,28757,0.0,0.0,0.0,202
7780984,27174,27174,0.0,0.0,0.0,212
7780984,0,0,0.0,0.0,0.0,222
7780984,0,0,0.0,0.0,0.0,232
7780984,0,0,0.0,0.0,0.0,242
8414140,63315,63315,0.0,0.0,0.0,252
8968246,55410,55410,0.0,0.0,0.0,263
9669857,70161,70161,0.0,0.0,0.0,273
10236467,56661,56661,0.0,0.0,0.0,283
10774593,53812,53812,0.0,0.0,0.0,293
10824657,5006,5006,0.0,0.0,0.0,303
11165174,34051,34051,0.0,0.0,0.0,313
11165174,0,0,0.0,0.0,0.0,323
11165174,0,0,0.0,0.0,0.0,333
11165174,0,0,0.0,0.0,0.0,343
11304248,13907,13907,0.0,0.0,0.0,354
11927380,62313,62313,0.0,0.0,0.0,364
12526960,59958,59958,0.0,0.0,0.0,374
13234647,70768,70768,0.0,0.0,0.0,384
13792652,55800,55800,0.0,0.0,0.0,394
14329718,53706,53706,0.0,0.0,0.0,404
14512350,18263,18263,0.0,0.0,0.0,414
14512929,57,57,0.0,0.0,0.0,424
14710476,19754,19754,0.0,0.0,0.0,434
14710476,0,0,0.0,0.0,0.0,445
14710476,0,0,0.0,0.0,0.0,455
15061043,35056,35056,0.0,0.0,0.0,465
15760509,69946,69946,0.0,0.0,0.0,475
16461318,70080,70080,0.0,0.0,0.0,485
17126749,66543,66543,0.0,0.0,0.0,495
17708154,58140,58140,0.0,0.0,0.0,505
18226801,51864,51864,0.0,0.0,0.0,515
18226801,0,0,0.0,0.0,0.0,526
18227225,42,42,0.0,0.0,0.0,536
18858228,63100,63100,0.0,0.0,0.0,546
19459047,60081,60081,0.0,0.0,0.0,556
19988583,52953,52953,0.0,0.0,0.0,566
20000000,1141,1141,0.0,0.0,0.0,567
Averages from the middle 80% of values:
interval_op_rate : 34003
interval_key_rate : 34003
latency median : 0.0
latency 95th percentile : 0.0
latency 99.9th percentile : 0.0
Total operation time : 00:09:27
END
{noformat}
h2. Patched version
{noformat}
total,interval_op_rate,interval_key_rate,latency,95th,99.9th,elapsed_time
398380,39838,39838,0.0,0.0,0.0,10
1090332,69195,69195,0.0,0.0,0.0,20
1756859,66652,66652,0.0,0.0,0.0,30
2408330,65147,65147,0.0,0.0,0.0,40
3021314,61298,61298,0.0,0.0,0.0,50
3602221,58090,58090,0.0,0.0,0.0,60
3602221,0,0,0.0,0.0,0.0,70
4086404,48418,48418,0.0,0.0,0.0,80
4670997,58459,58459,0.0,0.0,0.0,91
5328657,65766,65766,0.0,0.0,0.0,101
5950535,62187,62187,0.0,0.0,0.0,111
6544475,59394,59394,0.0,0.0,0.0,121
7163644,61916,61916,0.0,0.0,0.0,131
7307634,14399,14399,0.0,0.0,0.0,141
7331684,2405,2405,0.0,0.0,0.0,151
7989707,65802,65802,0.0,0.0,0.0,161
8653302,66359,66359,0.0,0.0,0.0,172
9273188,61988,61988,0.0,0.0,0.0,182
9935986,66279,66279,0.0,0.0,0.0,192
10489010,55302,55302,0.0,0.0,0.0,202
10909996,42098,42098,0.0,0.0,0.0,212
10962871,5287,5287,0.0,0.0,0.0,222
11274293,31142,31142,0.0,0.0,0.0,232
11274293,0,0,0.0,0.0,0.0,242
11274293,0,0,0.0,0.0,0.0,252
11297105,2281,2281,0.0,0.0,0.0,263
11946842,64973,64973,0.0,0.0,0.0,273
12509283,56244,56244,0.0,0.0,0.0,283
13205933,69665,69665,0.0,0.0,0.0,293
13809534,60360,60360,0.0,0.0,0.0,303
14334735,52520,52520,0.0,0.0,0.0,313
14615255,28052,28052,0.0,0.0,0.0,323
14615958,70,70,0.0,0.0,0.0,333
14841997,22603,22603,0.0,0.0,0.0,343
14841997,0,0,0.0,0.0,0.0,354
14841997,0,0,0.0,0.0,0.0,364
15262968,42097,42097,0.0,0.0,0.0,374
15943731,68076,68076,0.0,0.0,0.0,384
16619205,67547,67547,0.0,0.0,0.0,394
17197417,57821,57821,0.0,0.0,0.0,404
17776353,57893,57893,0.0,0.0,0.0,414
18235461,45910,45910,0.0,0.0,0.0,424
18267460,3199,3199,0.0,0.0,0.0,434
18592152,32469,32469,0.0,0.0,0.0,445
18732480,14032,14032,0.0,0.0,0.0,455
19328150,59567,59567,0.0,0.0,0.0,465
19930114,60196,60196,0.0,0.0,0.0,475
20000000,6988,6988,0.0,0.0,0.0,479
Averages from the middle 80% of values:
interval_op_rate : 42156
interval_key_rate : 42156
latency median : 0.0
latency 95th percentile : 0.0
latency 99.9th percentile : 0.0
Total operation time : 00:07:59
END
{noformat}
Note that the trunk run was one of the good runs, many runs it just ended up in a GC loop not making any progress, that never happened on the patched version.
The code is a PoC, and will need more work and another (atleast) pair of eyes.
Maybe [~enigmacurry] can run a proper benchmark? Poke me on irc if you need help to get the adler32 stuff running.
> Experiment with direct buffer in SequentialWriter
> -------------------------------------------------
>
> Key: CASSANDRA-4338
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4338
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Reporter: Jonathan Ellis
> Assignee: Marcus Eriksson
> Priority: Minor
> Labels: performance
> Fix For: 2.1
>
> Attachments: 4338-gc.tar.gz, gc-4338-patched.png, gc-trunk-me.png, gc-trunk.png, gc-with-patch-me.png
>
>
> Using a direct buffer instead of a heap-based byte[] should let us avoid a copy into native memory when we flush the buffer.
--
This message was sent by Atlassian JIRA
(v6.1#6144)