You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Vijay (JIRA)" <ji...@apache.org> on 2014/07/22 07:50:39 UTC
[jira] [Comment Edited] (CASSANDRA-7438) Serializing Row cache
alternative (Fully off heap)
[ https://issues.apache.org/jira/browse/CASSANDRA-7438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14069862#comment-14069862 ]
Vijay edited comment on CASSANDRA-7438 at 7/22/14 5:49 AM:
-----------------------------------------------------------
Attached patch makes the off heap/Serialization Cache configurable. (the default is still SerializationCache).
Regarding performance, the performance of the new cache is obviously better when the JNI overhead is less than the GC overhead, but for the smaller sized caches which can fit in memory the performance is little lower which is understandable (but both of them out perform pagecache performance by a large number). Here are the numbers.
*OffheapCacheProvider*
{panel}
Running READ with 1200 threads for 10000000 iterations
ops , op/s, key/s, mean, med, .95, .99, .999, max, time, stderr
2030355 , 2029531, 2029531, 3.1, 3.1, 5.4, 5.7, 61.8, 3014.5, 1.0, 0.00000
2395480 , 202845, 202845, 5.8, 5.4, 5.8, 20.2, 522.4, 545.9, 2.8, 0.00000
2638600 , 221368, 221368, 5.4, 5.3, 5.8, 16.3, 78.8, 131.5, 3.9, 0.57860
2891705 , 221976, 221976, 5.4, 5.3, 5.6, 6.2, 15.2, 19.2, 5.0, 0.60478
3147747 , 222527, 222527, 5.4, 5.3, 5.6, 6.1, 15.4, 18.2, 6.2, 0.58659
3394999 , 221527, 221527, 5.4, 5.3, 5.6, 6.6, 15.9, 19.4, 7.3, 0.55884
3663559 , 226114, 226114, 5.3, 5.2, 5.6, 15.0, 84.4, 110.7, 8.5, 0.52924
3911154 , 223831, 223831, 5.4, 5.3, 5.6, 6.1, 15.6, 20.0, 9.6, 0.50018
4152946 , 223246, 223246, 5.4, 5.3, 5.6, 6.1, 15.7, 18.8, 10.7, 0.47323
4403162 , 228532, 228532, 5.2, 5.2, 5.6, 23.2, 107.4, 121.4, 11.8, 0.44856
4641021 , 225196, 225196, 5.3, 5.2, 5.6, 5.9, 15.3, 18.4, 12.8, 0.42557
4889523 , 222826, 222826, 5.4, 5.3, 5.6, 6.3, 16.2, 22.0, 13.9, 0.40476
5124891 , 223203, 223203, 5.4, 5.3, 5.6, 5.8, 6.2, 14.8, 15.0, 0.38602
5375262 , 221222, 221222, 5.4, 5.2, 5.6, 18.4, 94.2, 115.1, 16.1, 0.36899
5616470 , 224022, 224022, 5.4, 5.3, 5.6, 5.9, 14.3, 17.8, 17.2, 0.35349
5866825 , 223000, 223000, 5.4, 5.3, 5.6, 6.1, 15.5, 18.2, 18.3, 0.33882
6125601 , 225757, 225757, 5.2, 5.3, 5.6, 9.6, 49.4, 72.0, 19.5, 0.32535
6348030 , 192703, 192703, 6.3, 5.3, 9.3, 14.4, 77.1, 91.5, 20.6, 0.31282
6483574 , 128520, 128520, 9.3, 8.4, 10.9, 19.5, 88.7, 99.0, 21.7, 0.30329
6626176 , 137199, 137199, 8.7, 8.4, 10.6, 14.0, 32.7, 40.9, 22.7, 0.29771
6768401 , 136860, 136860, 8.8, 8.4, 10.3, 14.1, 35.1, 40.8, 23.8, 0.29213
6911785 , 138204, 138204, 8.7, 8.3, 10.2, 13.7, 34.1, 37.8, 24.8, 0.28669
7055951 , 138633, 138633, 8.7, 8.3, 10.5, 32.0, 40.5, 46.9, 25.8, 0.28130
7199084 , 137731, 137731, 8.7, 8.4, 10.2, 14.0, 33.4, 40.9, 26.9, 0.27623
7338032 , 133201, 133201, 9.0, 8.4, 10.9, 34.0, 39.4, 43.8, 27.9, 0.27116
7480439 , 137059, 137059, 8.8, 8.4, 10.2, 13.9, 35.9, 39.5, 29.0, 0.26663
7647810 , 161209, 161209, 7.5, 7.8, 9.6, 13.4, 33.9, 77.9, 30.0, 0.26185
7898882 , 226498, 226498, 5.3, 5.2, 5.6, 19.7, 108.5, 119.3, 31.1, 0.25629
8136305 , 223840, 223840, 5.4, 5.3, 5.6, 5.9, 17.3, 23.2, 32.2, 0.24838
8372076 , 223790, 223790, 5.4, 5.3, 5.6, 6.0, 15.2, 20.0, 33.2, 0.24095
8633758 , 232914, 232914, 5.1, 5.2, 5.6, 17.5, 138.4, 182.0, 34.4, 0.23397
8869214 , 222243, 222243, 5.4, 5.3, 5.6, 6.0, 15.2, 17.9, 35.4, 0.22717
9121652 , 223037, 223037, 5.4, 5.3, 5.6, 5.9, 15.4, 18.8, 36.5, 0.22105
9360286 , 225070, 225070, 5.3, 5.3, 5.6, 14.8, 82.7, 92.1, 37.6, 0.21524
9609676 , 224089, 224089, 5.4, 5.3, 5.6, 5.8, 6.2, 14.3, 38.7, 0.20967
9848551 , 222123, 222123, 5.4, 5.3, 5.6, 5.9, 24.2, 27.2, 39.8, 0.20440
10000000 , 229511, 229511, 5.0, 5.2, 5.8, 60.0, 74.3, 132.0, 40.5, 0.19935
Results:
real op rate : 247211
adjusted op rate stderr : 0
key rate : 247211
latency mean : 5.4
latency median : 3.5
latency 95th percentile : 5.5
latency 99th percentile : 6.1
latency 99.9th percentile : 83.4
latency max : 3014.5
Total operation time : 00:00:40
END
{panel}
*SerializingCacheProvider*
{panel}
Running READ with 1200 threads for 10000000 iterations
ops , op/s, key/s, mean, med, .95, .99, .999, max, time, stderr
2517463 , 2496430, 2496430, 1.5, 0.2, 4.1, 11.5, 94.7, 6217.0, 1.0, 0.00000
6690312 , 266503, 266503, 4.7, 4.6, 5.2, 15.6, 408.0, 21106.5, 16.7, 0.00000
6974745 , 233722, 233722, 5.1, 5.0, 5.3, 6.5, 15.5, 19.0, 17.9, 0.57070
7275403 , 277156, 277156, 3.9, 4.0, 5.4, 27.6, 62.8, 659.4, 19.0, 0.61144
7770460 , 271264, 271264, 4.7, 5.0, 5.3, 20.2, 426.0, 1037.5, 20.8, 0.59139
8030818 , 233497, 233497, 5.1, 5.0, 5.3, 6.4, 16.5, 20.9, 21.9, 0.56335
8299037 , 232178, 232178, 5.2, 5.1, 5.3, 5.8, 15.8, 21.9, 23.1, 0.54048
8556018 , 231981, 231981, 5.2, 5.1, 5.3, 5.9, 16.7, 20.3, 24.2, 0.51744
8811639 , 234401, 234401, 5.1, 5.0, 5.3, 5.8, 16.2, 19.2, 25.3, 0.49459
9245382 , 262824, 262824, 4.6, 4.9, 5.3, 20.5, 290.9, 546.4, 26.9, 0.47255
9546906 , 235163, 235163, 5.1, 5.0, 5.3, 5.8, 16.1, 21.5, 28.2, 0.44887
9819817 , 235254, 235254, 5.1, 5.0, 5.3, 5.7, 15.5, 19.7, 29.4, 0.43014
10000000 , 233793, 233793, 5.0, 5.0, 5.3, 15.4, 44.0, 51.2, 30.1, 0.41249
Results:
real op rate : 331952
adjusted op rate stderr : 0
key rate : 331952
latency mean : 4.0
latency median : 0.7
latency 95th percentile : 5.2
latency 99th percentile : 14.9
latency 99.9th percentile : 294.2
latency max : 21106.5
Total operation time : 00:00:30
END
{panel}
I think we can commit this changes, all known optimizations are done, Thanks!
was (Author: vijay2win@yahoo.com):
Attached patch makes the off heap/Serialization Cache configurable. (the default is still SerializationCache).
Regarding performance, the performance of the new cache is obviously better when the JNI overhead is less than the GC overhead, but for the smaller sized caches which can fit in memory the performance is little lower which is understandable (but both of them out perform pagecache performance by a large number). Here are the numbers.
OffheapCacheProvider
====================
Running READ with 1200 threads for 10000000 iterations
ops , op/s, key/s, mean, med, .95, .99, .999, max, time, stderr
2030355 , 2029531, 2029531, 3.1, 3.1, 5.4, 5.7, 61.8, 3014.5, 1.0, 0.00000
2395480 , 202845, 202845, 5.8, 5.4, 5.8, 20.2, 522.4, 545.9, 2.8, 0.00000
2638600 , 221368, 221368, 5.4, 5.3, 5.8, 16.3, 78.8, 131.5, 3.9, 0.57860
2891705 , 221976, 221976, 5.4, 5.3, 5.6, 6.2, 15.2, 19.2, 5.0, 0.60478
3147747 , 222527, 222527, 5.4, 5.3, 5.6, 6.1, 15.4, 18.2, 6.2, 0.58659
3394999 , 221527, 221527, 5.4, 5.3, 5.6, 6.6, 15.9, 19.4, 7.3, 0.55884
3663559 , 226114, 226114, 5.3, 5.2, 5.6, 15.0, 84.4, 110.7, 8.5, 0.52924
3911154 , 223831, 223831, 5.4, 5.3, 5.6, 6.1, 15.6, 20.0, 9.6, 0.50018
4152946 , 223246, 223246, 5.4, 5.3, 5.6, 6.1, 15.7, 18.8, 10.7, 0.47323
4403162 , 228532, 228532, 5.2, 5.2, 5.6, 23.2, 107.4, 121.4, 11.8, 0.44856
4641021 , 225196, 225196, 5.3, 5.2, 5.6, 5.9, 15.3, 18.4, 12.8, 0.42557
4889523 , 222826, 222826, 5.4, 5.3, 5.6, 6.3, 16.2, 22.0, 13.9, 0.40476
5124891 , 223203, 223203, 5.4, 5.3, 5.6, 5.8, 6.2, 14.8, 15.0, 0.38602
5375262 , 221222, 221222, 5.4, 5.2, 5.6, 18.4, 94.2, 115.1, 16.1, 0.36899
5616470 , 224022, 224022, 5.4, 5.3, 5.6, 5.9, 14.3, 17.8, 17.2, 0.35349
5866825 , 223000, 223000, 5.4, 5.3, 5.6, 6.1, 15.5, 18.2, 18.3, 0.33882
6125601 , 225757, 225757, 5.2, 5.3, 5.6, 9.6, 49.4, 72.0, 19.5, 0.32535
6348030 , 192703, 192703, 6.3, 5.3, 9.3, 14.4, 77.1, 91.5, 20.6, 0.31282
6483574 , 128520, 128520, 9.3, 8.4, 10.9, 19.5, 88.7, 99.0, 21.7, 0.30329
6626176 , 137199, 137199, 8.7, 8.4, 10.6, 14.0, 32.7, 40.9, 22.7, 0.29771
6768401 , 136860, 136860, 8.8, 8.4, 10.3, 14.1, 35.1, 40.8, 23.8, 0.29213
6911785 , 138204, 138204, 8.7, 8.3, 10.2, 13.7, 34.1, 37.8, 24.8, 0.28669
7055951 , 138633, 138633, 8.7, 8.3, 10.5, 32.0, 40.5, 46.9, 25.8, 0.28130
7199084 , 137731, 137731, 8.7, 8.4, 10.2, 14.0, 33.4, 40.9, 26.9, 0.27623
7338032 , 133201, 133201, 9.0, 8.4, 10.9, 34.0, 39.4, 43.8, 27.9, 0.27116
7480439 , 137059, 137059, 8.8, 8.4, 10.2, 13.9, 35.9, 39.5, 29.0, 0.26663
7647810 , 161209, 161209, 7.5, 7.8, 9.6, 13.4, 33.9, 77.9, 30.0, 0.26185
7898882 , 226498, 226498, 5.3, 5.2, 5.6, 19.7, 108.5, 119.3, 31.1, 0.25629
8136305 , 223840, 223840, 5.4, 5.3, 5.6, 5.9, 17.3, 23.2, 32.2, 0.24838
8372076 , 223790, 223790, 5.4, 5.3, 5.6, 6.0, 15.2, 20.0, 33.2, 0.24095
8633758 , 232914, 232914, 5.1, 5.2, 5.6, 17.5, 138.4, 182.0, 34.4, 0.23397
8869214 , 222243, 222243, 5.4, 5.3, 5.6, 6.0, 15.2, 17.9, 35.4, 0.22717
9121652 , 223037, 223037, 5.4, 5.3, 5.6, 5.9, 15.4, 18.8, 36.5, 0.22105
9360286 , 225070, 225070, 5.3, 5.3, 5.6, 14.8, 82.7, 92.1, 37.6, 0.21524
9609676 , 224089, 224089, 5.4, 5.3, 5.6, 5.8, 6.2, 14.3, 38.7, 0.20967
9848551 , 222123, 222123, 5.4, 5.3, 5.6, 5.9, 24.2, 27.2, 39.8, 0.20440
10000000 , 229511, 229511, 5.0, 5.2, 5.8, 60.0, 74.3, 132.0, 40.5, 0.19935
Results:
real op rate : 247211
adjusted op rate stderr : 0
key rate : 247211
latency mean : 5.4
latency median : 3.5
latency 95th percentile : 5.5
latency 99th percentile : 6.1
latency 99.9th percentile : 83.4
latency max : 3014.5
Total operation time : 00:00:40
END
SerializingCacheProvider
========================
Running READ with 1200 threads for 10000000 iterations
ops , op/s, key/s, mean, med, .95, .99, .999, max, time, stderr
2517463 , 2496430, 2496430, 1.5, 0.2, 4.1, 11.5, 94.7, 6217.0, 1.0, 0.00000
6690312 , 266503, 266503, 4.7, 4.6, 5.2, 15.6, 408.0, 21106.5, 16.7, 0.00000
6974745 , 233722, 233722, 5.1, 5.0, 5.3, 6.5, 15.5, 19.0, 17.9, 0.57070
7275403 , 277156, 277156, 3.9, 4.0, 5.4, 27.6, 62.8, 659.4, 19.0, 0.61144
7770460 , 271264, 271264, 4.7, 5.0, 5.3, 20.2, 426.0, 1037.5, 20.8, 0.59139
8030818 , 233497, 233497, 5.1, 5.0, 5.3, 6.4, 16.5, 20.9, 21.9, 0.56335
8299037 , 232178, 232178, 5.2, 5.1, 5.3, 5.8, 15.8, 21.9, 23.1, 0.54048
8556018 , 231981, 231981, 5.2, 5.1, 5.3, 5.9, 16.7, 20.3, 24.2, 0.51744
8811639 , 234401, 234401, 5.1, 5.0, 5.3, 5.8, 16.2, 19.2, 25.3, 0.49459
9245382 , 262824, 262824, 4.6, 4.9, 5.3, 20.5, 290.9, 546.4, 26.9, 0.47255
9546906 , 235163, 235163, 5.1, 5.0, 5.3, 5.8, 16.1, 21.5, 28.2, 0.44887
9819817 , 235254, 235254, 5.1, 5.0, 5.3, 5.7, 15.5, 19.7, 29.4, 0.43014
10000000 , 233793, 233793, 5.0, 5.0, 5.3, 15.4, 44.0, 51.2, 30.1, 0.41249
Results:
real op rate : 331952
adjusted op rate stderr : 0
key rate : 331952
latency mean : 4.0
latency median : 0.7
latency 95th percentile : 5.2
latency 99th percentile : 14.9
latency 99.9th percentile : 294.2
latency max : 21106.5
Total operation time : 00:00:30
END
I think we can commit this changes, all known optimizations are done, Thanks!
> Serializing Row cache alternative (Fully off heap)
> --------------------------------------------------
>
> Key: CASSANDRA-7438
> URL: https://issues.apache.org/jira/browse/CASSANDRA-7438
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Environment: Linux
> Reporter: Vijay
> Assignee: Vijay
> Labels: performance
> Fix For: 3.0
>
> Attachments: 0001-CASSANDRA-7438.patch
>
>
> Currently SerializingCache is partially off heap, keys are still stored in JVM heap as BB,
> * There is a higher GC costs for a reasonably big cache.
> * Some users have used the row cache efficiently in production for better results, but this requires careful tunning.
> * Overhead in Memory for the cache entries are relatively high.
> So the proposal for this ticket is to move the LRU cache logic completely off heap and use JNI to interact with cache. We might want to ensure that the new implementation match the existing API's (ICache), and the implementation needs to have safe memory access, low overhead in memory and less memcpy's (As much as possible).
> We might also want to make this cache configurable.
--
This message was sent by Atlassian JIRA
(v6.2#6252)