You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by GitBox <gi...@apache.org> on 2019/06/29 17:46:24 UTC
[GitHub] [incubator-pinot] Jackie-Jiang opened a new pull request #4385:
Implement OffHeapMutableBytesStore for real-time var-length bytes store
Jackie-Jiang opened a new pull request #4385: Implement OffHeapMutableBytesStore for real-time var-length bytes store
URL: https://github.com/apache/incubator-pinot/pull/4385
Comparing with the MutableOffHeapByteArrayStore:
1. Read/Write speed improvement
2. Buffer size is fixed and buffers are allocated on demand, so memory is always used effeciently
3. No need to estimate the buffer size beforehead, and performance is guaranteed (for MutableOffHeapByteArrayStore, if buffer size is under-estimated, pay huge penalty on allocating a huge buffer at runtime, and also read/write speed degration
Added benchmark to compare the read/write speed:
Benchmark (_maxValueLength) Mode Cnt Score Error Units
BenchmarkOffHeapMutableBytesStore.mutableOffHeapByteArrayStoreRead 8 avgt 5 29.875 ± 0.370 ms/op
BenchmarkOffHeapMutableBytesStore.mutableOffHeapByteArrayStoreRead 16 avgt 5 32.344 ± 0.765 ms/op
BenchmarkOffHeapMutableBytesStore.mutableOffHeapByteArrayStoreRead 32 avgt 5 35.027 ± 0.460 ms/op
BenchmarkOffHeapMutableBytesStore.mutableOffHeapByteArrayStoreRead 64 avgt 5 40.811 ± 0.414 ms/op
BenchmarkOffHeapMutableBytesStore.mutableOffHeapByteArrayStoreRead 128 avgt 5 51.767 ± 0.185 ms/op
BenchmarkOffHeapMutableBytesStore.mutableOffHeapByteArrayStoreRead 256 avgt 5 77.065 ± 1.627 ms/op
BenchmarkOffHeapMutableBytesStore.mutableOffHeapByteArrayStoreRead 512 avgt 5 127.167 ± 1.075 ms/op
BenchmarkOffHeapMutableBytesStore.mutableOffHeapByteArrayStoreRead 1024 avgt 5 283.145 ± 2.829 ms/op
BenchmarkOffHeapMutableBytesStore.mutableOffHeapByteArrayStoreWrite 8 avgt 5 16.755 ± 0.374 ms/op
BenchmarkOffHeapMutableBytesStore.mutableOffHeapByteArrayStoreWrite 16 avgt 5 25.579 ± 0.293 ms/op
BenchmarkOffHeapMutableBytesStore.mutableOffHeapByteArrayStoreWrite 32 avgt 5 23.953 ± 0.507 ms/op
BenchmarkOffHeapMutableBytesStore.mutableOffHeapByteArrayStoreWrite 64 avgt 5 51.442 ± 1.230 ms/op
BenchmarkOffHeapMutableBytesStore.mutableOffHeapByteArrayStoreWrite 128 avgt 5 64.334 ± 0.069 ms/op
BenchmarkOffHeapMutableBytesStore.mutableOffHeapByteArrayStoreWrite 256 avgt 5 260.415 ± 1.823 ms/op
BenchmarkOffHeapMutableBytesStore.mutableOffHeapByteArrayStoreWrite 512 avgt 5 615.528 ± 3.180 ms/op
BenchmarkOffHeapMutableBytesStore.mutableOffHeapByteArrayStoreWrite 1024 avgt 5 1168.779 ± 16.961 ms/op
BenchmarkOffHeapMutableBytesStore.offHeapMutableBytesStoreRead 8 avgt 5 28.978 ± 0.535 ms/op
BenchmarkOffHeapMutableBytesStore.offHeapMutableBytesStoreRead 16 avgt 5 33.420 ± 0.400 ms/op
BenchmarkOffHeapMutableBytesStore.offHeapMutableBytesStoreRead 32 avgt 5 37.120 ± 0.272 ms/op
BenchmarkOffHeapMutableBytesStore.offHeapMutableBytesStoreRead 64 avgt 5 39.552 ± 0.317 ms/op
BenchmarkOffHeapMutableBytesStore.offHeapMutableBytesStoreRead 128 avgt 5 42.861 ± 0.588 ms/op
BenchmarkOffHeapMutableBytesStore.offHeapMutableBytesStoreRead 256 avgt 5 47.025 ± 0.189 ms/op
BenchmarkOffHeapMutableBytesStore.offHeapMutableBytesStoreRead 512 avgt 5 60.740 ± 0.677 ms/op
BenchmarkOffHeapMutableBytesStore.offHeapMutableBytesStoreRead 1024 avgt 5 89.509 ± 1.838 ms/op
BenchmarkOffHeapMutableBytesStore.offHeapMutableBytesStoreWrite 8 avgt 5 21.327 ± 0.509 ms/op
BenchmarkOffHeapMutableBytesStore.offHeapMutableBytesStoreWrite 16 avgt 5 28.494 ± 0.304 ms/op
BenchmarkOffHeapMutableBytesStore.offHeapMutableBytesStoreWrite 32 avgt 5 33.399 ± 0.533 ms/op
BenchmarkOffHeapMutableBytesStore.offHeapMutableBytesStoreWrite 64 avgt 5 50.755 ± 1.700 ms/op
BenchmarkOffHeapMutableBytesStore.offHeapMutableBytesStoreWrite 128 avgt 5 75.702 ± 1.079 ms/op
BenchmarkOffHeapMutableBytesStore.offHeapMutableBytesStoreWrite 256 avgt 5 129.677 ± 1.942 ms/op
BenchmarkOffHeapMutableBytesStore.offHeapMutableBytesStoreWrite 512 avgt 5 222.697 ± 2.109 ms/op
BenchmarkOffHeapMutableBytesStore.offHeapMutableBytesStoreWrite 1024 avgt 5 392.053 ± 3.200 ms/op
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org