You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@kudu.apache.org by "Alexey Serbin (Code Review)" <ge...@cloudera.org> on 2020/06/16 00:06:26 UTC
[kudu-CR] [master] cache for table locations
Hello Kudu Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/15971
to look at the new patch set (#2).
Change subject: [master] cache for table locations
......................................................................
[master] cache for table locations
This patch introduces a cache for table locations in catalog manager.
When running with 48 concurrent client threads, the performance of
CatalogManager::GetTableLocations() method improved about 130%
when the cache is enabled.
A smaller 12% improvement is observed for GetTableLocations RPC
when run with the same number of concurrent client threads and cache
enabled. The test results are below.
I'm planning to add these test scenarios into kudu/scripts/benchmarks.sh
in a follow up patch.
========================================================================
After this patch with 128MByte cache enabled:
table_locations-itest \
--gtest_filter=TableLocationsTest.GetTableLocationsBenchmarkFunctionCall \
--benchmark_num_threads=48 \
--table_locations_cache_capacity_mb=128
GetTableLocations function call: 588056.2 req/sec
Before this patch:
table_locations-itest \
--gtest_filter=TableLocationsTest.GetTableLocationsBenchmarkFunctionCall \
--benchmark_num_threads=48
GetTableLocations function call: 252443.4 req/sec
========================================================================
After this patch with 128MByte cache enabled:
table_locations-itest \
--gtest_filter=TableLocationsTest.GetTableLocationsBenchmark \
--rpc_num_service_threads=24 \
--benchmark_num_threads=48 \
--table_locations_cache_capacity_mb=128
GetTableLocations RPC: 39328.6 req/sec
Stats on GetTableLocations RPC (times in microseconds):
Count: 196643
Mean: 149.526
Percentiles:
0% (min) = 40
25% = 120
50% (med) = 142
75% = 172
95% = 216
99% = 308
99.9% = 652
99.99% = 1424
100% (max) = 8480
Before this patch:
table_locations-itest \
--gtest_filter=TableLocationsTest.GetTableLocationsBenchmark \
--rpc_num_service_threads=24 \
--benchmark_num_threads=48
GetTableLocations RPC: 34833.2 req/sec
Stats on GetTableLocations RPC (times in microseconds):
Count: 174166
Mean: 241.475
Percentiles:
0% (min) = 61
25% = 189
50% (med) = 225
75% = 278
95% = 374
99% = 532
99.9% = 956
99.99% = 1848
100% (max) = 5834
Change-Id: I7d2a4771ddc455d92a1da00db91c555a21151a23
---
M src/kudu/integration-tests/table_locations-itest.cc
M src/kudu/master/CMakeLists.txt
M src/kudu/master/catalog_manager.cc
M src/kudu/master/catalog_manager.h
M src/kudu/master/master_service.cc
A src/kudu/master/table_locations_cache.cc
A src/kudu/master/table_locations_cache.h
A src/kudu/master/table_locations_cache_metrics.cc
A src/kudu/master/table_locations_cache_metrics.h
9 files changed, 529 insertions(+), 31 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/71/15971/2
--
To view, visit http://gerrit.cloudera.org:8080/15971
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I7d2a4771ddc455d92a1da00db91c555a21151a23
Gerrit-Change-Number: 15971
Gerrit-PatchSet: 2
Gerrit-Owner: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)