You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@kudu.apache.org by "Alexey Serbin (Code Review)" <ge...@cloudera.org> on 2026/05/14 17:26:30 UTC
[kudu-CR] WIP KUDU-1865 reduce cross-thread allocations in RPC hot paths
Alexey Serbin has uploaded this change for review. ( http://gerrit.cloudera.org:8080/24305
Change subject: WIP KUDU-1865 reduce cross-thread allocations in RPC hot paths
......................................................................
WIP KUDU-1865 reduce cross-thread allocations in RPC hot paths
WIP:
* add more details in the description
* add information on test performance before/after the patch
This patch was originally posted by Todd [1]. I'm re-posting it
with updates: that seems to be a simplest path forward, otherwise
there are too many conflicts if trying to revv the original patch.
[1] https://gerrit.cloudera.org/#/c/5905/
Change-Id: I86a2acbd1d8cb724728034c4e91907c99cbfe32e
---
M src/kudu/rpc/connection.cc
M src/kudu/rpc/connection.h
M src/kudu/rpc/messenger.cc
M src/kudu/rpc/messenger.h
M src/kudu/rpc/negotiation.cc
M src/kudu/rpc/reactor.cc
M src/kudu/rpc/reactor.h
7 files changed, 189 insertions(+), 269 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/05/24305/1
--
To view, visit http://gerrit.cloudera.org:8080/24305
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I86a2acbd1d8cb724728034c4e91907c99cbfe32e
Gerrit-Change-Number: 24305
Gerrit-PatchSet: 1
Gerrit-Owner: Alexey Serbin <al...@apache.org>
[kudu-CR] KUDU-1865 reduce cross-thread allocations in RPC (part 1)
Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Alexey Serbin has posted comments on this change. ( http://gerrit.cloudera.org:8080/24305 )
Change subject: KUDU-1865 reduce cross-thread allocations in RPC (part 1)
......................................................................
Patch Set 5:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/24305/5//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/24305/5//COMMIT_MSG@69
PS5, Line 69: before: 499 488 491
> What are the 3 columns of numbers?
I added these to represent the variability I could see running the test scenario. Of course, I ran many more than 3 iterations of the test, but these provide a good idea what it looks like. The pct numbers are based on the average between these three columns, 'before' vs 'after.
--
To view, visit http://gerrit.cloudera.org:8080/24305
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I86a2acbd1d8cb724728034c4e91907c99cbfe32e
Gerrit-Change-Number: 24305
Gerrit-PatchSet: 5
Gerrit-Owner: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Anonymous Coward (763)
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: David Rorke <dr...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Zoltan Martonka <zm...@cloudera.com>
Gerrit-Comment-Date: Mon, 18 May 2026 20:38:19 +0000
Gerrit-HasComments: Yes
[kudu-CR] WIP KUDU-1865 reduce cross-thread allocations in RPC hot paths
Posted by "Kudu Jenkins (Code Review)" <ge...@cloudera.org>.
Kudu Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24305 )
Change subject: WIP KUDU-1865 reduce cross-thread allocations in RPC hot paths
......................................................................
Patch Set 1:
Build Started http://jenkins.kudu.apache.org/job/pre_commit/2568/
--
To view, visit http://gerrit.cloudera.org:8080/24305
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I86a2acbd1d8cb724728034c4e91907c99cbfe32e
Gerrit-Change-Number: 24305
Gerrit-PatchSet: 1
Gerrit-Owner: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Thu, 14 May 2026 17:26:39 +0000
Gerrit-HasComments: No
[kudu-CR] KUDU-1865 reduce cross-thread allocations in RPC (part 1)
Posted by "Anonymous Coward (Code Review)" <ge...@cloudera.org>.
Anonymous Coward (763) has posted comments on this change. ( http://gerrit.cloudera.org:8080/24305 )
Change subject: KUDU-1865 reduce cross-thread allocations in RPC (part 1)
......................................................................
Patch Set 4:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/24305/4/src/kudu/rpc/connection.cc
File src/kudu/rpc/connection.cc:
http://gerrit.cloudera.org:8080/#/c/24305/4/src/kudu/rpc/connection.cc@801
PS4, Line 801: // The lambdas below need to hold a reference to the connection in case
: // it has been closed before the task gets scheduled/aborted.
: scoped_refptr<Connection> reffed_this(this);
: auto* rpc_error_raw = rpc_error.release();
: ReactorTask task{
: [=](ReactorThread* rt) {
: rt->CompleteConnectionNegotiation(reffed_this.get(),
: negotiation_status,
: unique_ptr<ErrorStatusPB>(rpc_error_raw));
: },
: [=](const Status& /*s*/) {
: DCHECK(reffed_this->reactor_thread()->reactor()->closing());
: },
: };
: reactor_thread_->reactor()->ScheduleReactorTask(std::move(task));
You should create an unique_ptr, then capture it in the lambda by move().
With the current way the rpc_error_raw will be leaked if the task is cancelled.
--
To view, visit http://gerrit.cloudera.org:8080/24305
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I86a2acbd1d8cb724728034c4e91907c99cbfe32e
Gerrit-Change-Number: 24305
Gerrit-PatchSet: 4
Gerrit-Owner: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward (763)
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: David Rorke <dr...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Zoltan Martonka <zm...@cloudera.com>
Gerrit-Comment-Date: Fri, 15 May 2026 12:52:55 +0000
Gerrit-HasComments: Yes
[kudu-CR] WIP KUDU-1865 reduce cross-thread allocations in RPC hot paths
Posted by "Kudu Jenkins (Code Review)" <ge...@cloudera.org>.
Kudu Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24305 )
Change subject: WIP KUDU-1865 reduce cross-thread allocations in RPC hot paths
......................................................................
Patch Set 1: Verified-1
Build Failed
http://jenkins.kudu.apache.org/job/pre_commit/2568/ : FAILURE
--
To view, visit http://gerrit.cloudera.org:8080/24305
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I86a2acbd1d8cb724728034c4e91907c99cbfe32e
Gerrit-Change-Number: 24305
Gerrit-PatchSet: 1
Gerrit-Owner: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Thu, 14 May 2026 18:25:31 +0000
Gerrit-HasComments: No
[kudu-CR] KUDU-1865 reduce cross-thread allocations in RPC (part 1)
Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Alexey Serbin has posted comments on this change. ( http://gerrit.cloudera.org:8080/24305 )
Change subject: KUDU-1865 reduce cross-thread allocations in RPC (part 1)
......................................................................
Patch Set 5: Verified+1
DEBUG: unrelated test failure due to KUDU-3216
--
To view, visit http://gerrit.cloudera.org:8080/24305
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I86a2acbd1d8cb724728034c4e91907c99cbfe32e
Gerrit-Change-Number: 24305
Gerrit-PatchSet: 5
Gerrit-Owner: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Anonymous Coward (763)
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: David Rorke <dr...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Zoltan Martonka <zm...@cloudera.com>
Gerrit-Comment-Date: Fri, 15 May 2026 19:16:08 +0000
Gerrit-HasComments: No
[kudu-CR] KUDU-1865 reduce cross-thread allocations in RPC (part 1)
Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/24305 )
Change subject: KUDU-1865 reduce cross-thread allocations in RPC (part 1)
......................................................................
Patch Set 5: Code-Review+1
--
To view, visit http://gerrit.cloudera.org:8080/24305
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I86a2acbd1d8cb724728034c4e91907c99cbfe32e
Gerrit-Change-Number: 24305
Gerrit-PatchSet: 5
Gerrit-Owner: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Anonymous Coward (763)
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: David Rorke <dr...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Zoltan Martonka <zm...@cloudera.com>
Gerrit-Comment-Date: Tue, 19 May 2026 23:39:27 +0000
Gerrit-HasComments: No
[kudu-CR] KUDU-1865 reduce cross-thread allocations in RPC (part 1)
Posted by "Zoltan Martonka (Code Review)" <ge...@cloudera.org>.
Zoltan Martonka has posted comments on this change. ( http://gerrit.cloudera.org:8080/24305 )
Change subject: KUDU-1865 reduce cross-thread allocations in RPC (part 1)
......................................................................
Patch Set 5: Code-Review+1
--
To view, visit http://gerrit.cloudera.org:8080/24305
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I86a2acbd1d8cb724728034c4e91907c99cbfe32e
Gerrit-Change-Number: 24305
Gerrit-PatchSet: 5
Gerrit-Owner: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Anonymous Coward (763)
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: David Rorke <dr...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Zoltan Martonka <zm...@cloudera.com>
Gerrit-Comment-Date: Wed, 20 May 2026 06:54:04 +0000
Gerrit-HasComments: No
[kudu-CR] KUDU-1865 reduce cross-thread allocations in RPC (part 1)
Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Alexey Serbin has posted comments on this change. ( http://gerrit.cloudera.org:8080/24305 )
Change subject: KUDU-1865 reduce cross-thread allocations in RPC (part 1)
......................................................................
Patch Set 4:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/24305/4/src/kudu/rpc/connection.cc
File src/kudu/rpc/connection.cc:
http://gerrit.cloudera.org:8080/#/c/24305/4/src/kudu/rpc/connection.cc@801
PS4, Line 801: // The lambdas below need to hold a reference to the connection in case
: // it has been closed before the task gets scheduled/aborted.
: scoped_refptr<Connection> reffed_this(this);
: auto* rpc_error_raw = rpc_error.release();
: ReactorTask task{
: [=](ReactorThread* rt) {
: rt->CompleteConnectionNegotiation(reffed_this.get(),
: negotiation_status,
: unique_ptr<ErrorStatusPB>(rpc_error_raw));
: },
: [=](const Status& /*s*/) {
: DCHECK(reffed_this->reactor_thread()->reactor()->closing());
: },
: };
: reactor_thread_->reactor()->ScheduleReactorTask(std::move(task));
> You should create an unique_ptr, then capture it in the lambda by move().
This regular approach isn't working here (yes, even with a mutable lambda). The problem stems from the requirement: a boost::function instance must always be copyable. Using a move capture in the lambda with a move-only type variable (like std::unique_ptr) makes an instance of boost::function non-copyable, resulting in a compiler error.
I addressed the possible memory leak by passing raw pointer and calling delete on it in the abort_func member. Thank you for pointing at that.
Maybe, I'm missing something and you have a recipe to make it work with move capture in C++17-based code? Please let me know.
--
To view, visit http://gerrit.cloudera.org:8080/24305
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I86a2acbd1d8cb724728034c4e91907c99cbfe32e
Gerrit-Change-Number: 24305
Gerrit-PatchSet: 4
Gerrit-Owner: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Anonymous Coward (763)
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: David Rorke <dr...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Zoltan Martonka <zm...@cloudera.com>
Gerrit-Comment-Date: Fri, 15 May 2026 15:00:20 +0000
Gerrit-HasComments: Yes
[kudu-CR] KUDU-1865 reduce cross-thread allocations in RPC (part 1)
Posted by "Marton Greber (Code Review)" <ge...@cloudera.org>.
Marton Greber has posted comments on this change. ( http://gerrit.cloudera.org:8080/24305 )
Change subject: KUDU-1865 reduce cross-thread allocations in RPC (part 1)
......................................................................
Patch Set 5: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/24305
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I86a2acbd1d8cb724728034c4e91907c99cbfe32e
Gerrit-Change-Number: 24305
Gerrit-PatchSet: 5
Gerrit-Owner: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Anonymous Coward (763)
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: David Rorke <dr...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Zoltan Martonka <zm...@cloudera.com>
Gerrit-Comment-Date: Wed, 20 May 2026 16:06:10 +0000
Gerrit-HasComments: No
[kudu-CR] KUDU-1865 reduce cross-thread allocations in RPC (part 1)
Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Alexey Serbin has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/24305 )
Change subject: KUDU-1865 reduce cross-thread allocations in RPC (part 1)
......................................................................
KUDU-1865 reduce cross-thread allocations in RPC (part 1)
This patch was originally posted by Todd [1]. I'm re-posting it
re-based and updated -- that's the best path forward; otherwise,
there are too many conflicts if trying to re-base and revv the
original patch in gerrit [1].
To verify that the number of cross-thread allocations is reduced with
this patch, I ran the target rpc-bench test scenario under customized
environment to trace tcmalloc allocations/deallocations, and then used
the Python script attached to KUDU-1865 ticket (updated a bit) to track
cross-thread allocations/deallocations in tcmalloc:
env TCMALLOC_TRACE=1 \
TCMALLOC_STACKTRACE_METHOD=generic_fp \
./bin/rpc-bench \
--server_reactors=1 \
--client_threads=1 \
--run_seconds=1 \
--gtest_filter='*BenchmarkCalls'
The results confirm that all the malloc/free asymmetry related to
ReactorTask objects is now gone. I'm planning to take care of
the rest of allocation/de-allocation asymmetry in hot RPC paths
in follow-up patches.
I ran the same test scenario with different parameters under a custom
environment to evaluate the total number of calls that lead to acquiring
locks guarding the per-size-classes' central free lists in tcmalloc.
I did so with the default and customized setting for the
TCMALLOC_MAX_TOTAL_THREAD_CACHE_BYTES environment variable. The
customization allowed to induce more activity on tcmalloc's central
free lists since the sizes of the free lists were shorter than with the
default setting.
The results of running rpc-bench under the perf utility on 8 CPU-core
machine are below. There is an improvement:
* ~1.37x times reduction in the total number of calls to the free lists
with the default setting of TCMALLOC_MAX_TOTAL_THREAD_CACHE_BYTES
* ~1.06x times reduction in the total number of calls to the free lists
with the custom setting of TCMALLOC_MAX_TOTAL_THREAD_CACHE_BYTES
As for requests-per-second performance, the improvement is minuscule.
The target test scenario doesn't create enough memory pressure and
concurrency to get to the point where too much of lock contention in
tcmalloc significantly affects RPC performance.
export TCMALLOC_MAX_TOTAL_THREAD_CACHE_BYTES=1048576
perf record -g -e 'syscalls:sys_enter_futex' \
./bin/rpc-bench \
--gtest_filter='*BenchmarkCalls' \
--server_reactors=2 \
--client_threads=4 \
--worker_threads=2 \
--run_seconds=60
perf script | \
grep -E 'tcmalloc::ThreadCache::(FetchFrom|ReleaseTo)CentralCache' | \
wc -l
TCMALLOC_MAX_TOTAL_THREAD_CACHE_BYTES unset (using default tcmalloc setting):
before: 499 488 491
after: 373 328 376
TCMALLOC_MAX_TOTAL_THREAD_CACHE_BYTES=1048576
before: 13794 12727 12215
after: 12695 12137 11857
[1] https://gerrit.cloudera.org/#/c/5905/
Change-Id: I86a2acbd1d8cb724728034c4e91907c99cbfe32e
Reviewed-on: http://gerrit.cloudera.org:8080/24305
Tested-by: Alexey Serbin <al...@apache.org>
Reviewed-by: Michael Smith <mi...@cloudera.com>
Reviewed-by: Zoltan Martonka <zm...@cloudera.com>
Reviewed-by: Marton Greber <gr...@gmail.com>
---
M src/kudu/rpc/connection.cc
M src/kudu/rpc/connection.h
M src/kudu/rpc/messenger.cc
M src/kudu/rpc/messenger.h
M src/kudu/rpc/negotiation.cc
M src/kudu/rpc/reactor.cc
M src/kudu/rpc/reactor.h
7 files changed, 190 insertions(+), 269 deletions(-)
Approvals:
Alexey Serbin: Verified
Michael Smith: Looks good to me, but someone else must approve
Zoltan Martonka: Looks good to me, but someone else must approve
Marton Greber: Looks good to me, approved
--
To view, visit http://gerrit.cloudera.org:8080/24305
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I86a2acbd1d8cb724728034c4e91907c99cbfe32e
Gerrit-Change-Number: 24305
Gerrit-PatchSet: 6
Gerrit-Owner: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Anonymous Coward (763)
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: David Rorke <dr...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Zoltan Martonka <zm...@cloudera.com>
[kudu-CR] KUDU-1865 reduce cross-thread allocations in RPC (part 1)
Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Hello Kurt Deschler, Zoltan Martonka, Abhishek Rawat, Kudu Jenkins, Abhishek Chennaka, David Rorke, Anonymous Coward (763), Joe McDonnell, Csaba Ringhofer, Michael Smith,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/24305
to look at the new patch set (#5).
Change subject: KUDU-1865 reduce cross-thread allocations in RPC (part 1)
......................................................................
KUDU-1865 reduce cross-thread allocations in RPC (part 1)
This patch was originally posted by Todd [1]. I'm re-posting it
re-based and updated -- that's the best path forward; otherwise,
there are too many conflicts if trying to re-base and revv the
original patch in gerrit [1].
To verify that the number of cross-thread allocations is reduced with
this patch, I ran the target rpc-bench test scenario under customized
environment to trace tcmalloc allocations/deallocations, and then used
the Python script attached to KUDU-1865 ticket (updated a bit) to track
cross-thread allocations/deallocations in tcmalloc:
env TCMALLOC_TRACE=1 \
TCMALLOC_STACKTRACE_METHOD=generic_fp \
./bin/rpc-bench \
--server_reactors=1 \
--client_threads=1 \
--run_seconds=1 \
--gtest_filter='*BenchmarkCalls'
The results confirm that all the malloc/free asymmetry related to
ReactorTask objects is now gone. I'm planning to take care of
the rest of allocation/de-allocation asymmetry in hot RPC paths
in follow-up patches.
I ran the same test scenario with different parameters under a custom
environment to evaluate the total number of calls that lead to acquiring
locks guarding the per-size-classes' central free lists in tcmalloc.
I did so with the default and customized setting for the
TCMALLOC_MAX_TOTAL_THREAD_CACHE_BYTES environment variable. The
customization allowed to induce more activity on tcmalloc's central
free lists since the sizes of the free lists were shorter than with the
default setting.
The results of running rpc-bench under the perf utility on 8 CPU-core
machine are below. There is an improvement:
* ~1.37x times reduction in the total number of calls to the free lists
with the default setting of TCMALLOC_MAX_TOTAL_THREAD_CACHE_BYTES
* ~1.06x times reduction in the total number of calls to the free lists
with the custom setting of TCMALLOC_MAX_TOTAL_THREAD_CACHE_BYTES
As for requests-per-second performance, the improvement is minuscule.
The target test scenario doesn't create enough memory pressure and
concurrency to get to the point where too much of lock contention in
tcmalloc significantly affects RPC performance.
export TCMALLOC_MAX_TOTAL_THREAD_CACHE_BYTES=1048576
perf record -g -e 'syscalls:sys_enter_futex' \
./bin/rpc-bench \
--gtest_filter='*BenchmarkCalls' \
--server_reactors=2 \
--client_threads=4 \
--worker_threads=2 \
--run_seconds=60
perf script | \
grep -E 'tcmalloc::ThreadCache::(FetchFrom|ReleaseTo)CentralCache' | \
wc -l
TCMALLOC_MAX_TOTAL_THREAD_CACHE_BYTES unset (using default tcmalloc setting):
before: 499 488 491
after: 373 328 376
TCMALLOC_MAX_TOTAL_THREAD_CACHE_BYTES=1048576
before: 13794 12727 12215
after: 12695 12137 11857
[1] https://gerrit.cloudera.org/#/c/5905/
Change-Id: I86a2acbd1d8cb724728034c4e91907c99cbfe32e
---
M src/kudu/rpc/connection.cc
M src/kudu/rpc/connection.h
M src/kudu/rpc/messenger.cc
M src/kudu/rpc/messenger.h
M src/kudu/rpc/negotiation.cc
M src/kudu/rpc/reactor.cc
M src/kudu/rpc/reactor.h
7 files changed, 190 insertions(+), 269 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/05/24305/5
--
To view, visit http://gerrit.cloudera.org:8080/24305
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I86a2acbd1d8cb724728034c4e91907c99cbfe32e
Gerrit-Change-Number: 24305
Gerrit-PatchSet: 5
Gerrit-Owner: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward (763)
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: David Rorke <dr...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Zoltan Martonka <zm...@cloudera.com>
[kudu-CR] KUDU-1865 reduce cross-thread allocations in RPC (part 1)
Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Alexey Serbin has removed a vote on this change.
Change subject: KUDU-1865 reduce cross-thread allocations in RPC (part 1)
......................................................................
Removed Verified-1 by Kudu Jenkins (120)
--
To view, visit http://gerrit.cloudera.org:8080/24305
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: deleteVote
Gerrit-Change-Id: I86a2acbd1d8cb724728034c4e91907c99cbfe32e
Gerrit-Change-Number: 24305
Gerrit-PatchSet: 5
Gerrit-Owner: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Anonymous Coward (763)
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: David Rorke <dr...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Zoltan Martonka <zm...@cloudera.com>
[kudu-CR] KUDU-1865 reduce cross-thread allocations in RPC hot paths (part 1)
Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Hello Kudu Jenkins, Abhishek Chennaka,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/24305
to look at the new patch set (#2).
Change subject: KUDU-1865 reduce cross-thread allocations in RPC hot paths (part 1)
......................................................................
KUDU-1865 reduce cross-thread allocations in RPC hot paths (part 1)
This patch was originally posted by Todd [1]. I'm re-posting it
with updates -- that's the best path forward, otherwise there are too
many conflicts if trying to re-base and revv the original patch
in gerrit [1].
To verify that the number of cross-thread allocations is reduced with
this patch, I ran the target rpc-bench test scenario under customized
environment to trace tcmalloc allocations/deallocations, and then used
the Python script attached to KUDU-1865 ticket (with modifications)
to track cross-thread allocations/deallocations in tcmalloc:
env TCMALLOC_TRACE=1 \
TCMALLOC_RECLAIM_MEMORY=0 \
TCMALLOC_STACKTRACE_METHOD=generic_fp \
./bin/rpc-bench \
--server_reactors=1 \
--client_threads=1 \
--run_seconds=1 \
--gtest_filter='*BenchmarkCalls'
The results confirm that all the malloc/free asymmetry related to
ReactorTask objects is now gone. I'm planning to take care of
the rest of allocation/de-allocation asymmetry in hot RPC paths
in follow-up patches.
I ran the same test scenario with different parameters under customized
environment to evaluate the total amount of lock contention and number
of I/O wait syscalls: the TCMALLOC_MAX_TOTAL_THREAD_CACHE_BYTES env
variable was customized to induce more activity on tcmalloc's central
free list.
The result of running rpc-bench under the perf utility on 8 CPU-core
machine are below. While the numbers have shown some amount of
variation, the two samples below reflect the picture seen after running
the test scenario many times. Since I didn't specify the address of
the speenlock in tcmalloc, this provides quite a fuzzy picture on
the actual improvement in libtcmalloc's contention in particular,
but it shows overall improvement.
export TCMALLOC_MAX_TOTAL_THREAD_CACHE_BYTES=2097152
perf stat \
-e 'syscalls:sys_enter_futex' \
-e 'syscalls:sys_enter_epoll_wait*' \
./bin/rpc-bench \
--gtest_filter='*BenchmarkCalls' \
--server_reactors=2 \
--client_threads=4 \
--worker_threads=2 \
--run_seconds=60
before:
11,907,775 syscalls:sys_enter_futex
8,983,211 syscalls:sys_enter_epoll_wait
60.014426125 seconds time elapsed
41.389530000 seconds user
128.630758000 seconds sys
after:
9,951,308 syscalls:sys_enter_futex
7,910,928 syscalls:sys_enter_epoll_wait
60.014275152 seconds time elapsed
37.602754000 seconds user
116.237592000 seconds sys
[1] https://gerrit.cloudera.org/#/c/5905/
Change-Id: I86a2acbd1d8cb724728034c4e91907c99cbfe32e
---
M src/kudu/rpc/connection.cc
M src/kudu/rpc/connection.h
M src/kudu/rpc/messenger.cc
M src/kudu/rpc/messenger.h
M src/kudu/rpc/negotiation.cc
M src/kudu/rpc/reactor.cc
M src/kudu/rpc/reactor.h
7 files changed, 189 insertions(+), 269 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/05/24305/2
--
To view, visit http://gerrit.cloudera.org:8080/24305
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I86a2acbd1d8cb724728034c4e91907c99cbfe32e
Gerrit-Change-Number: 24305
Gerrit-PatchSet: 2
Gerrit-Owner: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
[kudu-CR] KUDU-1865 reduce cross-thread allocations in RPC (part 1)
Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Hello Kurt Deschler, Zoltan Martonka, Abhishek Rawat, Kudu Jenkins, Abhishek Chennaka, David Rorke, Joe McDonnell, Csaba Ringhofer, Michael Smith,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/24305
to look at the new patch set (#4).
Change subject: KUDU-1865 reduce cross-thread allocations in RPC (part 1)
......................................................................
KUDU-1865 reduce cross-thread allocations in RPC (part 1)
This patch was originally posted by Todd [1]. I'm re-posting it
re-based and updated -- that's the best path forward; otherwise,
there are too many conflicts if trying to re-base and revv the
original patch in gerrit [1].
To verify that the number of cross-thread allocations is reduced with
this patch, I ran the target rpc-bench test scenario under customized
environment to trace tcmalloc allocations/deallocations, and then used
the Python script attached to KUDU-1865 ticket (updated a bit) to track
cross-thread allocations/deallocations in tcmalloc:
env TCMALLOC_TRACE=1 \
TCMALLOC_RECLAIM_MEMORY=0 \
TCMALLOC_STACKTRACE_METHOD=generic_fp \
./bin/rpc-bench \
--server_reactors=1 \
--client_threads=1 \
--run_seconds=1 \
--gtest_filter='*BenchmarkCalls'
The results confirm that all the malloc/free asymmetry related to
ReactorTask objects is now gone. I'm planning to take care of
the rest of allocation/de-allocation asymmetry in hot RPC paths
in follow-up patches.
I ran the same test scenario with different parameters under a custom
environment to evaluate the total number of calls that lead to acquiring
locks guarding the per-size-classes' central free lists in tcmalloc.
I did so with the default and customized setting for the
TCMALLOC_MAX_TOTAL_THREAD_CACHE_BYTES environment variable. The
customization allowed to induce more activity on tcmalloc's central
free lists since the sizes of the free lists were shorter than with the
default setting.
The results of running rpc-bench under the perf utility on 8 CPU-core
machine are below. While the numbers have shown some amount of
variation from run to run, the samples below reflect the picture
seen after running the test scenario many times. Since I didn't specify
the address of the corresponding spinlock in tcmalloc, I used an ad-hoc
filtering via grep on the stacks output by 'perf script'.
There is an improvement, indeed:
* ~1.37x times reduction in the total number of calls to the free lists
with the default setting of TCMALLOC_MAX_TOTAL_THREAD_CACHE_BYTES
* ~1.06x times reduction in the total number of calls to the free lists
with the custom setting of TCMALLOC_MAX_TOTAL_THREAD_CACHE_BYTES
The test shows better improvement with higher number of client and worker
threads, but the deviation in results from run to run jumps as well when
saturating the CPU. I think it a good idea to compare the performance on
a non-saturated CPU.
As for requests-per-second performance, the improvement is minuscule.
The target test scenario doesn't create enough memory pressure and
concurrency to get to the point where too much of lock contention in
tcmalloc significantly affects RPC performance.
export TCMALLOC_MAX_TOTAL_THREAD_CACHE_BYTES=1048576
perf record -g -e 'syscalls:sys_enter_futex' \
./bin/rpc-bench \
--gtest_filter='*BenchmarkCalls' \
--server_reactors=2 \
--client_threads=4 \
--worker_threads=2 \
--run_seconds=60
perf script | \
grep -E 'tcmalloc::ThreadCache::(FetchFrom|ReleaseTo)CentralCache' | \
wc -l
TCMALLOC_MAX_TOTAL_THREAD_CACHE_BYTES unset (using default tcmalloc setting):
before: 499 488 491
after: 373 328 376
TCMALLOC_MAX_TOTAL_THREAD_CACHE_BYTES=1048576
before: 13794 12727 12215
after: 12695 12137 11857
[1] https://gerrit.cloudera.org/#/c/5905/
Change-Id: I86a2acbd1d8cb724728034c4e91907c99cbfe32e
---
M src/kudu/rpc/connection.cc
M src/kudu/rpc/connection.h
M src/kudu/rpc/messenger.cc
M src/kudu/rpc/messenger.h
M src/kudu/rpc/negotiation.cc
M src/kudu/rpc/reactor.cc
M src/kudu/rpc/reactor.h
7 files changed, 189 insertions(+), 269 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/05/24305/4
--
To view, visit http://gerrit.cloudera.org:8080/24305
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I86a2acbd1d8cb724728034c4e91907c99cbfe32e
Gerrit-Change-Number: 24305
Gerrit-PatchSet: 4
Gerrit-Owner: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: David Rorke <dr...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Zoltan Martonka <zm...@cloudera.com>
[kudu-CR] KUDU-1865 reduce cross-thread allocations in RPC (part 1)
Posted by "Kudu Jenkins (Code Review)" <ge...@cloudera.org>.
Kudu Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24305 )
Change subject: KUDU-1865 reduce cross-thread allocations in RPC (part 1)
......................................................................
Patch Set 5:
Build Started http://jenkins.kudu.apache.org/job/pre_commit/2573/
--
To view, visit http://gerrit.cloudera.org:8080/24305
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I86a2acbd1d8cb724728034c4e91907c99cbfe32e
Gerrit-Change-Number: 24305
Gerrit-PatchSet: 5
Gerrit-Owner: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward (763)
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: David Rorke <dr...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Zoltan Martonka <zm...@cloudera.com>
Gerrit-Comment-Date: Fri, 15 May 2026 14:58:34 +0000
Gerrit-HasComments: No
[kudu-CR] KUDU-1865 reduce cross-thread allocations in RPC (part 1)
Posted by "Kudu Jenkins (Code Review)" <ge...@cloudera.org>.
Kudu Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24305 )
Change subject: KUDU-1865 reduce cross-thread allocations in RPC (part 1)
......................................................................
Patch Set 5: Verified-1
Build Failed
http://jenkins.kudu.apache.org/job/pre_commit/2573/ : FAILURE
--
To view, visit http://gerrit.cloudera.org:8080/24305
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I86a2acbd1d8cb724728034c4e91907c99cbfe32e
Gerrit-Change-Number: 24305
Gerrit-PatchSet: 5
Gerrit-Owner: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Anonymous Coward (763)
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: David Rorke <dr...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Zoltan Martonka <zm...@cloudera.com>
Gerrit-Comment-Date: Fri, 15 May 2026 15:38:50 +0000
Gerrit-HasComments: No
[kudu-CR] KUDU-1865 reduce cross-thread allocations in RPC (part 1)
Posted by "Kurt Deschler (Code Review)" <ge...@cloudera.org>.
Kurt Deschler has posted comments on this change. ( http://gerrit.cloudera.org:8080/24305 )
Change subject: KUDU-1865 reduce cross-thread allocations in RPC (part 1)
......................................................................
Patch Set 5:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/24305/5//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/24305/5//COMMIT_MSG@69
PS5, Line 69: before: 499 488 491
What are the 3 columns of numbers?
--
To view, visit http://gerrit.cloudera.org:8080/24305
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I86a2acbd1d8cb724728034c4e91907c99cbfe32e
Gerrit-Change-Number: 24305
Gerrit-PatchSet: 5
Gerrit-Owner: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Anonymous Coward (763)
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: David Rorke <dr...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Zoltan Martonka <zm...@cloudera.com>
Gerrit-Comment-Date: Mon, 18 May 2026 20:10:02 +0000
Gerrit-HasComments: Yes
[kudu-CR] KUDU-1865 reduce cross-thread allocations in RPC (part 1)
Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Hello Kudu Jenkins, Abhishek Chennaka,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/24305
to look at the new patch set (#3).
Change subject: KUDU-1865 reduce cross-thread allocations in RPC (part 1)
......................................................................
KUDU-1865 reduce cross-thread allocations in RPC (part 1)
This patch was originally posted by Todd [1]. I'm re-posting it
re-based and updated -- that's the best path forward; otherwise,
there are too many conflicts if trying to re-base and revv the
original patch in gerrit [1].
To verify that the number of cross-thread allocations is reduced with
this patch, I ran the target rpc-bench test scenario under customized
environment to trace tcmalloc allocations/deallocations, and then used
the Python script attached to KUDU-1865 ticket (updated a bit) to track
cross-thread allocations/deallocations in tcmalloc:
env TCMALLOC_TRACE=1 \
TCMALLOC_RECLAIM_MEMORY=0 \
TCMALLOC_STACKTRACE_METHOD=generic_fp \
./bin/rpc-bench \
--server_reactors=1 \
--client_threads=1 \
--run_seconds=1 \
--gtest_filter='*BenchmarkCalls'
The results confirm that all the malloc/free asymmetry related to
ReactorTask objects is now gone. I'm planning to take care of
the rest of allocation/de-allocation asymmetry in hot RPC paths
in follow-up patches.
I ran the same test scenario with different parameters under a custom
environment to evaluate the total number of calls that lead to acquiring
locks guarding the per-size-classes' central free lists in tcmalloc.
I did so with the default and customized setting for the
TCMALLOC_MAX_TOTAL_THREAD_CACHE_BYTES environment variable. The
customization allowed to induce more activity on tcmalloc's central
free lists since the sizes of the free lists were shorter than with the
default setting.
The result of running rpc-bench under the perf utility on 8 CPU-core
machine are below. While the numbers have shown some amount of
variation from run to run, the two samples below reflect the picture
seen after running the test scenario many times. Since I didn't specify
the address of the corresponding spinlock in tcmalloc, I used an ad-hoc
filtering via grep on the stacks output by 'perf script'.
There is an improvement, indeed:
* ~1.37x times reduction in the total number of calls to the free lists
with the default setting of TCMALLOC_MAX_TOTAL_THREAD_CACHE_BYTES
* ~1.06x times reduction in the total number of calls to the free lists
with the custom setting of TCMALLOC_MAX_TOTAL_THREAD_CACHE_BYTES
export TCMALLOC_MAX_TOTAL_THREAD_CACHE_BYTES=1048576
perf record -g -e 'syscalls:sys_enter_futex' \
./bin/rpc-bench \
--gtest_filter='*BenchmarkCalls' \
--server_reactors=2 \
--client_threads=4 \
--worker_threads=2 \
--run_seconds=60
perf script | \
grep -E 'tcmalloc::ThreadCache::(FetchFrom|ReleaseTo)CentralCache' | \
wc -l
TCMALLOC_MAX_TOTAL_THREAD_CACHE_BYTES unset (using default tcmalloc setting):
before: 499 488 491
after: 373 328 376
TCMALLOC_MAX_TOTAL_THREAD_CACHE_BYTES=1048576
before: 13794 12727 12215
after: 12695 12137 11857
[1] https://gerrit.cloudera.org/#/c/5905/
Change-Id: I86a2acbd1d8cb724728034c4e91907c99cbfe32e
---
M src/kudu/rpc/connection.cc
M src/kudu/rpc/connection.h
M src/kudu/rpc/messenger.cc
M src/kudu/rpc/messenger.h
M src/kudu/rpc/negotiation.cc
M src/kudu/rpc/reactor.cc
M src/kudu/rpc/reactor.h
7 files changed, 189 insertions(+), 269 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/05/24305/3
--
To view, visit http://gerrit.cloudera.org:8080/24305
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I86a2acbd1d8cb724728034c4e91907c99cbfe32e
Gerrit-Change-Number: 24305
Gerrit-PatchSet: 3
Gerrit-Owner: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)