You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@kudu.apache.org by "Alexey Serbin (Code Review)" <ge...@cloudera.org> on 2020/09/02 01:25:20 UTC
[kudu-CR] [util] add performance test for ThreadPool
Alexey Serbin has uploaded this change for review. ( http://gerrit.cloudera.org:8080/16401
Change subject: [util] add performance test for ThreadPool
......................................................................
[util] add performance test for ThreadPool
Added test scenario to assess ThreadPool's performance in the presence
and absence of QueueLoadMeter. The newly added scenario uses a mix of
serial and concurrent tasks. The results are as follows when running
a RELEASE build on a 48-core E5-2680 Xeon @ 2.50GHz machine:
Processed 600000 tasks in real 1.366s user 0.933s sys 54.512s
Processing rate (QueueLoadMeter disabled): 439345.75398195261 tasks/sec
Processed 600000 tasks in real 2.484s user 1.815s sys 111.655s
Processing rate (QueueLoadMeter enabled): 241564.74471673707 tasks/sec
Change-Id: I470d204f3b3a22407361a920882a93d6f21dd274
---
M src/kudu/util/threadpool-test.cc
1 file changed, 82 insertions(+), 4 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/01/16401/1
--
To view, visit http://gerrit.cloudera.org:8080/16401
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I470d204f3b3a22407361a920882a93d6f21dd274
Gerrit-Change-Number: 16401
Gerrit-PatchSet: 1
Gerrit-Owner: Alexey Serbin <as...@cloudera.com>
[kudu-CR] [util] add performance test scenario for ThreadPool
Posted by "Andrew Wong (Code Review)" <ge...@cloudera.org>.
Andrew Wong has posted comments on this change. ( http://gerrit.cloudera.org:8080/16401 )
Change subject: [util] add performance test scenario for ThreadPool
......................................................................
Patch Set 2:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/16401/2/src/kudu/util/threadpool-test.cc
File src/kudu/util/threadpool-test.cc:
http://gerrit.cloudera.org:8080/#/c/16401/2/src/kudu/util/threadpool-test.cc@736
PS2, Line 736: const auto kQueueTimeThreshold = MonoDelta::FromMilliseconds(100);
Is the motivation of this patch to determine a good default value for this? Or is the queue time threshold not important to throughput in the context of this test?
http://gerrit.cloudera.org:8080/#/c/16401/2/src/kudu/util/threadpool-test.cc@762
PS2, Line 762: if (i % 100 == 0) {
: SleepFor(MonoDelta::FromMicroseconds(1));
: }
Does this sleep make a noticeable difference on the throughputs reported by this test?
--
To view, visit http://gerrit.cloudera.org:8080/16401
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I470d204f3b3a22407361a920882a93d6f21dd274
Gerrit-Change-Number: 16401
Gerrit-PatchSet: 2
Gerrit-Owner: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Wed, 02 Sep 2020 19:32:21 +0000
Gerrit-HasComments: Yes
[kudu-CR] [util] add performance test scenario for ThreadPool
Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Alexey Serbin has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/16401 )
Change subject: [util] add performance test scenario for ThreadPool
......................................................................
[util] add performance test scenario for ThreadPool
Added a test scenario to assess ThreadPool's performance in the absence
and the presence of QueueLoadMeter. The newly added scenario uses
a mix of serial and concurrent ThreadPool tokens.
Below are the results when running a RELEASE build on a 48-core
E5-2680 Xeon @ 2.50GHz machine:
Processed 600000 tasks in real 1.366s user 0.933s sys 54.512s
Processing rate (QueueLoadMeter disabled): 439345.75398195261 tasks/sec
Processed 600000 tasks in real 2.484s user 1.815s sys 111.655s
Processing rate (QueueLoadMeter enabled): 241564.74471673707 tasks/sec
Change-Id: I470d204f3b3a22407361a920882a93d6f21dd274
Reviewed-on: http://gerrit.cloudera.org:8080/16401
Tested-by: Alexey Serbin <as...@cloudera.com>
Reviewed-by: Andrew Wong <aw...@cloudera.com>
---
M src/kudu/util/threadpool-test.cc
1 file changed, 84 insertions(+), 4 deletions(-)
Approvals:
Alexey Serbin: Verified
Andrew Wong: Looks good to me, approved
--
To view, visit http://gerrit.cloudera.org:8080/16401
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I470d204f3b3a22407361a920882a93d6f21dd274
Gerrit-Change-Number: 16401
Gerrit-PatchSet: 5
Gerrit-Owner: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
[kudu-CR] [util] add performance test scenario for ThreadPool
Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Alexey Serbin has posted comments on this change. ( http://gerrit.cloudera.org:8080/16401 )
Change subject: [util] add performance test scenario for ThreadPool
......................................................................
Patch Set 3:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/16401/2/src/kudu/util/threadpool-test.cc
File src/kudu/util/threadpool-test.cc:
http://gerrit.cloudera.org:8080/#/c/16401/2/src/kudu/util/threadpool-test.cc@762
PS2, Line 762: for (auto i = 0; i < kNumTasksPerSchedulerThread; ++i) {
: CHECK_OK(token->Submit([](){}));
: }
> It doesn't make a noticeable difference in the throughput, but I think it m
After playing with that a couple more times, I removed this. The idea was to have a scenario closer to real ones and induce some irregularities in growing and shrinking the queue, but it's not crucial in the scope of this benchmark.
http://gerrit.cloudera.org:8080/#/c/16401/3/src/kudu/util/threadpool-test.cc
File src/kudu/util/threadpool-test.cc:
http://gerrit.cloudera.org:8080/#/c/16401/3/src/kudu/util/threadpool-test.cc@737
PS3, Line 737: t
> nit: enough
Done
--
To view, visit http://gerrit.cloudera.org:8080/16401
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I470d204f3b3a22407361a920882a93d6f21dd274
Gerrit-Change-Number: 16401
Gerrit-PatchSet: 3
Gerrit-Owner: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Thu, 03 Sep 2020 20:01:04 +0000
Gerrit-HasComments: Yes
[kudu-CR] [util] add performance test scenario for ThreadPool
Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Hello Kudu Jenkins, Andrew Wong, Grant Henke,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/16401
to look at the new patch set (#2).
Change subject: [util] add performance test scenario for ThreadPool
......................................................................
[util] add performance test scenario for ThreadPool
Added a test scenario to assess ThreadPool's performance in the absence
and the presence of QueueLoadMeter. The newly added scenario uses
a mix of serial and concurrent ThreadPool tokens.
Below are the results when running a RELEASE build on a 48-core
E5-2680 Xeon @ 2.50GHz machine:
Processed 600000 tasks in real 1.366s user 0.933s sys 54.512s
Processing rate (QueueLoadMeter disabled): 439345.75398195261 tasks/sec
Processed 600000 tasks in real 2.484s user 1.815s sys 111.655s
Processing rate (QueueLoadMeter enabled): 241564.74471673707 tasks/sec
Change-Id: I470d204f3b3a22407361a920882a93d6f21dd274
---
M src/kudu/util/threadpool-test.cc
1 file changed, 85 insertions(+), 4 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/01/16401/2
--
To view, visit http://gerrit.cloudera.org:8080/16401
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I470d204f3b3a22407361a920882a93d6f21dd274
Gerrit-Change-Number: 16401
Gerrit-PatchSet: 2
Gerrit-Owner: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
[kudu-CR] [util] add performance test scenario for ThreadPool
Posted by "Andrew Wong (Code Review)" <ge...@cloudera.org>.
Andrew Wong has posted comments on this change. ( http://gerrit.cloudera.org:8080/16401 )
Change subject: [util] add performance test scenario for ThreadPool
......................................................................
Patch Set 4: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/16401
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I470d204f3b3a22407361a920882a93d6f21dd274
Gerrit-Change-Number: 16401
Gerrit-PatchSet: 4
Gerrit-Owner: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Thu, 03 Sep 2020 21:01:11 +0000
Gerrit-HasComments: No
[kudu-CR] [util] add performance test scenario for ThreadPool
Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Hello Kudu Jenkins, Andrew Wong, Grant Henke,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/16401
to look at the new patch set (#4).
Change subject: [util] add performance test scenario for ThreadPool
......................................................................
[util] add performance test scenario for ThreadPool
Added a test scenario to assess ThreadPool's performance in the absence
and the presence of QueueLoadMeter. The newly added scenario uses
a mix of serial and concurrent ThreadPool tokens.
Below are the results when running a RELEASE build on a 48-core
E5-2680 Xeon @ 2.50GHz machine:
Processed 600000 tasks in real 1.366s user 0.933s sys 54.512s
Processing rate (QueueLoadMeter disabled): 439345.75398195261 tasks/sec
Processed 600000 tasks in real 2.484s user 1.815s sys 111.655s
Processing rate (QueueLoadMeter enabled): 241564.74471673707 tasks/sec
Change-Id: I470d204f3b3a22407361a920882a93d6f21dd274
---
M src/kudu/util/threadpool-test.cc
1 file changed, 84 insertions(+), 4 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/01/16401/4
--
To view, visit http://gerrit.cloudera.org:8080/16401
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I470d204f3b3a22407361a920882a93d6f21dd274
Gerrit-Change-Number: 16401
Gerrit-PatchSet: 4
Gerrit-Owner: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
[kudu-CR] [util] add performance test scenario for ThreadPool
Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Alexey Serbin has posted comments on this change. ( http://gerrit.cloudera.org:8080/16401 )
Change subject: [util] add performance test scenario for ThreadPool
......................................................................
Patch Set 4: Verified+1
Unrelated test failure in DeleteTabletITest.TestLeaderElectionDuringDeleteTablet (ASAN build)
--
To view, visit http://gerrit.cloudera.org:8080/16401
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I470d204f3b3a22407361a920882a93d6f21dd274
Gerrit-Change-Number: 16401
Gerrit-PatchSet: 4
Gerrit-Owner: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Thu, 03 Sep 2020 20:38:20 +0000
Gerrit-HasComments: No
[kudu-CR] [util] add performance test scenario for ThreadPool
Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Alexey Serbin has posted comments on this change. ( http://gerrit.cloudera.org:8080/16401 )
Change subject: [util] add performance test scenario for ThreadPool
......................................................................
Patch Set 2:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/16401/2/src/kudu/util/threadpool-test.cc
File src/kudu/util/threadpool-test.cc:
http://gerrit.cloudera.org:8080/#/c/16401/2/src/kudu/util/threadpool-test.cc@736
PS2, Line 736: const auto kQueueTimeThreshold = MonoDelta::FromMilliseconds(100);
> Is the motivation of this patch to determine a good default value for this?
It's just some value to set; it's important. The only thought was to have a threshold that most likely lead to detecting queue overload status.
http://gerrit.cloudera.org:8080/#/c/16401/2/src/kudu/util/threadpool-test.cc@762
PS2, Line 762: if (i % 100 == 0) {
: SleepFor(MonoDelta::FromMicroseconds(1));
: }
> Does this sleep make a noticeable difference on the throughputs reported by
It doesn't make a noticeable difference in the throughput, but I think it makes the scenario a bit closer to the real world scenarios: no real task usually just schedules thousands of tasks without doing some other work in the middle.
--
To view, visit http://gerrit.cloudera.org:8080/16401
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I470d204f3b3a22407361a920882a93d6f21dd274
Gerrit-Change-Number: 16401
Gerrit-PatchSet: 2
Gerrit-Owner: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Thu, 03 Sep 2020 00:32:24 +0000
Gerrit-HasComments: Yes
[kudu-CR] [util] add performance test scenario for ThreadPool
Posted by "Andrew Wong (Code Review)" <ge...@cloudera.org>.
Andrew Wong has posted comments on this change. ( http://gerrit.cloudera.org:8080/16401 )
Change subject: [util] add performance test scenario for ThreadPool
......................................................................
Patch Set 3: Code-Review+2
(2 comments)
http://gerrit.cloudera.org:8080/#/c/16401/2/src/kudu/util/threadpool-test.cc
File src/kudu/util/threadpool-test.cc:
http://gerrit.cloudera.org:8080/#/c/16401/2/src/kudu/util/threadpool-test.cc@736
PS2, Line 736: // The exact value of the queue overload threshold isn't important
> It's just some value to set; it's important. The only thought was to have
I see, thanks for clarifying.
http://gerrit.cloudera.org:8080/#/c/16401/3/src/kudu/util/threadpool-test.cc
File src/kudu/util/threadpool-test.cc:
http://gerrit.cloudera.org:8080/#/c/16401/3/src/kudu/util/threadpool-test.cc@737
PS3, Line 737: t
nit: enough
--
To view, visit http://gerrit.cloudera.org:8080/16401
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I470d204f3b3a22407361a920882a93d6f21dd274
Gerrit-Change-Number: 16401
Gerrit-PatchSet: 3
Gerrit-Owner: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Thu, 03 Sep 2020 19:45:18 +0000
Gerrit-HasComments: Yes
[kudu-CR] [util] add performance test scenario for ThreadPool
Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Hello Kudu Jenkins, Andrew Wong, Grant Henke,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/16401
to look at the new patch set (#3).
Change subject: [util] add performance test scenario for ThreadPool
......................................................................
[util] add performance test scenario for ThreadPool
Added a test scenario to assess ThreadPool's performance in the absence
and the presence of QueueLoadMeter. The newly added scenario uses
a mix of serial and concurrent ThreadPool tokens.
Below are the results when running a RELEASE build on a 48-core
E5-2680 Xeon @ 2.50GHz machine:
Processed 600000 tasks in real 1.366s user 0.933s sys 54.512s
Processing rate (QueueLoadMeter disabled): 439345.75398195261 tasks/sec
Processed 600000 tasks in real 2.484s user 1.815s sys 111.655s
Processing rate (QueueLoadMeter enabled): 241564.74471673707 tasks/sec
Change-Id: I470d204f3b3a22407361a920882a93d6f21dd274
---
M src/kudu/util/threadpool-test.cc
1 file changed, 84 insertions(+), 4 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/01/16401/3
--
To view, visit http://gerrit.cloudera.org:8080/16401
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I470d204f3b3a22407361a920882a93d6f21dd274
Gerrit-Change-Number: 16401
Gerrit-PatchSet: 3
Gerrit-Owner: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
[kudu-CR] [util] add performance test scenario for ThreadPool
Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Alexey Serbin has removed a vote on this change.
Change subject: [util] add performance test scenario for ThreadPool
......................................................................
Removed Verified-1 by Kudu Jenkins (120)
--
To view, visit http://gerrit.cloudera.org:8080/16401
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: deleteVote
Gerrit-Change-Id: I470d204f3b3a22407361a920882a93d6f21dd274
Gerrit-Change-Number: 16401
Gerrit-PatchSet: 4
Gerrit-Owner: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)