You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@kudu.apache.org by "Alexey Serbin (Code Review)" <ge...@cloudera.org> on 2020/08/15 02:41:07 UTC
[kudu-CR] KUDU-1587 part2: reject write request if apply queue is overloaded
Alexey Serbin has uploaded this change for review. ( http://gerrit.cloudera.org:8080/16343
Change subject: KUDU-1587 part2: reject write request if apply queue is overloaded
......................................................................
KUDU-1587 part2: reject write request if apply queue is overloaded
This patch implements control admission for write requests in tablet
servers based on the load status of their apply queue. With this change,
the recently introduced OpApplyQueueTest.ApplyQueueBackpressure scenario
successfully passes.
By default, the overload threshold for the tablet server's apply queue
is set to 500 milliseconds. If the queue times of the tasks in the apply
queue become higher than the specified threshold, the apply queue enters
overloaded state. When the queue is overloaded, the tablet server
rejects incoming write requests with some probability. The longer
the queue stays overloaded, the greater the probability of rejections.
In addition, the more row operations a write request has, the greater
the probability of such request to be the rejected. The apply queue
exits the overloaded state when queue times drop below the specified
threshold.
This new behavior is now enabled by default. To disable it and return
to the legacy one, set --tablet_apply_pool_overload_threshold_ms to 0.
Change-Id: I6d7688d6fa832e606b8efc4549568fa52dfa1931
---
M src/kudu/kserver/kserver.cc
M src/kudu/tablet/tablet_replica.cc
M src/kudu/tablet/tablet_replica.h
M src/kudu/tserver/tablet_server-test.cc
M src/kudu/util/threadpool-test.cc
M src/kudu/util/threadpool.cc
M src/kudu/util/threadpool.h
7 files changed, 345 insertions(+), 296 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/43/16343/1
--
To view, visit http://gerrit.cloudera.org:8080/16343
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I6d7688d6fa832e606b8efc4549568fa52dfa1931
Gerrit-Change-Number: 16343
Gerrit-PatchSet: 1
Gerrit-Owner: Alexey Serbin <as...@cloudera.com>
[kudu-CR] KUDU-1587 part 2: reject write ops if apply queue is overloaded
Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Hello Kudu Jenkins, Andrew Wong, Todd Lipcon,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/16343
to look at the new patch set (#3).
Change subject: KUDU-1587 part 2: reject write ops if apply queue is overloaded
......................................................................
KUDU-1587 part 2: reject write ops if apply queue is overloaded
This patch implements control admission for write requests in tablet
servers based on the load status of their apply queue. With this change,
the recently introduced OpApplyQueueTest.ApplyQueueBackpressure scenario
successfully passes.
If the queue times of the tasks in the apply queue become higher than
the specified threshold, the apply queue enters overloaded state. When
the queue is overloaded, the tablet server rejects incoming write
requests with some probability. The longer the queue stays overloaded,
the greater the probability of rejections. In addition, the more row
operations a write request has, the greater the probability of such
request to be the rejected. The apply queue exits the overloaded state
when queue times drop below the specified threshold.
This new behavior is not yet enabled by default, keeping the legacy
behavior of unbounded/uncontrolled queue times as is. To enable it,
set --tablet_apply_pool_overload_threshold_ms to something greater
than 0 (e.g., 500).
Change-Id: I6d7688d6fa832e606b8efc4549568fa52dfa1931
---
M src/kudu/kserver/kserver.cc
M src/kudu/tablet/tablet_replica.cc
M src/kudu/tablet/tablet_replica.h
M src/kudu/tserver/tablet_server-test.cc
M src/kudu/util/threadpool-test.cc
M src/kudu/util/threadpool.cc
M src/kudu/util/threadpool.h
7 files changed, 354 insertions(+), 299 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/43/16343/3
--
To view, visit http://gerrit.cloudera.org:8080/16343
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I6d7688d6fa832e606b8efc4549568fa52dfa1931
Gerrit-Change-Number: 16343
Gerrit-PatchSet: 3
Gerrit-Owner: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
[kudu-CR] KUDU-1587 part 2: reject write ops if apply queue is overloaded
Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Alexey Serbin has posted comments on this change. ( http://gerrit.cloudera.org:8080/16343 )
Change subject: KUDU-1587 part 2: reject write ops if apply queue is overloaded
......................................................................
Patch Set 6:
(8 comments)
http://gerrit.cloudera.org:8080/#/c/16343/6//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/16343/6//COMMIT_MSG@23
PS6, Line 23: new behavior is not yet enabled by default
> What is the rational for not enabling by default? Is there some threshold_m
I think we can put 1000 ms (i.e. one second) as somewhat safe default value.
However, due to the timing of the upcoming release, I don't think I have enough time to make exhaustive testing of various workloads to be sure the default value is completely safe. But I'll try to run various workloads, at least those that Andrew pointed at.
http://gerrit.cloudera.org:8080/#/c/16343/6//COMMIT_MSG@23
PS6, Line 23: This new behavior is not yet enabled by default, keeping the legacy
: behavior of unbounded/uncontrolled queue times as is. To enable it,
: set --tablet_apply_pool_overload_threshold_ms to something greater
: than 0 (e.g., 500).
> If not in an automated test, it'd be great if you could manually run a simi
I added OpApplyQueueOverloadedTest.ClientRetriesOperations (based on SameTabletConcurrentWritesTest). The new test scenario verifies that the throttled requests are retried. This new test scenario shows almost the same request rate with and without op apply queue-based throttling: at my test machine, it's 492.25 vs 487 req/sec with 100 ms delay injected into the apply phase of a write operation.
I'll take a look at how it looks if running against a real cluster with real workload.
http://gerrit.cloudera.org:8080/#/c/16343/6/src/kudu/tablet/tablet_replica.h
File src/kudu/tablet/tablet_replica.h:
http://gerrit.cloudera.org:8080/#/c/16343/6/src/kudu/tablet/tablet_replica.h@411
PS6, Line 411: Random rng_;
> Shouldn't we be using ThreadSafeRandom, given we call this from the reactor
Whoops. I thought Random was thread-safe itself.
http://gerrit.cloudera.org:8080/#/c/16343/6/src/kudu/tablet/tablet_replica.cc
File src/kudu/tablet/tablet_replica.cc:
http://gerrit.cloudera.org:8080/#/c/16343/6/src/kudu/tablet/tablet_replica.cc@456
PS6, Line 456: size_factor
> I am not surely I fully understand why we want to give preference to smalle
The intuition was that smaller batches put less load on the apply queue, so rejecting heavier patches would lead returning to non-overloaded queue faster. Another point was to give workloads with small batches higher priority, assuming that the load on the apply queue is induced mostly by workloads with larger batches.
We discussed this offline. The resolution is that without backing this by actual evidence provided by tests, it's better to not discriminate based on the size of batches. An alternative might be keeping the size factor, but making it configurable via a flag, but it looks exactly as a YAGNI case. So, I'm removing the size factor for now: we can add it later on once we have more data on this.
http://gerrit.cloudera.org:8080/#/c/16343/6/src/kudu/tablet/tablet_replica.cc@445
PS6, Line 445: // If the apply queue is overloaded, reject the incoming operation with
: // some probability.
: MonoDelta queue_otime;
: MonoDelta threshold;
: if (apply_pool_->QueueOverloaded(&queue_otime, &threshold)) {
: auto overload_threshold_ms = threshold.ToMilliseconds();
: // The longer the queue has been in the overloaded state, the higher the
: // probability of rejection.
: auto time_factor = queue_otime.ToMilliseconds() / overload_threshold_ms + 1;
: // The heavier the request in terms of number of rows, the higher the
: // probability of rejecting it if the apply queue is overloaded.
: auto size_factor = op_state->request()->row_operations().rows().size();
: auto factor = 1 + size_factor * time_factor;
: if (!rng_.OneIn(factor)) {
: return Status::ServiceUnavailable("op apply queue is overloaded");
: }
: }
> +1 to this if it's not to complicated.
Done
http://gerrit.cloudera.org:8080/#/c/16343/6/src/kudu/tablet/tablet_replica.cc@445
PS6, Line 445: // If the apply queue is overloaded, reject the incoming operation with
: // some probability.
: MonoDelta queue_otime;
: MonoDelta threshold;
: if (apply_pool_->QueueOverloaded(&queue_otime, &threshold)) {
: auto overload_threshold_ms = threshold.ToMilliseconds();
: // The longer the queue has been in the overloaded state, the higher the
: // probability of rejection.
: auto time_factor = queue_otime.ToMilliseconds() / overload_threshold_ms + 1;
: // The heavier the request in terms of number of rows, the higher the
: // probability of rejecting it if the apply queue is overloaded.
: auto size_factor = op_state->request()->row_operations().rows().size();
: auto factor = 1 + size_factor * time_factor;
: if (!rng_.OneIn(factor)) {
: return Status::ServiceUnavailable("op apply queue is overloaded");
: }
: }
> Could we instead bake this into the TabletService endpoint? We already do s
Done
http://gerrit.cloudera.org:8080/#/c/16343/6/src/kudu/tserver/tablet_server-test.cc
File src/kudu/tserver/tablet_server-test.cc:
http://gerrit.cloudera.org:8080/#/c/16343/6/src/kudu/tserver/tablet_server-test.cc@4375
PS6, Line 4375: intjected
> nit: injected
Done
http://gerrit.cloudera.org:8080/#/c/16343/6/src/kudu/tserver/tablet_server-test.cc@4468
PS6, Line 4468: that
> nit: rather than
Done
--
To view, visit http://gerrit.cloudera.org:8080/16343
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6d7688d6fa832e606b8efc4549568fa52dfa1931
Gerrit-Change-Number: 16343
Gerrit-PatchSet: 6
Gerrit-Owner: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Tidy Bot (241)
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Comment-Date: Wed, 26 Aug 2020 08:41:35 +0000
Gerrit-HasComments: Yes
[kudu-CR] KUDU-1587 part 2: reject write ops if apply queue is overloaded
Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Hello Kudu Jenkins, Andrew Wong, Todd Lipcon,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/16343
to look at the new patch set (#4).
Change subject: KUDU-1587 part 2: reject write ops if apply queue is overloaded
......................................................................
KUDU-1587 part 2: reject write ops if apply queue is overloaded
This patch implements control admission for write requests in tablet
servers based on the load status of their apply queue. With this change,
the recently introduced OpApplyQueueTest.ApplyQueueBackpressure scenario
successfully passes.
If the queue times of the tasks in the apply queue become higher than
the specified threshold, the apply queue enters overloaded state. When
the queue is overloaded, the tablet server rejects incoming write
requests with some probability. The longer the queue stays overloaded,
the greater the probability of rejections. In addition, the more row
operations a write request has, the greater the probability of such
request to be the rejected. The apply queue exits the overloaded state
when queue times drop below the specified threshold.
This new behavior is not yet enabled by default, keeping the legacy
behavior of unbounded/uncontrolled queue times as is. To enable it,
set --tablet_apply_pool_overload_threshold_ms to something greater
than 0 (e.g., 500).
Change-Id: I6d7688d6fa832e606b8efc4549568fa52dfa1931
---
M src/kudu/kserver/kserver.cc
M src/kudu/tablet/tablet_replica.cc
M src/kudu/tablet/tablet_replica.h
M src/kudu/tserver/tablet_server-test.cc
4 files changed, 90 insertions(+), 34 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/43/16343/4
--
To view, visit http://gerrit.cloudera.org:8080/16343
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I6d7688d6fa832e606b8efc4549568fa52dfa1931
Gerrit-Change-Number: 16343
Gerrit-PatchSet: 4
Gerrit-Owner: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
[kudu-CR] KUDU-1587 part 2: reject write ops if apply queue is overloaded
Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Alexey Serbin has removed a vote on this change.
Change subject: KUDU-1587 part 2: reject write ops if apply queue is overloaded
......................................................................
Removed Verified-1 by Kudu Jenkins (120)
--
To view, visit http://gerrit.cloudera.org:8080/16343
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: deleteVote
Gerrit-Change-Id: I6d7688d6fa832e606b8efc4549568fa52dfa1931
Gerrit-Change-Number: 16343
Gerrit-PatchSet: 3
Gerrit-Owner: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
[kudu-CR] KUDU-1587 part 2: reject write ops if apply queue is overloaded
Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Hello Attila Bukor, Kudu Jenkins, Andrew Wong, Grant Henke, Todd Lipcon,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/16343
to look at the new patch set (#7).
Change subject: KUDU-1587 part 2: reject write ops if apply queue is overloaded
......................................................................
KUDU-1587 part 2: reject write ops if apply queue is overloaded
This patch implements control admission for write requests in tablet
servers based on the load status of their apply queue. With this change,
the recently introduced OpApplyQueueTest.ApplyQueueBackpressure scenario
successfully passes.
If the queue times of the tasks in the apply queue become higher than
the specified threshold, the apply queue enters overloaded state. When
the queue is overloaded, the tablet server rejects incoming write
requests with some probability. The longer the queue stays overloaded,
the greater the probability of rejections. In addition, the more row
operations a write request has, the greater the probability of such
request to be the rejected. The apply queue exits the overloaded state
when queue times drop below the specified threshold.
This new behavior is not yet enabled by default, keeping the legacy
behavior of unbounded/uncontrolled queue times as is. To enable it,
set --tablet_apply_pool_overload_threshold_ms to something greater
than 0 (e.g., 500).
Change-Id: I6d7688d6fa832e606b8efc4549568fa52dfa1931
---
M src/kudu/integration-tests/same_tablet_concurrent_writes-itest.cc
M src/kudu/kserver/kserver.cc
M src/kudu/tablet/tablet_replica.cc
M src/kudu/tserver/tablet_server-test.cc
M src/kudu/tserver/tablet_service.cc
M src/kudu/tserver/tablet_service.h
6 files changed, 267 insertions(+), 76 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/43/16343/7
--
To view, visit http://gerrit.cloudera.org:8080/16343
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I6d7688d6fa832e606b8efc4549568fa52dfa1931
Gerrit-Change-Number: 16343
Gerrit-PatchSet: 7
Gerrit-Owner: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
[kudu-CR] KUDU-1587 part 2: reject write ops if apply queue is overloaded
Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Hello Tidy Bot, Attila Bukor, Kudu Jenkins, Andrew Wong, Grant Henke, Todd Lipcon,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/16343
to look at the new patch set (#11).
Change subject: KUDU-1587 part 2: reject write ops if apply queue is overloaded
......................................................................
KUDU-1587 part 2: reject write ops if apply queue is overloaded
This patch implements control admission for write requests in tablet
servers based on the load status of their apply queue. With this change,
the recently introduced OpApplyQueueTest.ApplyQueueBackpressure scenario
successfully passes.
If the queue times of the tasks in the apply queue become higher than
the specified threshold, the apply queue enters overloaded state. When
the queue is overloaded, the tablet server rejects incoming write
requests with some probability. The longer the queue stays overloaded,
the greater the probability of rejections. The apply queue exits the
overloaded state when queue times drop below the specified threshold.
This new behavior is not yet enabled by default, keeping the legacy
behavior of unbounded/uncontrolled queue times as is. To enable it,
set --tablet_apply_pool_overload_threshold_ms to something greater
than 0 (e.g., 500).
Change-Id: I6d7688d6fa832e606b8efc4549568fa52dfa1931
---
M src/kudu/integration-tests/same_tablet_concurrent_writes-itest.cc
M src/kudu/kserver/kserver.cc
M src/kudu/kserver/kserver.h
A src/kudu/kserver/kserver_options.h
M src/kudu/master/master_options.h
M src/kudu/tablet/tablet_replica.cc
M src/kudu/tserver/tablet_server-test.cc
M src/kudu/tserver/tablet_server.cc
M src/kudu/tserver/tablet_server_options.cc
M src/kudu/tserver/tablet_server_options.h
M src/kudu/tserver/tablet_server_runner.cc
M src/kudu/tserver/tablet_service.cc
M src/kudu/tserver/tablet_service.h
13 files changed, 352 insertions(+), 107 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/43/16343/11
--
To view, visit http://gerrit.cloudera.org:8080/16343
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I6d7688d6fa832e606b8efc4549568fa52dfa1931
Gerrit-Change-Number: 16343
Gerrit-PatchSet: 11
Gerrit-Owner: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Tidy Bot (241)
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
[kudu-CR] KUDU-1587 part 2: reject write ops if apply queue is overloaded
Posted by "Andrew Wong (Code Review)" <ge...@cloudera.org>.
Andrew Wong has posted comments on this change. ( http://gerrit.cloudera.org:8080/16343 )
Change subject: KUDU-1587 part 2: reject write ops if apply queue is overloaded
......................................................................
Patch Set 6:
(6 comments)
http://gerrit.cloudera.org:8080/#/c/16343/6//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/16343/6//COMMIT_MSG@23
PS6, Line 23: This new behavior is not yet enabled by default, keeping the legacy
: behavior of unbounded/uncontrolled queue times as is. To enable it,
: set --tablet_apply_pool_overload_threshold_ms to something greater
: than 0 (e.g., 500).
If not in an automated test, it'd be great if you could manually run a similar workload as that described by the Jira, if anything, to validate that throttled requests are retried, examine effects on overall throughput and memory usage, etc.
http://gerrit.cloudera.org:8080/#/c/16343/6/src/kudu/tablet/tablet_replica.h
File src/kudu/tablet/tablet_replica.h:
http://gerrit.cloudera.org:8080/#/c/16343/6/src/kudu/tablet/tablet_replica.h@411
PS6, Line 411: Random rng_;
Shouldn't we be using ThreadSafeRandom, given we call this from the reactors?
http://gerrit.cloudera.org:8080/#/c/16343/6/src/kudu/tablet/tablet_replica.cc
File src/kudu/tablet/tablet_replica.cc:
http://gerrit.cloudera.org:8080/#/c/16343/6/src/kudu/tablet/tablet_replica.cc@445
PS6, Line 445: // If the apply queue is overloaded, reject the incoming operation with
: // some probability.
: MonoDelta queue_otime;
: MonoDelta threshold;
: if (apply_pool_->QueueOverloaded(&queue_otime, &threshold)) {
: auto overload_threshold_ms = threshold.ToMilliseconds();
: // The longer the queue has been in the overloaded state, the higher the
: // probability of rejection.
: auto time_factor = queue_otime.ToMilliseconds() / overload_threshold_ms + 1;
: // The heavier the request in terms of number of rows, the higher the
: // probability of rejecting it if the apply queue is overloaded.
: auto size_factor = op_state->request()->row_operations().rows().size();
: auto factor = 1 + size_factor * time_factor;
: if (!rng_.OneIn(factor)) {
: return Status::ServiceUnavailable("op apply queue is overloaded");
: }
: }
Could we instead bake this into the TabletService endpoint? We already do some throttling there -- it'd be nice to coalesce if possible, unless there's a reason for spreading it out across subsystems. Plus, that'd allow us to share a single RNG.
http://gerrit.cloudera.org:8080/#/c/16343/6/src/kudu/tserver/tablet_server-test.cc
File src/kudu/tserver/tablet_server-test.cc:
http://gerrit.cloudera.org:8080/#/c/16343/6/src/kudu/tserver/tablet_server-test.cc@4365
PS6, Line 4365: class OpApplyQueueTest : public TabletServerTestBase {
Could you also add a small test that even with a heavily throttled queue, clients will retry and eventually succeed?
http://gerrit.cloudera.org:8080/#/c/16343/6/src/kudu/tserver/tablet_server-test.cc@4375
PS6, Line 4375: intjected
nit: injected
http://gerrit.cloudera.org:8080/#/c/16343/6/src/kudu/tserver/tablet_server-test.cc@4468
PS6, Line 4468: that
nit: rather than
--
To view, visit http://gerrit.cloudera.org:8080/16343
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6d7688d6fa832e606b8efc4549568fa52dfa1931
Gerrit-Change-Number: 16343
Gerrit-PatchSet: 6
Gerrit-Owner: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Comment-Date: Mon, 24 Aug 2020 20:27:20 +0000
Gerrit-HasComments: Yes
[kudu-CR] KUDU-1587 part 2: reject write ops if apply queue is overloaded
Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Alexey Serbin has removed a vote on this change.
Change subject: KUDU-1587 part 2: reject write ops if apply queue is overloaded
......................................................................
Removed Verified-1 by Kudu Jenkins (120)
--
To view, visit http://gerrit.cloudera.org:8080/16343
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: deleteVote
Gerrit-Change-Id: I6d7688d6fa832e606b8efc4549568fa52dfa1931
Gerrit-Change-Number: 16343
Gerrit-PatchSet: 8
Gerrit-Owner: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Tidy Bot (241)
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
[kudu-CR] KUDU-1587 part 2: reject write ops if apply queue is overloaded
Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Alexey Serbin has posted comments on this change. ( http://gerrit.cloudera.org:8080/16343 )
Change subject: KUDU-1587 part 2: reject write ops if apply queue is overloaded
......................................................................
Patch Set 7:
> Build Failed
>
> http://jenkins.kudu.apache.org/job/kudu-gerrit/22277/ : FAILURE
It seems the updated behavior in PS7 has affected some heuristics -- I need to update them.
--
To view, visit http://gerrit.cloudera.org:8080/16343
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6d7688d6fa832e606b8efc4549568fa52dfa1931
Gerrit-Change-Number: 16343
Gerrit-PatchSet: 7
Gerrit-Owner: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Tidy Bot (241)
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Comment-Date: Wed, 26 Aug 2020 18:14:06 +0000
Gerrit-HasComments: No
[kudu-CR] KUDU-1587 part 2: reject write ops if apply queue is overloaded
Posted by "Grant Henke (Code Review)" <ge...@cloudera.org>.
Grant Henke has posted comments on this change. ( http://gerrit.cloudera.org:8080/16343 )
Change subject: KUDU-1587 part 2: reject write ops if apply queue is overloaded
......................................................................
Patch Set 6:
(3 comments)
http://gerrit.cloudera.org:8080/#/c/16343/6//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/16343/6//COMMIT_MSG@23
PS6, Line 23: new behavior is not yet enabled by default
What is the rational for not enabling by default? Is there some threshold_ms that is long enough that would be a good/safe default without negatively impacting any existing workloads?
http://gerrit.cloudera.org:8080/#/c/16343/6/src/kudu/tablet/tablet_replica.cc
File src/kudu/tablet/tablet_replica.cc:
http://gerrit.cloudera.org:8080/#/c/16343/6/src/kudu/tablet/tablet_replica.cc@456
PS6, Line 456: size_factor
I am not surely I fully understand why we want to give preference to smaller batches. Can you explain this a bit more?
http://gerrit.cloudera.org:8080/#/c/16343/6/src/kudu/tablet/tablet_replica.cc@445
PS6, Line 445: // If the apply queue is overloaded, reject the incoming operation with
: // some probability.
: MonoDelta queue_otime;
: MonoDelta threshold;
: if (apply_pool_->QueueOverloaded(&queue_otime, &threshold)) {
: auto overload_threshold_ms = threshold.ToMilliseconds();
: // The longer the queue has been in the overloaded state, the higher the
: // probability of rejection.
: auto time_factor = queue_otime.ToMilliseconds() / overload_threshold_ms + 1;
: // The heavier the request in terms of number of rows, the higher the
: // probability of rejecting it if the apply queue is overloaded.
: auto size_factor = op_state->request()->row_operations().rows().size();
: auto factor = 1 + size_factor * time_factor;
: if (!rng_.OneIn(factor)) {
: return Status::ServiceUnavailable("op apply queue is overloaded");
: }
: }
> Could we instead bake this into the TabletService endpoint? We already do s
+1 to this if it's not to complicated.
--
To view, visit http://gerrit.cloudera.org:8080/16343
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6d7688d6fa832e606b8efc4549568fa52dfa1931
Gerrit-Change-Number: 16343
Gerrit-PatchSet: 6
Gerrit-Owner: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Comment-Date: Tue, 25 Aug 2020 16:42:48 +0000
Gerrit-HasComments: Yes
[kudu-CR] KUDU-1587 part 2: reject write ops if apply queue is overloaded
Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Alexey Serbin has posted comments on this change. ( http://gerrit.cloudera.org:8080/16343 )
Change subject: KUDU-1587 part 2: reject write ops if apply queue is overloaded
......................................................................
Patch Set 11:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/16343/11/src/kudu/tserver/tablet_service.cc
File src/kudu/tserver/tablet_service.cc:
http://gerrit.cloudera.org:8080/#/c/16343/11/src/kudu/tserver/tablet_service.cc@1514
PS11, Line 1514: time_factor * t
> Is linear not sufficient here in your experiments? Why bother squaring here
It's not that I found that linear isn't enough. I thought that being a bit more aggressive with rejections might be a better choice, especially if queue stays overloaded for longer times. I don't have any data yet to back it up, just some intuition.
--
To view, visit http://gerrit.cloudera.org:8080/16343
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6d7688d6fa832e606b8efc4549568fa52dfa1931
Gerrit-Change-Number: 16343
Gerrit-PatchSet: 11
Gerrit-Owner: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Tidy Bot (241)
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Comment-Date: Sun, 30 Aug 2020 23:06:31 +0000
Gerrit-HasComments: Yes
[kudu-CR] KUDU-1587 part 2: reject write ops if apply queue is overloaded
Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Hello Kudu Jenkins, Andrew Wong, Todd Lipcon,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/16343
to look at the new patch set (#2).
Change subject: KUDU-1587 part 2: reject write ops if apply queue is overloaded
......................................................................
KUDU-1587 part 2: reject write ops if apply queue is overloaded
This patch implements control admission for write requests in tablet
servers based on the load status of their apply queue. With this change,
the recently introduced OpApplyQueueTest.ApplyQueueBackpressure scenario
successfully passes.
By default, the overload threshold for the tablet server's apply queue
is set to 500 milliseconds. If the queue times of the tasks in the apply
queue become higher than the specified threshold, the apply queue enters
overloaded state. When the queue is overloaded, the tablet server
rejects incoming write requests with some probability. The longer
the queue stays overloaded, the greater the probability of rejections.
In addition, the more row operations a write request has, the greater
the probability of such request to be the rejected. The apply queue
exits the overloaded state when queue times drop below the specified
threshold.
This new behavior is now enabled by default. To disable it and return
to the legacy one, set --tablet_apply_pool_overload_threshold_ms to 0.
Change-Id: I6d7688d6fa832e606b8efc4549568fa52dfa1931
---
M src/kudu/kserver/kserver.cc
M src/kudu/tablet/tablet_replica.cc
M src/kudu/tablet/tablet_replica.h
M src/kudu/tserver/tablet_server-test.cc
M src/kudu/util/threadpool-test.cc
M src/kudu/util/threadpool.cc
M src/kudu/util/threadpool.h
7 files changed, 352 insertions(+), 299 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/43/16343/2
--
To view, visit http://gerrit.cloudera.org:8080/16343
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I6d7688d6fa832e606b8efc4549568fa52dfa1931
Gerrit-Change-Number: 16343
Gerrit-PatchSet: 2
Gerrit-Owner: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
[kudu-CR] KUDU-1587 part 2: reject write ops if apply queue is overloaded
Posted by "Andrew Wong (Code Review)" <ge...@cloudera.org>.
Andrew Wong has posted comments on this change. ( http://gerrit.cloudera.org:8080/16343 )
Change subject: KUDU-1587 part 2: reject write ops if apply queue is overloaded
......................................................................
Patch Set 10: Code-Review+1
(3 comments)
http://gerrit.cloudera.org:8080/#/c/16343/10/src/kudu/integration-tests/same_tablet_concurrent_writes-itest.cc
File src/kudu/integration-tests/same_tablet_concurrent_writes-itest.cc:
http://gerrit.cloudera.org:8080/#/c/16343/10/src/kudu/integration-tests/same_tablet_concurrent_writes-itest.cc@386
PS10, Line 386: ASSERT_NE("", SetCommandLineOptionWithMode("num_inserter_threads",
Why can't we use FLAGS_* for these configurations? Or just ignore the gflags altogether and just pass these values directly to Run()?
http://gerrit.cloudera.org:8080/#/c/16343/10/src/kudu/kserver/kserver.cc
File src/kudu/kserver/kserver.cc:
http://gerrit.cloudera.org:8080/#/c/16343/10/src/kudu/kserver/kserver.cc@59
PS10, Line 59: DEFINE_uint32(tablet_apply_pool_overload_threshold_ms, 0,
: "The threshold for the queue time of the 'apply' thread pool "
: "to enter and exit overloaded state. Once the queue stalls and "
: "its queue times become longer than the specified threshold, it "
: "enters the overloaded state. Tablet server rejects incoming "
: "write requests with some probability when its apply queue is "
: "overloaded. The longer the apply queue stays overloaded, the "
: "greater the probability of the rejection. In addition, the more "
: "row operations a write request has, the greater the probablity "
: "of the rejection. The apply queue exits the overloaded state "
: "when queue times drop below the specified threshold. Set this "
: "flag to 0 to disable the behavior described above.");
: TAG_FLAG(tablet_apply_pool_overload_threshold_ms, advanced);
This mentions Tablet Servers specifically, but masters also use KuduServers. Will this be a published gflag for masters as well? Is it possible to define this in tablet_service.cc and declare it here?
http://gerrit.cloudera.org:8080/#/c/16343/6/src/kudu/tserver/tablet_server-test.cc
File src/kudu/tserver/tablet_server-test.cc:
http://gerrit.cloudera.org:8080/#/c/16343/6/src/kudu/tserver/tablet_server-test.cc@4365
PS6, Line 4365: class OpApplyQueueTest : public TabletServerTestBase {
> Could you also add a small test that even with a heavily throttled queue, c
Done
--
To view, visit http://gerrit.cloudera.org:8080/16343
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6d7688d6fa832e606b8efc4549568fa52dfa1931
Gerrit-Change-Number: 16343
Gerrit-PatchSet: 10
Gerrit-Owner: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Tidy Bot (241)
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Comment-Date: Fri, 28 Aug 2020 23:53:37 +0000
Gerrit-HasComments: Yes
[kudu-CR] KUDU-1587 part 2: reject write ops if apply queue is overloaded
Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Alexey Serbin has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/16343 )
Change subject: KUDU-1587 part 2: reject write ops if apply queue is overloaded
......................................................................
KUDU-1587 part 2: reject write ops if apply queue is overloaded
This patch implements control admission for write requests in tablet
servers based on the load status of their apply queue. With this change,
the recently introduced OpApplyQueueTest.ApplyQueueBackpressure scenario
successfully passes.
If the queue times of the tasks in the apply queue become higher than
the specified threshold, the apply queue enters overloaded state. When
the queue is overloaded, the tablet server rejects incoming write
requests with some probability. The longer the queue stays overloaded,
the greater the probability of rejections. The apply queue exits the
overloaded state when queue times drop below the specified threshold.
This new behavior is not yet enabled by default, keeping the legacy
behavior of unbounded/uncontrolled queue times as is. To enable it,
set --tablet_apply_pool_overload_threshold_ms to something greater
than 0 (e.g., 500).
Change-Id: I6d7688d6fa832e606b8efc4549568fa52dfa1931
Reviewed-on: http://gerrit.cloudera.org:8080/16343
Tested-by: Kudu Jenkins
Reviewed-by: Andrew Wong <aw...@cloudera.com>
---
M src/kudu/integration-tests/same_tablet_concurrent_writes-itest.cc
M src/kudu/kserver/kserver.cc
M src/kudu/kserver/kserver.h
A src/kudu/kserver/kserver_options.h
M src/kudu/master/master_options.h
M src/kudu/tablet/tablet_replica.cc
M src/kudu/tserver/tablet_server-test.cc
M src/kudu/tserver/tablet_server.cc
M src/kudu/tserver/tablet_server_options.cc
M src/kudu/tserver/tablet_server_options.h
M src/kudu/tserver/tablet_server_runner.cc
M src/kudu/tserver/tablet_service.cc
M src/kudu/tserver/tablet_service.h
13 files changed, 352 insertions(+), 107 deletions(-)
Approvals:
Kudu Jenkins: Verified
Andrew Wong: Looks good to me, approved
--
To view, visit http://gerrit.cloudera.org:8080/16343
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I6d7688d6fa832e606b8efc4549568fa52dfa1931
Gerrit-Change-Number: 16343
Gerrit-PatchSet: 12
Gerrit-Owner: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Tidy Bot (241)
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
[kudu-CR] KUDU-1587 part 2: reject write ops if apply queue is overloaded
Posted by "Andrew Wong (Code Review)" <ge...@cloudera.org>.
Andrew Wong has posted comments on this change. ( http://gerrit.cloudera.org:8080/16343 )
Change subject: KUDU-1587 part 2: reject write ops if apply queue is overloaded
......................................................................
Patch Set 10:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/16343/10/src/kudu/integration-tests/same_tablet_concurrent_writes-itest.cc
File src/kudu/integration-tests/same_tablet_concurrent_writes-itest.cc:
http://gerrit.cloudera.org:8080/#/c/16343/10/src/kudu/integration-tests/same_tablet_concurrent_writes-itest.cc@386
PS10, Line 386: ASSERT_NE("", SetCommandLineOptionWithMode("num_inserter_threads",
> The idea is to allow for setting of these flags from the command line, if n
Ack
http://gerrit.cloudera.org:8080/#/c/16343/10/src/kudu/kserver/kserver.cc
File src/kudu/kserver/kserver.cc:
http://gerrit.cloudera.org:8080/#/c/16343/10/src/kudu/kserver/kserver.cc@59
PS10, Line 59: DEFINE_uint32(tablet_apply_pool_overload_threshold_ms, 0,
: "The threshold for the queue time of the 'apply' thread pool "
: "to enter and exit overloaded state. Once the queue stalls and "
: "its queue times become longer than the specified threshold, it "
: "enters the overloaded state. Tablet server rejects incoming "
: "write requests with some probability when its apply queue is "
: "overloaded. The longer the apply queue stays overloaded, the "
: "greater the probability of the rejection. In addition, the more "
: "row operations a write request has, the greater the probablity "
: "of the rejection. The apply queue exits the overloaded state "
: "when queue times drop below the specified threshold. Set this "
: "flag to 0 to disable the behavior described above.");
: TAG_FLAG(tablet_apply_pool_overload_threshold_ms, advanced);
> Sure, it will be published for master as well. I guess I can update the de
If we can, I think the ideal state is making it tserver-only unless we intend on using it in the near future on masters.
--
To view, visit http://gerrit.cloudera.org:8080/16343
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6d7688d6fa832e606b8efc4549568fa52dfa1931
Gerrit-Change-Number: 16343
Gerrit-PatchSet: 10
Gerrit-Owner: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Tidy Bot (241)
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Comment-Date: Sat, 29 Aug 2020 00:56:46 +0000
Gerrit-HasComments: Yes
[kudu-CR] KUDU-1587 part 2: reject write ops if apply queue is overloaded
Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Hello Tidy Bot, Attila Bukor, Kudu Jenkins, Andrew Wong, Grant Henke, Todd Lipcon,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/16343
to look at the new patch set (#8).
Change subject: KUDU-1587 part 2: reject write ops if apply queue is overloaded
......................................................................
KUDU-1587 part 2: reject write ops if apply queue is overloaded
This patch implements control admission for write requests in tablet
servers based on the load status of their apply queue. With this change,
the recently introduced OpApplyQueueTest.ApplyQueueBackpressure scenario
successfully passes.
If the queue times of the tasks in the apply queue become higher than
the specified threshold, the apply queue enters overloaded state. When
the queue is overloaded, the tablet server rejects incoming write
requests with some probability. The longer the queue stays overloaded,
the greater the probability of rejections. The apply queue exits the
overloaded state when queue times drop below the specified threshold.
This new behavior is not yet enabled by default, keeping the legacy
behavior of unbounded/uncontrolled queue times as is. To enable it,
set --tablet_apply_pool_overload_threshold_ms to something greater
than 0 (e.g., 500).
Change-Id: I6d7688d6fa832e606b8efc4549568fa52dfa1931
---
M src/kudu/integration-tests/same_tablet_concurrent_writes-itest.cc
M src/kudu/kserver/kserver.cc
M src/kudu/tablet/tablet_replica.cc
M src/kudu/tserver/tablet_server-test.cc
M src/kudu/tserver/tablet_service.cc
M src/kudu/tserver/tablet_service.h
6 files changed, 268 insertions(+), 76 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/43/16343/8
--
To view, visit http://gerrit.cloudera.org:8080/16343
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I6d7688d6fa832e606b8efc4549568fa52dfa1931
Gerrit-Change-Number: 16343
Gerrit-PatchSet: 8
Gerrit-Owner: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Tidy Bot (241)
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
[kudu-CR] KUDU-1587 part 2: reject write ops if apply queue is overloaded
Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Alexey Serbin has posted comments on this change. ( http://gerrit.cloudera.org:8080/16343 )
Change subject: KUDU-1587 part 2: reject write ops if apply queue is overloaded
......................................................................
Patch Set 3: Verified+1
Unrelated test failure in TabletHistoryGcITest.TestUndoDeltaBlockGc:
W0815 06:52:31.092829 15439 env_posix.cc:1636] Error running callback with file /tmp/dist-test-taskNORsH5/test-tmp/tablet_history_gc-itest.0.TabletHistoryGcITest.TestUndoDeltaBlockGc.1597474349024731-15439/minicluster-data/ts-0-root/tablet-meta/3f4a937d12e240088179739f4cc746b7.kudutmp.R5Zjnq during walk: Not found: /tmp/dist-test-taskNORsH5/test-tmp/tablet_history_gc-itest.0.TabletHistoryGcITest.TestUndoDeltaBlockGc.1597474349024731-15439/minicluster-data/ts-0-root/tablet-meta/3f4a937d12e240088179739f4cc746b7.kudutmp.R5Zjnq: No such file or directory (error 2)
--
To view, visit http://gerrit.cloudera.org:8080/16343
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6d7688d6fa832e606b8efc4549568fa52dfa1931
Gerrit-Change-Number: 16343
Gerrit-PatchSet: 3
Gerrit-Owner: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Comment-Date: Sat, 15 Aug 2020 07:27:44 +0000
Gerrit-HasComments: No
[kudu-CR] KUDU-1587 part 2: reject write ops if apply queue is overloaded
Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Alexey Serbin has posted comments on this change. ( http://gerrit.cloudera.org:8080/16343 )
Change subject: KUDU-1587 part 2: reject write ops if apply queue is overloaded
......................................................................
Patch Set 8: Verified+1
unrelated test failure in TabletHistoryGcITest.TestUndoDeltaBlockGc:
W0827 06:30:46.970253 15025 env_posix.cc:1636] Error running callback with file /tmp/dist-test-taskaEC8yK/test-tmp/tablet_history_gc-itest.0.TabletHistoryGcITest.TestUndoDeltaBlockGc.1598509844768031-15025/minicluster-data/ts-0-root/tablet-meta/1f16194650dc45289614df21a6e9aba9.kudutmp.ujD28C during walk: Not found: /tmp/dist-test-taskaEC8yK/test-tmp/tablet_history_gc-itest.0.TabletHistoryGcITest.TestUndoDeltaBlockGc.1598509844768031-15025/minicluster-data/ts-0-root/tablet-meta/1f16194650dc45289614df21a6e9aba9.kudutmp.ujD28C: No such file or directory (error 2)
--
To view, visit http://gerrit.cloudera.org:8080/16343
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6d7688d6fa832e606b8efc4549568fa52dfa1931
Gerrit-Change-Number: 16343
Gerrit-PatchSet: 8
Gerrit-Owner: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Tidy Bot (241)
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Comment-Date: Thu, 27 Aug 2020 14:47:42 +0000
Gerrit-HasComments: No
[kudu-CR] KUDU-1587 part 2: reject write ops if apply queue is overloaded
Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Alexey Serbin has posted comments on this change. ( http://gerrit.cloudera.org:8080/16343 )
Change subject: KUDU-1587 part 2: reject write ops if apply queue is overloaded
......................................................................
Patch Set 10:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/16343/10/src/kudu/kserver/kserver.cc
File src/kudu/kserver/kserver.cc:
http://gerrit.cloudera.org:8080/#/c/16343/10/src/kudu/kserver/kserver.cc@59
PS10, Line 59: DEFINE_uint32(tablet_apply_pool_overload_threshold_ms, 0,
: "The threshold for the queue time of the 'apply' thread pool "
: "to enter and exit overloaded state. Once the queue stalls and "
: "its queue times become longer than the specified threshold, it "
: "enters the overloaded state. Tablet server rejects incoming "
: "write requests with some probability when its apply queue is "
: "overloaded. The longer the apply queue stays overloaded, the "
: "greater the probability of the rejection. In addition, the more "
: "row operations a write request has, the greater the probablity "
: "of the rejection. The apply queue exits the overloaded state "
: "when queue times drop below the specified threshold. Set this "
: "flag to 0 to disable the behavior described above.");
: TAG_FLAG(tablet_apply_pool_overload_threshold_ms, advanced);
> If we can, I think the ideal state is making it tserver-only unless we inte
Done
--
To view, visit http://gerrit.cloudera.org:8080/16343
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6d7688d6fa832e606b8efc4549568fa52dfa1931
Gerrit-Change-Number: 16343
Gerrit-PatchSet: 10
Gerrit-Owner: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Tidy Bot (241)
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Comment-Date: Sat, 29 Aug 2020 02:17:08 +0000
Gerrit-HasComments: Yes
[kudu-CR] KUDU-1587 part 2: reject write ops if apply queue is overloaded
Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Alexey Serbin has posted comments on this change. ( http://gerrit.cloudera.org:8080/16343 )
Change subject: KUDU-1587 part 2: reject write ops if apply queue is overloaded
......................................................................
Patch Set 10:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/16343/10/src/kudu/integration-tests/same_tablet_concurrent_writes-itest.cc
File src/kudu/integration-tests/same_tablet_concurrent_writes-itest.cc:
http://gerrit.cloudera.org:8080/#/c/16343/10/src/kudu/integration-tests/same_tablet_concurrent_writes-itest.cc@386
PS10, Line 386: ASSERT_NE("", SetCommandLineOptionWithMode("num_inserter_threads",
> Why can't we use FLAGS_* for these configurations? Or just ignore the gflag
The idea is to allow for setting of these flags from the command line, if needed (as in other performance tests).
http://gerrit.cloudera.org:8080/#/c/16343/10/src/kudu/kserver/kserver.cc
File src/kudu/kserver/kserver.cc:
http://gerrit.cloudera.org:8080/#/c/16343/10/src/kudu/kserver/kserver.cc@59
PS10, Line 59: DEFINE_uint32(tablet_apply_pool_overload_threshold_ms, 0,
: "The threshold for the queue time of the 'apply' thread pool "
: "to enter and exit overloaded state. Once the queue stalls and "
: "its queue times become longer than the specified threshold, it "
: "enters the overloaded state. Tablet server rejects incoming "
: "write requests with some probability when its apply queue is "
: "overloaded. The longer the apply queue stays overloaded, the "
: "greater the probability of the rejection. In addition, the more "
: "row operations a write request has, the greater the probablity "
: "of the rejection. The apply queue exits the overloaded state "
: "when queue times drop below the specified threshold. Set this "
: "flag to 0 to disable the behavior described above.");
: TAG_FLAG(tablet_apply_pool_overload_threshold_ms, advanced);
> This mentions Tablet Servers specifically, but masters also use KuduServers
Sure, it will be published for master as well. I guess I can update the description accordingly, removing reference to tablet servers.
Or you think the best option is to make this flag specific to tablet servers only?
--
To view, visit http://gerrit.cloudera.org:8080/16343
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6d7688d6fa832e606b8efc4549568fa52dfa1931
Gerrit-Change-Number: 16343
Gerrit-PatchSet: 10
Gerrit-Owner: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Tidy Bot (241)
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Comment-Date: Sat, 29 Aug 2020 00:37:28 +0000
Gerrit-HasComments: Yes
[kudu-CR] KUDU-1587 part 2: reject write ops if apply queue is overloaded
Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Hello Kudu Jenkins, Andrew Wong, Todd Lipcon,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/16343
to look at the new patch set (#5).
Change subject: KUDU-1587 part 2: reject write ops if apply queue is overloaded
......................................................................
KUDU-1587 part 2: reject write ops if apply queue is overloaded
This patch implements control admission for write requests in tablet
servers based on the load status of their apply queue. With this change,
the recently introduced OpApplyQueueTest.ApplyQueueBackpressure scenario
successfully passes.
If the queue times of the tasks in the apply queue become higher than
the specified threshold, the apply queue enters overloaded state. When
the queue is overloaded, the tablet server rejects incoming write
requests with some probability. The longer the queue stays overloaded,
the greater the probability of rejections. In addition, the more row
operations a write request has, the greater the probability of such
request to be the rejected. The apply queue exits the overloaded state
when queue times drop below the specified threshold.
This new behavior is not yet enabled by default, keeping the legacy
behavior of unbounded/uncontrolled queue times as is. To enable it,
set --tablet_apply_pool_overload_threshold_ms to something greater
than 0 (e.g., 500).
Change-Id: I6d7688d6fa832e606b8efc4549568fa52dfa1931
---
M src/kudu/kserver/kserver.cc
M src/kudu/tablet/tablet_replica.cc
M src/kudu/tablet/tablet_replica.h
M src/kudu/tserver/tablet_server-test.cc
4 files changed, 95 insertions(+), 38 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/43/16343/5
--
To view, visit http://gerrit.cloudera.org:8080/16343
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I6d7688d6fa832e606b8efc4549568fa52dfa1931
Gerrit-Change-Number: 16343
Gerrit-PatchSet: 5
Gerrit-Owner: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
[kudu-CR] KUDU-1587 part 2: reject write ops if apply queue is overloaded
Posted by "Andrew Wong (Code Review)" <ge...@cloudera.org>.
Andrew Wong has posted comments on this change. ( http://gerrit.cloudera.org:8080/16343 )
Change subject: KUDU-1587 part 2: reject write ops if apply queue is overloaded
......................................................................
Patch Set 11: Code-Review+2
(1 comment)
http://gerrit.cloudera.org:8080/#/c/16343/11/src/kudu/tserver/tablet_service.cc
File src/kudu/tserver/tablet_service.cc:
http://gerrit.cloudera.org:8080/#/c/16343/11/src/kudu/tserver/tablet_service.cc@1514
PS11, Line 1514: time_factor * t
> It's not that I found that linear isn't enough. I thought that being a bit
Ack
--
To view, visit http://gerrit.cloudera.org:8080/16343
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6d7688d6fa832e606b8efc4549568fa52dfa1931
Gerrit-Change-Number: 16343
Gerrit-PatchSet: 11
Gerrit-Owner: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Tidy Bot (241)
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Comment-Date: Mon, 31 Aug 2020 17:00:12 +0000
Gerrit-HasComments: Yes
[kudu-CR] KUDU-1587 part 2: reject write ops if apply queue is overloaded
Posted by "Andrew Wong (Code Review)" <ge...@cloudera.org>.
Andrew Wong has posted comments on this change. ( http://gerrit.cloudera.org:8080/16343 )
Change subject: KUDU-1587 part 2: reject write ops if apply queue is overloaded
......................................................................
Patch Set 11: Code-Review+1
(1 comment)
http://gerrit.cloudera.org:8080/#/c/16343/11/src/kudu/tserver/tablet_service.cc
File src/kudu/tserver/tablet_service.cc:
http://gerrit.cloudera.org:8080/#/c/16343/11/src/kudu/tserver/tablet_service.cc@1514
PS11, Line 1514: time_factor * t
Is linear not sufficient here in your experiments? Why bother squaring here?
--
To view, visit http://gerrit.cloudera.org:8080/16343
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6d7688d6fa832e606b8efc4549568fa52dfa1931
Gerrit-Change-Number: 16343
Gerrit-PatchSet: 11
Gerrit-Owner: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Tidy Bot (241)
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Comment-Date: Sat, 29 Aug 2020 18:36:11 +0000
Gerrit-HasComments: Yes