You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@kudu.apache.org by "Andrew Wong (Code Review)" <ge...@cloudera.org> on 2020/01/08 02:16:43 UTC
[kudu-CR] consensus: throttle warnings when quiescing server
Andrew Wong has uploaded this change for review. ( http://gerrit.cloudera.org:8080/14989
Change subject: consensus: throttle warnings when quiescing server
......................................................................
consensus: throttle warnings when quiescing server
When an election fails to start because the server is quiescing, it will
log a warning. When there are many replicas on a server, this logging
can become very verbose.
This throttles such logs to happen at most once per second. Here's a
snippet of the logs from TestQuiescingServerDoesntTriggerElections:
Before:
W0107 17:55:46.943423 55670 raft_consensus.cc:655] T d1bc343b18e348ada0a4fadb835d8cc1 P f0fd8e85d7a843e6862dfd846ca47351: failed to trigger leader election: Illegal state: leader elections are disabled
W0107 17:55:46.947376 55670 raft_consensus.cc:655] T 4fb4aec4dbb94dc9b2af086b761d2632 P f0fd8e85d7a843e6862dfd846ca47351: failed to trigger leader election: Illegal state: leader elections are disabled
W0107 17:55:46.949579 55670 raft_consensus.cc:655] T 49bf5837ae0c4d95bf1a176a856c9f37 P f0fd8e85d7a843e6862dfd846ca47351: failed to trigger leader election: Illegal state: leader elections are disabled
W0107 17:55:46.992861 55670 raft_consensus.cc:655] T 842d307a87f846efa00bbf0a537fe812 P f0fd8e85d7a843e6862dfd846ca47351: failed to trigger leader election: Illegal state: leader elections are disabled
After:
W0107 18:04:42.348387 64502 raft_consensus.cc:655] T 346a0544fe95444eb3becd158f40135c P 66f6cf3b32b94f7e9eacd1f012a4386f: failed to trigger leader election: Illegal state: leader elections are disabled [suppressed 21 similar messages]
While it would be nice to retain logs for every replica for grep-ability, I
think the improved readability of the logs justifies the loss.
Change-Id: I01e3c88c2bd4c53d095b9b5a061439231dcfc9e9
---
M src/kudu/consensus/raft_consensus.cc
M src/kudu/util/logging.h
2 files changed, 13 insertions(+), 6 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/89/14989/1
--
To view, visit http://gerrit.cloudera.org:8080/14989
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I01e3c88c2bd4c53d095b9b5a061439231dcfc9e9
Gerrit-Change-Number: 14989
Gerrit-PatchSet: 1
Gerrit-Owner: Andrew Wong <aw...@cloudera.com>
[kudu-CR] consensus: throttle warnings when quiescing server
Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Alexey Serbin has posted comments on this change. ( http://gerrit.cloudera.org:8080/14989 )
Change subject: consensus: throttle warnings when quiescing server
......................................................................
Patch Set 2: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/14989
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I01e3c88c2bd4c53d095b9b5a061439231dcfc9e9
Gerrit-Change-Number: 14989
Gerrit-PatchSet: 2
Gerrit-Owner: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Tidy Bot (241)
Gerrit-Comment-Date: Wed, 08 Jan 2020 04:13:08 +0000
Gerrit-HasComments: No
[kudu-CR] consensus: throttle warnings when quiescing server
Posted by "Andrew Wong (Code Review)" <ge...@cloudera.org>.
Andrew Wong has posted comments on this change. ( http://gerrit.cloudera.org:8080/14989 )
Change subject: consensus: throttle warnings when quiescing server
......................................................................
Patch Set 2: Verified+1
Failure seems unrelated, though I'm not sure what caused it yet:
19:22:00 [ 4%] Building CXX object src/kudu/gutil/CMakeFiles/gutil_exported.dir/strings/stringpiece.cc.o
19:22:00 [ 4%] Building CXX object src/kudu/gutil/CMakeFiles/gutil.dir/threading/thread_collision_warner.cc.o
19:22:00 [ 4%] Building CXX object src/kudu/hms/CMakeFiles/hms_thrift.dir/fb303_types.cpp.o
19:22:00 [ 4%] Building C object src/kudu/gutil/CMakeFiles/gutil.dir/utf/rune.c.o
19:22:00 [ 4%] Building CXX object src/kudu/hms/CMakeFiles/hms_thrift.dir/fb303_constants.cpp.o
19:22:00 [ 4%] Building CXX object src/kudu/gutil/CMakeFiles/gutil.dir/walltime.cc.o
19:22:00 javac: file not found: /home/jenkins-slave/workspace/kudu-master/1/build/release/bin/CMakeFiles/hms-plugin.dir/java_sources (No such file or directory)
19:22:00 make[2]: *** [bin/CMakeFiles/hms-plugin.dir/java_compiled_hms-plugin] Error 3
19:22:00 make[1]: *** [src/kudu/hms/CMakeFiles/hms-plugin.dir/all] Error 2
19:22:00 make[1]: *** Waiting for unfinished jobs....
--
To view, visit http://gerrit.cloudera.org:8080/14989
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I01e3c88c2bd4c53d095b9b5a061439231dcfc9e9
Gerrit-Change-Number: 14989
Gerrit-PatchSet: 2
Gerrit-Owner: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Tidy Bot (241)
Gerrit-Comment-Date: Wed, 08 Jan 2020 04:34:05 +0000
Gerrit-HasComments: No
[kudu-CR] consensus: throttle warnings when quiescing server
Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Alexey Serbin has posted comments on this change. ( http://gerrit.cloudera.org:8080/14989 )
Change subject: consensus: throttle warnings when quiescing server
......................................................................
Patch Set 1:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/14989/1/src/kudu/consensus/raft_consensus.cc
File src/kudu/consensus/raft_consensus.cc:
http://gerrit.cloudera.org:8080/#/c/14989/1/src/kudu/consensus/raft_consensus.cc@383
PS1, Line 383: 1
Given it can take long time to drain a tablet server of pending scan requests, etc., maybe bump the interval up to 10 seconds?
Also, consider introducing a file-scope constant for the interval duration and using it elsewhere in this file for suppressing similar messages.
--
To view, visit http://gerrit.cloudera.org:8080/14989
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I01e3c88c2bd4c53d095b9b5a061439231dcfc9e9
Gerrit-Change-Number: 14989
Gerrit-PatchSet: 1
Gerrit-Owner: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Tidy Bot (241)
Gerrit-Comment-Date: Wed, 08 Jan 2020 03:05:51 +0000
Gerrit-HasComments: Yes
[kudu-CR] consensus: throttle warnings when quiescing server
Posted by "Andrew Wong (Code Review)" <ge...@cloudera.org>.
Hello Tidy Bot, Alexey Serbin, Kudu Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/14989
to look at the new patch set (#2).
Change subject: consensus: throttle warnings when quiescing server
......................................................................
consensus: throttle warnings when quiescing server
When an election fails to start because the server is quiescing, it will
log a warning. When there are many replicas on a server, this logging
can become very verbose.
This throttles such logs to happen at most once per ten seconds. Here's
a snippet of the logs from TestQuiescingServerDoesntTriggerElections:
Before:
W0107 17:55:46.943423 55670 raft_consensus.cc:655] T d1bc343b18e348ada0a4fadb835d8cc1 P f0fd8e85d7a843e6862dfd846ca47351: failed to trigger leader election: Illegal state: leader elections are disabled
W0107 17:55:46.947376 55670 raft_consensus.cc:655] T 4fb4aec4dbb94dc9b2af086b761d2632 P f0fd8e85d7a843e6862dfd846ca47351: failed to trigger leader election: Illegal state: leader elections are disabled
W0107 17:55:46.949579 55670 raft_consensus.cc:655] T 49bf5837ae0c4d95bf1a176a856c9f37 P f0fd8e85d7a843e6862dfd846ca47351: failed to trigger leader election: Illegal state: leader elections are disabled
W0107 17:55:46.992861 55670 raft_consensus.cc:655] T 842d307a87f846efa00bbf0a537fe812 P f0fd8e85d7a843e6862dfd846ca47351: failed to trigger leader election: Illegal state: leader elections are disabled
After:
W0107 19:16:47.518998 87519 raft_consensus.cc:655] T 1e0aed8879c643d6a8684f6da729a1a3 P 14907cae807b4560b4b8417ba5f24393: failed to trigger leader election: Illegal state: leader elections are disabled
(in After, the test didn't last 10 seconds)
While it would be nice to retain logs for every replica for grep-ability, I
think the improved readability of the logs justifies the loss.
Change-Id: I01e3c88c2bd4c53d095b9b5a061439231dcfc9e9
---
M src/kudu/consensus/raft_consensus.cc
M src/kudu/util/logging.h
2 files changed, 13 insertions(+), 6 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/89/14989/2
--
To view, visit http://gerrit.cloudera.org:8080/14989
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I01e3c88c2bd4c53d095b9b5a061439231dcfc9e9
Gerrit-Change-Number: 14989
Gerrit-PatchSet: 2
Gerrit-Owner: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Tidy Bot (241)
[kudu-CR] consensus: throttle warnings when quiescing server
Posted by "Andrew Wong (Code Review)" <ge...@cloudera.org>.
Andrew Wong has posted comments on this change. ( http://gerrit.cloudera.org:8080/14989 )
Change subject: consensus: throttle warnings when quiescing server
......................................................................
Patch Set 2:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/14989/1/src/kudu/consensus/raft_consensus.cc
File src/kudu/consensus/raft_consensus.cc:
http://gerrit.cloudera.org:8080/#/c/14989/1/src/kudu/consensus/raft_consensus.cc@383
PS1, Line 383: 1
> Given it can take long time to drain a tablet server of pending scan reques
Done
--
To view, visit http://gerrit.cloudera.org:8080/14989
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I01e3c88c2bd4c53d095b9b5a061439231dcfc9e9
Gerrit-Change-Number: 14989
Gerrit-PatchSet: 2
Gerrit-Owner: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Tidy Bot (241)
Gerrit-Comment-Date: Wed, 08 Jan 2020 03:19:58 +0000
Gerrit-HasComments: Yes
[kudu-CR] consensus: throttle warnings when quiescing server
Posted by "Andrew Wong (Code Review)" <ge...@cloudera.org>.
Andrew Wong has removed a vote on this change.
Change subject: consensus: throttle warnings when quiescing server
......................................................................
Removed Verified-1 by Kudu Jenkins (120)
--
To view, visit http://gerrit.cloudera.org:8080/14989
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: deleteVote
Gerrit-Change-Id: I01e3c88c2bd4c53d095b9b5a061439231dcfc9e9
Gerrit-Change-Number: 14989
Gerrit-PatchSet: 2
Gerrit-Owner: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Tidy Bot (241)
[kudu-CR] consensus: throttle warnings when quiescing server
Posted by "Andrew Wong (Code Review)" <ge...@cloudera.org>.
Andrew Wong has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/14989 )
Change subject: consensus: throttle warnings when quiescing server
......................................................................
consensus: throttle warnings when quiescing server
When an election fails to start because the server is quiescing, it will
log a warning. When there are many replicas on a server, this logging
can become very verbose.
This throttles such logs to happen at most once per ten seconds. Here's
a snippet of the logs from TestQuiescingServerDoesntTriggerElections:
Before:
W0107 17:55:46.943423 55670 raft_consensus.cc:655] T d1bc343b18e348ada0a4fadb835d8cc1 P f0fd8e85d7a843e6862dfd846ca47351: failed to trigger leader election: Illegal state: leader elections are disabled
W0107 17:55:46.947376 55670 raft_consensus.cc:655] T 4fb4aec4dbb94dc9b2af086b761d2632 P f0fd8e85d7a843e6862dfd846ca47351: failed to trigger leader election: Illegal state: leader elections are disabled
W0107 17:55:46.949579 55670 raft_consensus.cc:655] T 49bf5837ae0c4d95bf1a176a856c9f37 P f0fd8e85d7a843e6862dfd846ca47351: failed to trigger leader election: Illegal state: leader elections are disabled
W0107 17:55:46.992861 55670 raft_consensus.cc:655] T 842d307a87f846efa00bbf0a537fe812 P f0fd8e85d7a843e6862dfd846ca47351: failed to trigger leader election: Illegal state: leader elections are disabled
After:
W0107 19:16:47.518998 87519 raft_consensus.cc:655] T 1e0aed8879c643d6a8684f6da729a1a3 P 14907cae807b4560b4b8417ba5f24393: failed to trigger leader election: Illegal state: leader elections are disabled
(in After, the test didn't last 10 seconds)
While it would be nice to retain logs for every replica for grep-ability, I
think the improved readability of the logs justifies the loss.
Change-Id: I01e3c88c2bd4c53d095b9b5a061439231dcfc9e9
Reviewed-on: http://gerrit.cloudera.org:8080/14989
Reviewed-by: Alexey Serbin <as...@cloudera.com>
Tested-by: Andrew Wong <aw...@cloudera.com>
---
M src/kudu/consensus/raft_consensus.cc
M src/kudu/util/logging.h
2 files changed, 13 insertions(+), 6 deletions(-)
Approvals:
Alexey Serbin: Looks good to me, approved
Andrew Wong: Verified
--
To view, visit http://gerrit.cloudera.org:8080/14989
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I01e3c88c2bd4c53d095b9b5a061439231dcfc9e9
Gerrit-Change-Number: 14989
Gerrit-PatchSet: 3
Gerrit-Owner: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Tidy Bot (241)