You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@kudu.apache.org by "Todd Lipcon (Code Review)" <ge...@cloudera.org> on 2016/08/29 19:53:18 UTC
[kudu-CR] transaction tracker: back-off when logging in-flight transactions
Hello Mike Percy,
I'd like you to do a code review. Please visit
http://gerrit.cloudera.org:8080/4159
to review the following change.
Change subject: transaction_tracker: back-off when logging in-flight transactions
......................................................................
transaction_tracker: back-off when logging in-flight transactions
On a test cluster pushed into overload, I ended up with thousands of
in-flight transactions which were only being drained at the rate of one
or two per second. Dumping the entire list of transactions once a second
ended up flooding the glog with many MB per second work of transaction
strings.
This changes the dumping to back-off exponentially.
Change-Id: I7d1a2563dd57c59ae8c130a5b26966f4964048de
---
M src/kudu/tablet/transactions/transaction_tracker.cc
1 file changed, 16 insertions(+), 13 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/59/4159/1
--
To view, visit http://gerrit.cloudera.org:8080/4159
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: I7d1a2563dd57c59ae8c130a5b26966f4964048de
Gerrit-PatchSet: 1
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Mike Percy <mp...@apache.org>
[kudu-CR] transaction tracker: back-off when logging in-flight transactions
Posted by "David Ribeiro Alves (Code Review)" <ge...@cloudera.org>.
David Ribeiro Alves has posted comments on this change.
Change subject: transaction_tracker: back-off when logging in-flight transactions
......................................................................
Patch Set 1:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/4159/1/src/kudu/tablet/transactions/transaction_tracker.cc
File src/kudu/tablet/transactions/transaction_tracker.cc:
Line 207: MonoTime next_log_time = start_time + MonoDelta::FromSeconds(1);
can you pull the log backoff logic into a generic logging helper, seems like it would be useful elsewhere
--
To view, visit http://gerrit.cloudera.org:8080/4159
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: I7d1a2563dd57c59ae8c130a5b26966f4964048de
Gerrit-PatchSet: 1
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: David Ribeiro Alves <dr...@apache.org>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Mike Percy <mp...@apache.org>
Gerrit-HasComments: Yes
[kudu-CR] transaction tracker: back-off when logging in-flight transactions
Posted by "Todd Lipcon (Code Review)" <ge...@cloudera.org>.
Todd Lipcon has submitted this change and it was merged.
Change subject: transaction_tracker: back-off when logging in-flight transactions
......................................................................
transaction_tracker: back-off when logging in-flight transactions
On a test cluster pushed into overload, I ended up with thousands of
in-flight transactions which were only being drained at the rate of one
or two per second. Dumping the entire list of transactions once a second
ended up flooding the glog with many MB per second work of transaction
strings.
This changes the dumping to back-off exponentially.
Change-Id: I7d1a2563dd57c59ae8c130a5b26966f4964048de
Reviewed-on: http://gerrit.cloudera.org:8080/4159
Tested-by: Kudu Jenkins
Reviewed-by: Alexey Serbin <as...@cloudera.com>
Reviewed-by: David Ribeiro Alves <dr...@apache.org>
---
M src/kudu/tablet/transactions/transaction_tracker.cc
1 file changed, 16 insertions(+), 13 deletions(-)
Approvals:
David Ribeiro Alves: Looks good to me, approved
Alexey Serbin: Looks good to me, but someone else must approve
Kudu Jenkins: Verified
--
To view, visit http://gerrit.cloudera.org:8080/4159
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: merged
Gerrit-Change-Id: I7d1a2563dd57c59ae8c130a5b26966f4964048de
Gerrit-PatchSet: 2
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: David Ribeiro Alves <dr...@apache.org>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Mike Percy <mp...@apache.org>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
[kudu-CR] transaction tracker: back-off when logging in-flight transactions
Posted by "David Ribeiro Alves (Code Review)" <ge...@cloudera.org>.
David Ribeiro Alves has posted comments on this change.
Change subject: transaction_tracker: back-off when logging in-flight transactions
......................................................................
Patch Set 1: Code-Review+2
(1 comment)
http://gerrit.cloudera.org:8080/#/c/4159/1/src/kudu/tablet/transactions/transaction_tracker.cc
File src/kudu/tablet/transactions/transaction_tracker.cc:
Line 207: MonoTime next_log_time = start_time + MonoDelta::FromSeconds(1);
> not sure how to do that in a nice way since we're also using the times for
fair enough
--
To view, visit http://gerrit.cloudera.org:8080/4159
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: I7d1a2563dd57c59ae8c130a5b26966f4964048de
Gerrit-PatchSet: 1
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: David Ribeiro Alves <dr...@apache.org>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Mike Percy <mp...@apache.org>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-HasComments: Yes
[kudu-CR] transaction tracker: back-off when logging in-flight transactions
Posted by "Kudu Jenkins (Code Review)" <ge...@cloudera.org>.
Kudu Jenkins has posted comments on this change.
Change subject: transaction_tracker: back-off when logging in-flight transactions
......................................................................
Patch Set 1:
Build Started http://104.196.14.100/job/kudu-gerrit/3133/
--
To view, visit http://gerrit.cloudera.org:8080/4159
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: I7d1a2563dd57c59ae8c130a5b26966f4964048de
Gerrit-PatchSet: 1
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Mike Percy <mp...@apache.org>
Gerrit-HasComments: No
[kudu-CR] transaction tracker: back-off when logging in-flight transactions
Posted by "Todd Lipcon (Code Review)" <ge...@cloudera.org>.
Todd Lipcon has posted comments on this change.
Change subject: transaction_tracker: back-off when logging in-flight transactions
......................................................................
Patch Set 1:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/4159/1/src/kudu/tablet/transactions/transaction_tracker.cc
File src/kudu/tablet/transactions/transaction_tracker.cc:
Line 207: MonoTime next_log_time = start_time + MonoDelta::FromSeconds(1);
> can you pull the log backoff logic into a generic logging helper, seems lik
not sure how to do that in a nice way since we're also using the times for timeout calculation, etc
--
To view, visit http://gerrit.cloudera.org:8080/4159
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: I7d1a2563dd57c59ae8c130a5b26966f4964048de
Gerrit-PatchSet: 1
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: David Ribeiro Alves <dr...@apache.org>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Mike Percy <mp...@apache.org>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-HasComments: Yes
[kudu-CR] transaction tracker: back-off when logging in-flight transactions
Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Alexey Serbin has posted comments on this change.
Change subject: transaction_tracker: back-off when logging in-flight transactions
......................................................................
Patch Set 1: Code-Review+1
--
To view, visit http://gerrit.cloudera.org:8080/4159
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: I7d1a2563dd57c59ae8c130a5b26966f4964048de
Gerrit-PatchSet: 1
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Mike Percy <mp...@apache.org>
Gerrit-HasComments: No