You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@kudu.apache.org by "Will Berkeley (Code Review)" <ge...@cloudera.org> on 2019/01/18 02:00:00 UTC

[kudu-CR] Reduce DMS flush logging 

Will Berkeley has uploaded this change for review. ( http://gerrit.cloudera.org:8080/12235


Change subject: Reduce DMS flush logging
......................................................................

Reduce DMS flush logging

Just like delta compactions, DMS flush logging can get verbose:

I0113 21:55:33.620579 67917 maintenance_manager.cc:302] P 0fe9d23b0e9c49f1af1b1095bfd395fb: Scheduling FlushDeltaMemStoresOp(e8359b48a4c1478babf8d6e14275f3d1): perf score=0.035064
I0113 21:55:33.621395 63803 delta_tracker.cc:722] T e8359b48a4c1478babf8d6e14275f3d1 P 0fe9d23b0e9c49f1af1b1095bfd395fb: Flushing 1 deltas (1759 bytes in memory) from DMS 2
I0113 21:55:33.622797 63803 delta_tracker.cc:663] T e8359b48a4c1478babf8d6e14275f3d1 P 0fe9d23b0e9c49f1af1b1095bfd395fb: Flushed delta block 0000000297638505 (1453 bytes on disk) stats: ts range=[6338172999262683136, 6338172999262683136], delete_count=[0], reinsert_count=[0], update_counts_by_col_id=[110:1,109:1,108:1,107:1,106:1,105:1,104:1,103:1,102:1,101:1,100:1,99:1,98:1,97:1,46:1,45:1,44:1,43:1,42:1,41:1,40:1,39:1,38:1,37:1,36:1,35:1,34:1,33:1,32:1,31:1,30:1,29:1,28:1,27:1,26:1,25:1,24:1,23:1,10:1,9:1,8:1,7:1,6:1,5:1,4:1,3:1,2:1,1:1,11:1,12:1,13:1,14:1,15:1,16:1,17:1,18:1,19:1,20:1,21:1,22:1,47:1,48:1,49:1,50:1,51:1,52:1,53:1,54:1,55:1,56:1,57:1,58:1,59:1,60:1,61:1,62:1,63:1,64:1,65:1,66:1,67:1,68:1,69:1,70:1,71:1,72:1,73:1,74:1,75:1,76:1,77:1,78:1,79:1,80:1,81:1,82:1,83:1,84:1,85:1,86:1,87:1,88:1,89:1,90:1,91:1,92:1,93:1,94:1,95:1,96:1]
I0113 21:55:34.064790 63804 maintenance_manager.cc:492] P 0fe9d23b0e9c49f1af1b1095bfd395fb: FlushDeltaMemStoresOp(cd897718aa9a41d6bae8ebfe5b36ca57) complete. Timing: real 3.726s       user 1.376s     sys 0.064s Metrics: {"fdatasync":3,"fdatasync_us":15451,"lbm_write_time_us":52,"lbm_writes_lt_1ms":4,"mutex_wait_us":2251275}

This reduces the logging by relegating the messages in the middle to
VLOG(1) and adding the most useful info from them as trace metrics.
Namely,

- the number of bytes written
- the numbers of deletes, reinserts, and column value updates

Change-Id: Ibe6547793f76e492926baf8b73fa2b2b067f2104
---
M src/kudu/tablet/delta_tracker.cc
1 file changed, 13 insertions(+), 5 deletions(-)



  git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/35/12235/1
-- 
To view, visit http://gerrit.cloudera.org:8080/12235
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: Ibe6547793f76e492926baf8b73fa2b2b067f2104
Gerrit-Change-Number: 12235
Gerrit-PatchSet: 1
Gerrit-Owner: Will Berkeley <wd...@gmail.com>

[kudu-CR] Reduce DMS flush logging

Posted by "Will Berkeley (Code Review)" <ge...@cloudera.org>.
Will Berkeley has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/12235 )

Change subject: Reduce DMS flush logging
......................................................................

Reduce DMS flush logging

Just like delta compactions, DMS flush logging can get verbose:

I0113 21:55:33.620579 67917 maintenance_manager.cc:302] P 0fe9d23b0e9c49f1af1b1095bfd395fb: Scheduling FlushDeltaMemStoresOp(e8359b48a4c1478babf8d6e14275f3d1): perf score=0.035064
I0113 21:55:33.621395 63803 delta_tracker.cc:722] T e8359b48a4c1478babf8d6e14275f3d1 P 0fe9d23b0e9c49f1af1b1095bfd395fb: Flushing 1 deltas (1759 bytes in memory) from DMS 2
I0113 21:55:33.622797 63803 delta_tracker.cc:663] T e8359b48a4c1478babf8d6e14275f3d1 P 0fe9d23b0e9c49f1af1b1095bfd395fb: Flushed delta block 0000000297638505 (1453 bytes on disk) stats: ts range=[6338172999262683136, 6338172999262683136], delete_count=[0], reinsert_count=[0], update_counts_by_col_id=[110:1,109:1,108:1,107:1,106:1,105:1,104:1,103:1,102:1,101:1,100:1,99:1,98:1,97:1,46:1,45:1,44:1,43:1,42:1,41:1,40:1,39:1,38:1,37:1,36:1,35:1,34:1,33:1,32:1,31:1,30:1,29:1,28:1,27:1,26:1,25:1,24:1,23:1,10:1,9:1,8:1,7:1,6:1,5:1,4:1,3:1,2:1,1:1,11:1,12:1,13:1,14:1,15:1,16:1,17:1,18:1,19:1,20:1,21:1,22:1,47:1,48:1,49:1,50:1,51:1,52:1,53:1,54:1,55:1,56:1,57:1,58:1,59:1,60:1,61:1,62:1,63:1,64:1,65:1,66:1,67:1,68:1,69:1,70:1,71:1,72:1,73:1,74:1,75:1,76:1,77:1,78:1,79:1,80:1,81:1,82:1,83:1,84:1,85:1,86:1,87:1,88:1,89:1,90:1,91:1,92:1,93:1,94:1,95:1,96:1]
I0113 21:55:34.064790 63804 maintenance_manager.cc:492] P 0fe9d23b0e9c49f1af1b1095bfd395fb: FlushDeltaMemStoresOp(cd897718aa9a41d6bae8ebfe5b36ca57) complete. Timing: real 3.726s       user 1.376s     sys 0.064s Metrics: {"fdatasync":3,"fdatasync_us":15451,"lbm_write_time_us":52,"lbm_writes_lt_1ms":4,"mutex_wait_us":2251275}

This reduces the logging by relegating the messages in the middle to
VLOG(1) and adding the most useful info from them as trace metrics.
Namely,

- the number of bytes written
- the numbers of deletes, reinserts, and column value updates

Change-Id: Ibe6547793f76e492926baf8b73fa2b2b067f2104
Reviewed-on: http://gerrit.cloudera.org:8080/12235
Tested-by: Will Berkeley <wd...@gmail.com>
Reviewed-by: Adar Dembo <ad...@cloudera.com>
---
M src/kudu/tablet/delta_tracker.cc
1 file changed, 13 insertions(+), 5 deletions(-)

Approvals:
  Will Berkeley: Verified
  Adar Dembo: Looks good to me, approved

-- 
To view, visit http://gerrit.cloudera.org:8080/12235
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: Ibe6547793f76e492926baf8b73fa2b2b067f2104
Gerrit-Change-Number: 12235
Gerrit-PatchSet: 2
Gerrit-Owner: Will Berkeley <wd...@gmail.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Mitch Barnett <mb...@cloudera.com>
Gerrit-Reviewer: Will Berkeley <wd...@gmail.com>

[kudu-CR] Reduce DMS flush logging

Posted by "Will Berkeley (Code Review)" <ge...@cloudera.org>.
Will Berkeley has removed a vote on this change.

Change subject: Reduce DMS flush logging
......................................................................


Removed Verified-1 by Kudu Jenkins (120)
-- 
To view, visit http://gerrit.cloudera.org:8080/12235
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: deleteVote
Gerrit-Change-Id: Ibe6547793f76e492926baf8b73fa2b2b067f2104
Gerrit-Change-Number: 12235
Gerrit-PatchSet: 1
Gerrit-Owner: Will Berkeley <wd...@gmail.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Mitch Barnett <mb...@cloudera.com>

[kudu-CR] Reduce DMS flush logging

Posted by "Will Berkeley (Code Review)" <ge...@cloudera.org>.
Will Berkeley has posted comments on this change. ( http://gerrit.cloudera.org:8080/12235 )

Change subject: Reduce DMS flush logging
......................................................................


Patch Set 1: Verified+1

Unrelated flake.


-- 
To view, visit http://gerrit.cloudera.org:8080/12235
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ibe6547793f76e492926baf8b73fa2b2b067f2104
Gerrit-Change-Number: 12235
Gerrit-PatchSet: 1
Gerrit-Owner: Will Berkeley <wd...@gmail.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Mitch Barnett <mb...@cloudera.com>
Gerrit-Reviewer: Will Berkeley <wd...@gmail.com>
Gerrit-Comment-Date: Fri, 18 Jan 2019 04:59:00 +0000
Gerrit-HasComments: No

[kudu-CR] Reduce DMS flush logging

Posted by "Adar Dembo (Code Review)" <ge...@cloudera.org>.
Adar Dembo has posted comments on this change. ( http://gerrit.cloudera.org:8080/12235 )

Change subject: Reduce DMS flush logging
......................................................................


Patch Set 1: Code-Review+2


-- 
To view, visit http://gerrit.cloudera.org:8080/12235
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ibe6547793f76e492926baf8b73fa2b2b067f2104
Gerrit-Change-Number: 12235
Gerrit-PatchSet: 1
Gerrit-Owner: Will Berkeley <wd...@gmail.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Mitch Barnett <mb...@cloudera.com>
Gerrit-Reviewer: Will Berkeley <wd...@gmail.com>
Gerrit-Comment-Date: Fri, 18 Jan 2019 23:58:07 +0000
Gerrit-HasComments: No