You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@kudu.apache.org by "Fengling Wang (Code Review)" <ge...@cloudera.org> on 2018/06/12 16:38:03 UTC
[kudu-CR] Add a tool to dump master data in a readable format
Fengling Wang has uploaded this change for review. ( http://gerrit.cloudera.org:8080/10697
Change subject: Add a tool to dump master data in a readable format
......................................................................
Add a tool to dump master data in a readable format
e.g.
$ kudu fs dump master -fs_data_dirs=/tmp/kudu/data/master/0/0 -fs_wal_dir=/tmp/kudu/wal/master/0
$ ...
(int8 entry_type=1, string entry_id="4baf0e9a04644f228e2e0dbb79a9563e", string metadata="\n-loadgen_auto_80714f4ada0e4e468c927c3e369170c6\020\000\032J\n\023\010\000\022\003key\030\007 \001(\000@\000H\000P\000\n\027\010\001\022\007int_val\030\005 \000(\001@\000H\000P\000\n\032\010\002\022\nstring_val\030\010 \000(\001@\000H\000P\000(\0010\002@\003J\020\n\010\n\002\010\000\020\010\030\000\022\004\n\002\010\000")
(int8 entry_type=1, string entry_id="796e861f1e994494a9e75ad951557d3e", string metadata="\n-loadgen_auto_3ef746f1f6d0476aad1ef2b173b4e349\020\000\032J\n\023\010\000\022\003key\030\007 \001(\000@\000H\000P\000\n\027\010\001\022\007int_val\030\005 \000(\001@\000H\000P\000\n\032\010\002\022\nstring_val\030\010 \000(\001@\000H\000P\000(\0010\002@\003J\020\n\010\n\002\010\000\020\010\030\000\022\004\n\002\010\000")
...
Change-Id: Ie210f93e1d8bc106f0cb3b1cd48f89808da95f08
---
M src/kudu/tools/kudu-tool-test.cc
M src/kudu/tools/tool_action_fs.cc
2 files changed, 86 insertions(+), 0 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/97/10697/1
--
To view, visit http://gerrit.cloudera.org:8080/10697
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: Ie210f93e1d8bc106f0cb3b1cd48f89808da95f08
Gerrit-Change-Number: 10697
Gerrit-PatchSet: 1
Gerrit-Owner: Fengling Wang <fw...@cloudera.com>
[kudu-CR] Add a tool to dump master data in a readable format
Posted by "Fengling Wang (Code Review)" <ge...@cloudera.org>.
Fengling Wang has abandoned this change. ( http://gerrit.cloudera.org:8080/10697 )
Change subject: Add a tool to dump master data in a readable format
......................................................................
Abandoned
--
To view, visit http://gerrit.cloudera.org:8080/10697
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: abandon
Gerrit-Change-Id: Ie210f93e1d8bc106f0cb3b1cd48f89808da95f08
Gerrit-Change-Number: 10697
Gerrit-PatchSet: 1
Gerrit-Owner: Fengling Wang <fw...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Fengling Wang <fw...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Tidy Bot
Gerrit-Reviewer: Will Berkeley <wd...@gmail.com>
[kudu-CR] Add a tool to dump master data in a readable format
Posted by "Adar Dembo (Code Review)" <ge...@cloudera.org>.
Adar Dembo has posted comments on this change. ( http://gerrit.cloudera.org:8080/10697 )
Change subject: Add a tool to dump master data in a readable format
......................................................................
Patch Set 1:
> To dump the master data (which is essentially the sys log table), I basically grep and parse the first 3 blocks of CFiles in master data dir (I'm not sure this is always the pattern?). I tested the code manually and it works fine. But the current approach is probably not a robust way to solve the problem.
>
> Hoping for suggestions. Thanks!
I agree that this isn't going to be robust. The master data may not yet be flushed and thus may still be in the WAL. Or there may be more than three blocks, and the other blocks contain deltas that should be applied on top of the first three. Plus, I think a goal is to dump the PB-ified part of the schema (the third column) in a human-readable way, no?
Anyway, I think you're going to need to create a SysCatalogTable. That'll bring with it a Tablet, which will take care of providing a unified view of the master data on top of the WAL and data blocks. It'll also take care of interpreting the master schema for you. You _might_ be able to get by with just a Tablet and rely on the tool to interpret the master schema, but I think the SysCatalogTable will yield more dividends in the future.
It's a little tricky because:
1) You want to make sure that whatever object you're creating doesn't do any writing; this should be a read-only operation.
2) I don't know if it's possible to create a SysCatalogTable without starting up a Raft consensus state machine, which you definitely don't want to do.
Will took a crack at something like this recently; see if he has any ideas on how to best proceed.
--
To view, visit http://gerrit.cloudera.org:8080/10697
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie210f93e1d8bc106f0cb3b1cd48f89808da95f08
Gerrit-Change-Number: 10697
Gerrit-PatchSet: 1
Gerrit-Owner: Fengling Wang <fw...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Fengling Wang <fw...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Tidy Bot
Gerrit-Reviewer: Will Berkeley <wd...@gmail.com>
Gerrit-Comment-Date: Tue, 12 Jun 2018 18:11:00 +0000
Gerrit-HasComments: No
[kudu-CR] Add a tool to dump master data in a readable format
Posted by "Fengling Wang (Code Review)" <ge...@cloudera.org>.
Fengling Wang has posted comments on this change. ( http://gerrit.cloudera.org:8080/10697 )
Change subject: Add a tool to dump master data in a readable format
......................................................................
Patch Set 1:
(3 comments)
To dump the master data (which is essentially the sys log table), I basically grep and parse the first 3 blocks of CFiles in master data dir (I'm not sure this is always the pattern?). I tested the code manually and it works fine. But the current approach is probably not a robust way to solve the problem.
Hoping for suggestions. Thanks!
http://gerrit.cloudera.org:8080/#/c/10697/1//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/10697/1//COMMIT_MSG@7
PS1, Line 7: A
I'll add KUDU-2308 later.
http://gerrit.cloudera.org:8080/#/c/10697/1/src/kudu/tools/kudu-tool-test.cc
File src/kudu/tools/kudu-tool-test.cc:
http://gerrit.cloudera.org:8080/#/c/10697/1/src/kudu/tools/kudu-tool-test.cc@913
PS1, Line 913: TEST_F
Sorry for the incomplete code here since the test is not passing and I'm a bit stuck on why it's not passing. A suggestion would be appreciated.
http://gerrit.cloudera.org:8080/#/c/10697/1/src/kudu/tools/tool_action_fs.cc
File src/kudu/tools/tool_action_fs.cc:
http://gerrit.cloudera.org:8080/#/c/10697/1/src/kudu/tools/tool_action_fs.cc@282
PS1, Line 282: Status DumpMaster(const RunnerContext& context) {
> warning: parameter 'context' is unused [misc-unused-parameters]
I'll fix this later.
--
To view, visit http://gerrit.cloudera.org:8080/10697
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie210f93e1d8bc106f0cb3b1cd48f89808da95f08
Gerrit-Change-Number: 10697
Gerrit-PatchSet: 1
Gerrit-Owner: Fengling Wang <fw...@cloudera.com>
Gerrit-Reviewer: Fengling Wang <fw...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Tidy Bot
Gerrit-Comment-Date: Tue, 12 Jun 2018 16:56:33 +0000
Gerrit-HasComments: Yes
[kudu-CR] Add a tool to dump master data in a readable format
Posted by "Fengling Wang (Code Review)" <ge...@cloudera.org>.
Fengling Wang has posted comments on this change. ( http://gerrit.cloudera.org:8080/10697 )
Change subject: Add a tool to dump master data in a readable format
......................................................................
Patch Set 1:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/10697/1/src/kudu/tools/tool_action_fs.cc
File src/kudu/tools/tool_action_fs.cc:
http://gerrit.cloudera.org:8080/#/c/10697/1/src/kudu/tools/tool_action_fs.cc@317
PS1, Line 317:
opps.
--
To view, visit http://gerrit.cloudera.org:8080/10697
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie210f93e1d8bc106f0cb3b1cd48f89808da95f08
Gerrit-Change-Number: 10697
Gerrit-PatchSet: 1
Gerrit-Owner: Fengling Wang <fw...@cloudera.com>
Gerrit-Reviewer: Fengling Wang <fw...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Tidy Bot
Gerrit-Reviewer: Will Berkeley <wd...@gmail.com>
Gerrit-Comment-Date: Tue, 12 Jun 2018 17:07:16 +0000
Gerrit-HasComments: Yes