You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@kudu.apache.org by "Hao Hao (Code Review)" <ge...@cloudera.org> on 2018/05/03 23:44:27 UTC
[kudu-CR] KUDU-2191: HMS inconsistent metadata fix tool
Hao Hao has uploaded this change for review. ( http://gerrit.cloudera.org:8080/10303
Change subject: KUDU-2191: HMS inconsistent metadata fix tool
......................................................................
KUDU-2191: HMS inconsistent metadata fix tool
This commit introduces a CLI tool that fixes inconsistent metadata for
Hive Metastore integration. It corrects the table name based on the
users input. Ant it fixes the master addresses property, column name
and type based on the values in Kudu. Test case is added using external
mini cluster to ensure the tool works as expected.
Change-Id: I63c694b5d9877cfbd218139f2e9ecf05fd69c997
---
M src/kudu/tools/kudu-tool-test.cc
M src/kudu/tools/tool_action_hms.cc
M src/kudu/tools/tool_test_util.cc
M src/kudu/tools/tool_test_util.h
4 files changed, 232 insertions(+), 44 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/03/10303/1
--
To view, visit http://gerrit.cloudera.org:8080/10303
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I63c694b5d9877cfbd218139f2e9ecf05fd69c997
Gerrit-Change-Number: 10303
Gerrit-PatchSet: 1
Gerrit-Owner: Hao Hao <ha...@cloudera.com>
[kudu-CR] KUDU-2191: HMS inconsistent metadata fix tool
Posted by "Hao Hao (Code Review)" <ge...@cloudera.org>.
Hao Hao has posted comments on this change. ( http://gerrit.cloudera.org:8080/10303 )
Change subject: KUDU-2191: HMS inconsistent metadata fix tool
......................................................................
Patch Set 6:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/10303/4/src/kudu/tools/tool_action_hms.cc
File src/kudu/tools/tool_action_hms.cc:
http://gerrit.cloudera.org:8080/#/c/10303/4/src/kudu/tools/tool_action_hms.cc@502
PS4, Line 502: // Fail the fix process as encountered unfixable errors.
> Renaming a table to the same name returns an error, so does this indicate a
Hmm, I did a quick test in master-hms-itest to rename a table with the same name. And it works fine without returning an error. You think this is unexpected?
--
To view, visit http://gerrit.cloudera.org:8080/10303
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I63c694b5d9877cfbd218139f2e9ecf05fd69c997
Gerrit-Change-Number: 10303
Gerrit-PatchSet: 6
Gerrit-Owner: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Dan Burkert <da...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Tidy Bot
Gerrit-Comment-Date: Mon, 25 Jun 2018 18:49:19 +0000
Gerrit-HasComments: Yes
[kudu-CR] KUDU-2191: HMS inconsistent metadata fix tool
Posted by "Hao Hao (Code Review)" <ge...@cloudera.org>.
Hao Hao has posted comments on this change. ( http://gerrit.cloudera.org:8080/10303 )
Change subject: KUDU-2191: HMS inconsistent metadata fix tool
......................................................................
Patch Set 6:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/10303/4/src/kudu/tools/tool_action_hms.cc
File src/kudu/tools/tool_action_hms.cc:
http://gerrit.cloudera.org:8080/#/c/10303/4/src/kudu/tools/tool_action_hms.cc@502
PS4, Line 502: // Fail the fix process as encountered unfixable errors.
> Can you not just alter the HMS entry directly through the HMS catalog? If
Done
http://gerrit.cloudera.org:8080/#/c/10303/5/src/kudu/tools/tool_action_hms.cc
File src/kudu/tools/tool_action_hms.cc:
http://gerrit.cloudera.org:8080/#/c/10303/5/src/kudu/tools/tool_action_hms.cc@73
PS5, Line 73: Hive
> s/hive/Hive
Done
--
To view, visit http://gerrit.cloudera.org:8080/10303
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I63c694b5d9877cfbd218139f2e9ecf05fd69c997
Gerrit-Change-Number: 10303
Gerrit-PatchSet: 6
Gerrit-Owner: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Dan Burkert <da...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Tidy Bot
Gerrit-Comment-Date: Sat, 23 Jun 2018 00:50:50 +0000
Gerrit-HasComments: Yes
[kudu-CR] KUDU-2191: HMS inconsistent metadata fix tool
Posted by "Hao Hao (Code Review)" <ge...@cloudera.org>.
Hao Hao has posted comments on this change. ( http://gerrit.cloudera.org:8080/10303 )
Change subject: KUDU-2191: HMS inconsistent metadata fix tool
......................................................................
Patch Set 5:
(7 comments)
http://gerrit.cloudera.org:8080/#/c/10303/4//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/10303/4//COMMIT_MSG@11
PS4, Line 11: And
> And
Done
http://gerrit.cloudera.org:8080/#/c/10303/2/src/kudu/tools/tool_action_hms.cc
File src/kudu/tools/tool_action_hms.cc:
http://gerrit.cloudera.org:8080/#/c/10303/2/src/kudu/tools/tool_action_hms.cc@237
PS2, Line 237:
> nit: present
Done
http://gerrit.cloudera.org:8080/#/c/10303/4/src/kudu/tools/tool_action_hms.cc
File src/kudu/tools/tool_action_hms.cc:
http://gerrit.cloudera.org:8080/#/c/10303/4/src/kudu/tools/tool_action_hms.cc@142
PS4, Line 142: install
> 'Warn'
Done
http://gerrit.cloudera.org:8080/#/c/10303/4/src/kudu/tools/tool_action_hms.cc@448
PS4, Line 448: }
> This is hard to follow, could the constraint instead be encoded in the type
Done
http://gerrit.cloudera.org:8080/#/c/10303/4/src/kudu/tools/tool_action_hms.cc@474
PS4, Line 474:
> Picking an HMS entry to rename arbitrarily makes me pretty nervous, since H
Discussed this offline, agreed it is not safe to automated fix for situation that there are multiple tables in the HMS sharing the same table ID. It would be better to have the admin manually fix it since the admin may have better idea of how to fix it. And this should be a very rare case to happen.
For the case that the table in Kudu and the HMS have different names, we default it to the one in HMS since we consider HMS is the source of truth of table names.
http://gerrit.cloudera.org:8080/#/c/10303/4/src/kudu/tools/tool_action_hms.cc@485
PS4, Line 485: TablesMap unsynced_tables_map;
> Shouldn't this be a 'kudu only' alter?
Yeah, AlterKuduTable() is only altering table in Kudu. Maybe I should rename it AlterKuduTableOnly() to be more clear.
http://gerrit.cloudera.org:8080/#/c/10303/4/src/kudu/tools/tool_action_hms.cc@502
PS4, Line 502:
> Why does this do a rename to the same table name?
It is to trigger the correction of the master addr, column name, etc in the HMS.
--
To view, visit http://gerrit.cloudera.org:8080/10303
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I63c694b5d9877cfbd218139f2e9ecf05fd69c997
Gerrit-Change-Number: 10303
Gerrit-PatchSet: 5
Gerrit-Owner: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Dan Burkert <da...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Tidy Bot
Gerrit-Comment-Date: Fri, 22 Jun 2018 01:29:37 +0000
Gerrit-HasComments: Yes
[kudu-CR] KUDU-2191: HMS inconsistent metadata fix tool
Posted by "Hao Hao (Code Review)" <ge...@cloudera.org>.
Hello Tidy Bot, Dan Burkert, Kudu Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/10303
to look at the new patch set (#8).
Change subject: KUDU-2191: HMS inconsistent metadata fix tool
......................................................................
KUDU-2191: HMS inconsistent metadata fix tool
This commit introduces a CLI tool that fixes inconsistent metadata for
Hive Metastore integration. It corrects the table name based on the
users input. And it fixes the master addresses property, column name
and type based on the values in Kudu. Test case is added using external
mini cluster to ensure the tool works as expected.
Change-Id: I63c694b5d9877cfbd218139f2e9ecf05fd69c997
---
M src/kudu/client/client.h
M src/kudu/tools/kudu-tool-test.cc
M src/kudu/tools/tool_action_hms.cc
3 files changed, 253 insertions(+), 27 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/03/10303/8
--
To view, visit http://gerrit.cloudera.org:8080/10303
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I63c694b5d9877cfbd218139f2e9ecf05fd69c997
Gerrit-Change-Number: 10303
Gerrit-PatchSet: 8
Gerrit-Owner: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Dan Burkert <da...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Tidy Bot
[kudu-CR] KUDU-2191: HMS inconsistent metadata fix tool
Posted by "Dan Burkert (Code Review)" <ge...@cloudera.org>.
Dan Burkert has posted comments on this change. ( http://gerrit.cloudera.org:8080/10303 )
Change subject: KUDU-2191: HMS inconsistent metadata fix tool
......................................................................
Patch Set 9: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/10303
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I63c694b5d9877cfbd218139f2e9ecf05fd69c997
Gerrit-Change-Number: 10303
Gerrit-PatchSet: 9
Gerrit-Owner: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Dan Burkert <da...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Tidy Bot
Gerrit-Comment-Date: Tue, 26 Jun 2018 23:14:41 +0000
Gerrit-HasComments: No
[kudu-CR] KUDU-2191: HMS inconsistent metadata fix tool
Posted by "Hao Hao (Code Review)" <ge...@cloudera.org>.
Hello Tidy Bot, Dan Burkert, Kudu Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/10303
to look at the new patch set (#5).
Change subject: KUDU-2191: HMS inconsistent metadata fix tool
......................................................................
KUDU-2191: HMS inconsistent metadata fix tool
This commit introduces a CLI tool that fixes inconsistent metadata for
Hive Metastore integration. It corrects the table name based on the
users input. And it fixes the master addresses property, column name
and type based on the values in Kudu. Test case is added using external
mini cluster to ensure the tool works as expected.
Change-Id: I63c694b5d9877cfbd218139f2e9ecf05fd69c997
---
M src/kudu/client/client.h
M src/kudu/tools/kudu-tool-test.cc
M src/kudu/tools/tool_action_hms.cc
3 files changed, 235 insertions(+), 23 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/03/10303/5
--
To view, visit http://gerrit.cloudera.org:8080/10303
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I63c694b5d9877cfbd218139f2e9ecf05fd69c997
Gerrit-Change-Number: 10303
Gerrit-PatchSet: 5
Gerrit-Owner: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Dan Burkert <da...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Tidy Bot
[kudu-CR] KUDU-2191: HMS inconsistent metadata fix tool
Posted by "Hao Hao (Code Review)" <ge...@cloudera.org>.
Hao Hao has removed a vote on this change.
Change subject: KUDU-2191: HMS inconsistent metadata fix tool
......................................................................
Removed Verified-1 by Kudu Jenkins (120)
--
To view, visit http://gerrit.cloudera.org:8080/10303
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: deleteVote
Gerrit-Change-Id: I63c694b5d9877cfbd218139f2e9ecf05fd69c997
Gerrit-Change-Number: 10303
Gerrit-PatchSet: 2
Gerrit-Owner: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Dan Burkert <da...@apache.org>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Tidy Bot
[kudu-CR] KUDU-2191: HMS inconsistent metadata fix tool
Posted by "Hao Hao (Code Review)" <ge...@cloudera.org>.
Hello Tidy Bot, Dan Burkert, Kudu Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/10303
to look at the new patch set (#3).
Change subject: KUDU-2191: HMS inconsistent metadata fix tool
......................................................................
KUDU-2191: HMS inconsistent metadata fix tool
This commit introduces a CLI tool that fixes inconsistent metadata for
Hive Metastore integration. It corrects the table name based on the
users input. Ant it fixes the master addresses property, column name
and type based on the values in Kudu. Test case is added using external
mini cluster to ensure the tool works as expected.
Change-Id: I63c694b5d9877cfbd218139f2e9ecf05fd69c997
---
M src/kudu/tools/kudu-tool-test.cc
M src/kudu/tools/tool_action_hms.cc
2 files changed, 191 insertions(+), 17 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/03/10303/3
--
To view, visit http://gerrit.cloudera.org:8080/10303
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I63c694b5d9877cfbd218139f2e9ecf05fd69c997
Gerrit-Change-Number: 10303
Gerrit-PatchSet: 3
Gerrit-Owner: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Dan Burkert <da...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Tidy Bot
[kudu-CR] KUDU-2191: HMS inconsistent metadata fix tool
Posted by "Hao Hao (Code Review)" <ge...@cloudera.org>.
Hao Hao has posted comments on this change. ( http://gerrit.cloudera.org:8080/10303 )
Change subject: KUDU-2191: HMS inconsistent metadata fix tool
......................................................................
Patch Set 9:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/10303/8/src/kudu/tools/tool_action_hms.cc
File src/kudu/tools/tool_action_hms.cc:
http://gerrit.cloudera.org:8080/#/c/10303/8/src/kudu/tools/tool_action_hms.cc@483
PS8, Line 483: cout << Substitute("Renaming Kudu table $0 [id=$1] to $2 to match the Hive "
: "Metastore catalog.", kudu_table_name, table_id, hms_table_name)
: << endl;
> Are these lines redundant with the 'Renaming...' message?
Done
http://gerrit.cloudera.org:8080/#/c/10303/8/src/kudu/tools/tool_action_hms.cc@486
PS8, Line 486: ent,
> add a space between the table name and opening brace here and below. This
Done
--
To view, visit http://gerrit.cloudera.org:8080/10303
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I63c694b5d9877cfbd218139f2e9ecf05fd69c997
Gerrit-Change-Number: 10303
Gerrit-PatchSet: 9
Gerrit-Owner: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Dan Burkert <da...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Tidy Bot
Gerrit-Comment-Date: Tue, 26 Jun 2018 21:24:23 +0000
Gerrit-HasComments: Yes
[kudu-CR] KUDU-2191: HMS inconsistent metadata fix tool
Posted by "Dan Burkert (Code Review)" <ge...@cloudera.org>.
Dan Burkert has posted comments on this change. ( http://gerrit.cloudera.org:8080/10303 )
Change subject: KUDU-2191: HMS inconsistent metadata fix tool
......................................................................
Patch Set 6:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/10303/4/src/kudu/tools/tool_action_hms.cc
File src/kudu/tools/tool_action_hms.cc:
http://gerrit.cloudera.org:8080/#/c/10303/4/src/kudu/tools/tool_action_hms.cc@502
PS4, Line 502: // Fail the fix process as encountered unfixable errors.
> Hmm, I did a quick test in master-hms-itest to rename a table with the same
Yep, I think it might. I tested the same thing, but without the HMS integration turned on. I'll fix it since it's almost certainly an issue with the notification log listener.
--
To view, visit http://gerrit.cloudera.org:8080/10303
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I63c694b5d9877cfbd218139f2e9ecf05fd69c997
Gerrit-Change-Number: 10303
Gerrit-PatchSet: 6
Gerrit-Owner: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Dan Burkert <da...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Tidy Bot
Gerrit-Comment-Date: Mon, 25 Jun 2018 18:53:42 +0000
Gerrit-HasComments: Yes
[kudu-CR] KUDU-2191: HMS inconsistent metadata fix tool
Posted by "Hao Hao (Code Review)" <ge...@cloudera.org>.
Hello Tidy Bot, Dan Burkert, Kudu Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/10303
to look at the new patch set (#9).
Change subject: KUDU-2191: HMS inconsistent metadata fix tool
......................................................................
KUDU-2191: HMS inconsistent metadata fix tool
This commit introduces a CLI tool that fixes inconsistent metadata for
Hive Metastore integration. It corrects the table name based on the
users input. And it fixes the master addresses property, column name
and type based on the values in Kudu. Test case is added using external
mini cluster to ensure the tool works as expected.
Change-Id: I63c694b5d9877cfbd218139f2e9ecf05fd69c997
---
M src/kudu/client/client.h
M src/kudu/tools/kudu-tool-test.cc
M src/kudu/tools/tool_action_hms.cc
3 files changed, 250 insertions(+), 27 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/03/10303/9
--
To view, visit http://gerrit.cloudera.org:8080/10303
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I63c694b5d9877cfbd218139f2e9ecf05fd69c997
Gerrit-Change-Number: 10303
Gerrit-PatchSet: 9
Gerrit-Owner: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Dan Burkert <da...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Tidy Bot
[kudu-CR] KUDU-2191: HMS inconsistent metadata fix tool
Posted by "Hao Hao (Code Review)" <ge...@cloudera.org>.
Hello Tidy Bot, Dan Burkert, Kudu Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/10303
to look at the new patch set (#4).
Change subject: KUDU-2191: HMS inconsistent metadata fix tool
......................................................................
KUDU-2191: HMS inconsistent metadata fix tool
This commit introduces a CLI tool that fixes inconsistent metadata for
Hive Metastore integration. It corrects the table name based on the
users input. Ant it fixes the master addresses property, column name
and type based on the values in Kudu. Test case is added using external
mini cluster to ensure the tool works as expected.
Change-Id: I63c694b5d9877cfbd218139f2e9ecf05fd69c997
---
M src/kudu/tools/kudu-tool-test.cc
M src/kudu/tools/tool_action_hms.cc
2 files changed, 211 insertions(+), 18 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/03/10303/4
--
To view, visit http://gerrit.cloudera.org:8080/10303
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I63c694b5d9877cfbd218139f2e9ecf05fd69c997
Gerrit-Change-Number: 10303
Gerrit-PatchSet: 4
Gerrit-Owner: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Dan Burkert <da...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Tidy Bot
[kudu-CR] KUDU-2191: HMS inconsistent metadata fix tool
Posted by "Dan Burkert (Code Review)" <ge...@cloudera.org>.
Dan Burkert has posted comments on this change. ( http://gerrit.cloudera.org:8080/10303 )
Change subject: KUDU-2191: HMS inconsistent metadata fix tool
......................................................................
Patch Set 6:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/10303/4/src/kudu/tools/tool_action_hms.cc
File src/kudu/tools/tool_action_hms.cc:
http://gerrit.cloudera.org:8080/#/c/10303/4/src/kudu/tools/tool_action_hms.cc@502
PS4, Line 502: // Fail the fix process as encountered unfixable errors.
> Done
Renaming a table to the same name returns an error, so does this indicate a missing test?
--
To view, visit http://gerrit.cloudera.org:8080/10303
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I63c694b5d9877cfbd218139f2e9ecf05fd69c997
Gerrit-Change-Number: 10303
Gerrit-PatchSet: 6
Gerrit-Owner: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Dan Burkert <da...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Tidy Bot
Gerrit-Comment-Date: Mon, 25 Jun 2018 18:02:41 +0000
Gerrit-HasComments: Yes
[kudu-CR] KUDU-2191: HMS inconsistent metadata fix tool
Posted by "Dan Burkert (Code Review)" <ge...@cloudera.org>.
Dan Burkert has posted comments on this change. ( http://gerrit.cloudera.org:8080/10303 )
Change subject: KUDU-2191: HMS inconsistent metadata fix tool
......................................................................
Patch Set 8:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/10303/8/src/kudu/tools/tool_action_hms.cc
File src/kudu/tools/tool_action_hms.cc:
http://gerrit.cloudera.org:8080/#/c/10303/8/src/kudu/tools/tool_action_hms.cc@483
PS8, Line 483: cout << "The table name in Kudu: " << kudu_table_name << endl;
: cout << "The database name, table name in the Hive Metastore: "
: << hms_table_name << endl;
Are these lines redundant with the 'Renaming...' message?
http://gerrit.cloudera.org:8080/#/c/10303/8/src/kudu/tools/tool_action_hms.cc@486
PS8, Line 486: 0[id
add a space between the table name and opening brace here and below. This format is somewhat standardized by the catalog manager. https://github.com/apache/kudu/blob/master/src/kudu/master/catalog_manager.cc#L4966-L4969
--
To view, visit http://gerrit.cloudera.org:8080/10303
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I63c694b5d9877cfbd218139f2e9ecf05fd69c997
Gerrit-Change-Number: 10303
Gerrit-PatchSet: 8
Gerrit-Owner: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Dan Burkert <da...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Tidy Bot
Gerrit-Comment-Date: Tue, 26 Jun 2018 21:18:36 +0000
Gerrit-HasComments: Yes
[kudu-CR] KUDU-2191: HMS inconsistent metadata fix tool
Posted by "Hao Hao (Code Review)" <ge...@cloudera.org>.
Hello Tidy Bot, Dan Burkert, Kudu Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/10303
to look at the new patch set (#6).
Change subject: KUDU-2191: HMS inconsistent metadata fix tool
......................................................................
KUDU-2191: HMS inconsistent metadata fix tool
This commit introduces a CLI tool that fixes inconsistent metadata for
Hive Metastore integration. It corrects the table name based on the
users input. And it fixes the master addresses property, column name
and type based on the values in Kudu. Test case is added using external
mini cluster to ensure the tool works as expected.
Change-Id: I63c694b5d9877cfbd218139f2e9ecf05fd69c997
---
M src/kudu/client/client.h
M src/kudu/tools/kudu-tool-test.cc
M src/kudu/tools/tool_action_hms.cc
3 files changed, 246 insertions(+), 26 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/03/10303/6
--
To view, visit http://gerrit.cloudera.org:8080/10303
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I63c694b5d9877cfbd218139f2e9ecf05fd69c997
Gerrit-Change-Number: 10303
Gerrit-PatchSet: 6
Gerrit-Owner: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Dan Burkert <da...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Tidy Bot
[kudu-CR] KUDU-2191: HMS inconsistent metadata fix tool
Posted by "Hao Hao (Code Review)" <ge...@cloudera.org>.
Hello Tidy Bot, Dan Burkert, Kudu Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/10303
to look at the new patch set (#7).
Change subject: KUDU-2191: HMS inconsistent metadata fix tool
......................................................................
KUDU-2191: HMS inconsistent metadata fix tool
This commit introduces a CLI tool that fixes inconsistent metadata for
Hive Metastore integration. It corrects the table name based on the
users input. And it fixes the master addresses property, column name
and type based on the values in Kudu. Test case is added using external
mini cluster to ensure the tool works as expected.
Change-Id: I63c694b5d9877cfbd218139f2e9ecf05fd69c997
---
M src/kudu/client/client.h
M src/kudu/tools/kudu-tool-test.cc
M src/kudu/tools/tool_action_hms.cc
3 files changed, 251 insertions(+), 26 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/03/10303/7
--
To view, visit http://gerrit.cloudera.org:8080/10303
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I63c694b5d9877cfbd218139f2e9ecf05fd69c997
Gerrit-Change-Number: 10303
Gerrit-PatchSet: 7
Gerrit-Owner: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Dan Burkert <da...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Tidy Bot
[kudu-CR] KUDU-2191: HMS inconsistent metadata fix tool
Posted by "Hao Hao (Code Review)" <ge...@cloudera.org>.
Hao Hao has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/10303 )
Change subject: KUDU-2191: HMS inconsistent metadata fix tool
......................................................................
KUDU-2191: HMS inconsistent metadata fix tool
This commit introduces a CLI tool that fixes inconsistent metadata for
Hive Metastore integration. It corrects the table name based on the
users input. And it fixes the master addresses property, column name
and type based on the values in Kudu. Test case is added using external
mini cluster to ensure the tool works as expected.
Change-Id: I63c694b5d9877cfbd218139f2e9ecf05fd69c997
Reviewed-on: http://gerrit.cloudera.org:8080/10303
Tested-by: Kudu Jenkins
Reviewed-by: Dan Burkert <da...@apache.org>
---
M src/kudu/client/client.h
M src/kudu/tools/kudu-tool-test.cc
M src/kudu/tools/tool_action_hms.cc
3 files changed, 250 insertions(+), 27 deletions(-)
Approvals:
Kudu Jenkins: Verified
Dan Burkert: Looks good to me, approved
--
To view, visit http://gerrit.cloudera.org:8080/10303
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I63c694b5d9877cfbd218139f2e9ecf05fd69c997
Gerrit-Change-Number: 10303
Gerrit-PatchSet: 10
Gerrit-Owner: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Dan Burkert <da...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Tidy Bot
[kudu-CR] KUDU-2191: HMS inconsistent metadata fix tool
Posted by "Hao Hao (Code Review)" <ge...@cloudera.org>.
Hao Hao has posted comments on this change. ( http://gerrit.cloudera.org:8080/10303 )
Change subject: KUDU-2191: HMS inconsistent metadata fix tool
......................................................................
Patch Set 8:
(5 comments)
http://gerrit.cloudera.org:8080/#/c/10303/7/src/kudu/tools/tool_action_hms.cc
File src/kudu/tools/tool_action_hms.cc:
http://gerrit.cloudera.org:8080/#/c/10303/7/src/kudu/tools/tool_action_hms.cc@404
PS7, Line 404: FilterOrphanedTables(unsynced_tables_map);
> Perhaps the function should be called FilterOrphanedTables to make it more
Done
http://gerrit.cloudera.org:8080/#/c/10303/7/src/kudu/tools/tool_action_hms.cc@450
PS7, Line 450: const KuduTable& kudu_table = *entry.second.first;
> Can kudu_table and hms_tables be const KuduTable& and const vector<hive::Ta
Done
http://gerrit.cloudera.org:8080/#/c/10303/7/src/kudu/tools/tool_action_hms.cc@483
PS7, Line 483: cout << "The table name in Kudu: " << kudu_table_name << endl;
> These would probably be simpler without the Substitute calls, e.g.
Done
http://gerrit.cloudera.org:8080/#/c/10303/7/src/kudu/tools/tool_action_hms.cc@486
PS7, Line 486: cout << Substitute("Renaming Kudu table $0[id=$1] to $2 to match the Hive "
> "Default to..." isn't very descriptive of what is about to occur. I'd sugg
Done
http://gerrit.cloudera.org:8080/#/c/10303/7/src/kudu/tools/tool_action_hms.cc@494
PS7, Line 494: // information in Kudu.
> nit: "Updating metadata of table foo.bar [id=abc123] in the Hive Metastore
Done
--
To view, visit http://gerrit.cloudera.org:8080/10303
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I63c694b5d9877cfbd218139f2e9ecf05fd69c997
Gerrit-Change-Number: 10303
Gerrit-PatchSet: 8
Gerrit-Owner: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Dan Burkert <da...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Tidy Bot
Gerrit-Comment-Date: Tue, 26 Jun 2018 20:39:29 +0000
Gerrit-HasComments: Yes
[kudu-CR] KUDU-2191: HMS inconsistent metadata fix tool
Posted by "Dan Burkert (Code Review)" <ge...@cloudera.org>.
Dan Burkert has posted comments on this change. ( http://gerrit.cloudera.org:8080/10303 )
Change subject: KUDU-2191: HMS inconsistent metadata fix tool
......................................................................
Patch Set 7:
(5 comments)
http://gerrit.cloudera.org:8080/#/c/10303/7/src/kudu/tools/tool_action_hms.cc
File src/kudu/tools/tool_action_hms.cc:
http://gerrit.cloudera.org:8080/#/c/10303/7/src/kudu/tools/tool_action_hms.cc@404
PS7, Line 404: FilterUnsyncedTables(unsynced_tables_map);
Perhaps the function should be called FilterOrphanedTables to make it more self documenting.
http://gerrit.cloudera.org:8080/#/c/10303/7/src/kudu/tools/tool_action_hms.cc@450
PS7, Line 450: shared_ptr<KuduTable> kudu_table = entry.second.first;
Can kudu_table and hms_tables be const KuduTable& and const vector<hive::Table>& respectively?
http://gerrit.cloudera.org:8080/#/c/10303/7/src/kudu/tools/tool_action_hms.cc@483
PS7, Line 483: cout << Substitute("The table name in Kudu: $0 ", kudu_table_name) << endl;
These would probably be simpler without the Substitute calls, e.g.
cout << "The table name in Kudu: " << kudu_table_name << endl;
http://gerrit.cloudera.org:8080/#/c/10303/7/src/kudu/tools/tool_action_hms.cc@486
PS7, Line 486: cout << Substitute("Default to table name ($0) in the Hive Metastore.",
"Default to..." isn't very descriptive of what is about to occur. I'd suggest something like "Renaming Kudu table foo.bar [id=abc123] to baz.quux to match the Hive Metastore catalog"
http://gerrit.cloudera.org:8080/#/c/10303/7/src/kudu/tools/tool_action_hms.cc@494
PS7, Line 494: cout << "Update master addresses property and column name and type "
nit: "Updating metadata of table foo.bar [id=abc123] in the Hive Metastore catalog"
--
To view, visit http://gerrit.cloudera.org:8080/10303
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I63c694b5d9877cfbd218139f2e9ecf05fd69c997
Gerrit-Change-Number: 10303
Gerrit-PatchSet: 7
Gerrit-Owner: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Dan Burkert <da...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Tidy Bot
Gerrit-Comment-Date: Tue, 26 Jun 2018 17:11:26 +0000
Gerrit-HasComments: Yes
[kudu-CR] KUDU-2191: HMS inconsistent metadata fix tool
Posted by "Dan Burkert (Code Review)" <ge...@cloudera.org>.
Dan Burkert has posted comments on this change. ( http://gerrit.cloudera.org:8080/10303 )
Change subject: KUDU-2191: HMS inconsistent metadata fix tool
......................................................................
Patch Set 5:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/10303/4/src/kudu/tools/tool_action_hms.cc
File src/kudu/tools/tool_action_hms.cc:
http://gerrit.cloudera.org:8080/#/c/10303/4/src/kudu/tools/tool_action_hms.cc@502
PS4, Line 502:
> It is to trigger the correction of the master addr, column name, etc in the
Can you not just alter the HMS entry directly through the HMS catalog? If not maybe add a doc note because this is a bit confusing.
http://gerrit.cloudera.org:8080/#/c/10303/5/src/kudu/tools/tool_action_hms.cc
File src/kudu/tools/tool_action_hms.cc:
http://gerrit.cloudera.org:8080/#/c/10303/5/src/kudu/tools/tool_action_hms.cc@73
PS5, Line 73: hive
s/hive/Hive
--
To view, visit http://gerrit.cloudera.org:8080/10303
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I63c694b5d9877cfbd218139f2e9ecf05fd69c997
Gerrit-Change-Number: 10303
Gerrit-PatchSet: 5
Gerrit-Owner: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Dan Burkert <da...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Tidy Bot
Gerrit-Comment-Date: Fri, 22 Jun 2018 19:02:00 +0000
Gerrit-HasComments: Yes
[kudu-CR] KUDU-2191: HMS inconsistent metadata fix tool
Posted by "Hao Hao (Code Review)" <ge...@cloudera.org>.
Hao Hao has posted comments on this change. ( http://gerrit.cloudera.org:8080/10303 )
Change subject: KUDU-2191: HMS inconsistent metadata fix tool
......................................................................
Patch Set 2: Verified+1
Tests failed because of clock synchronization issue.
--
To view, visit http://gerrit.cloudera.org:8080/10303
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I63c694b5d9877cfbd218139f2e9ecf05fd69c997
Gerrit-Change-Number: 10303
Gerrit-PatchSet: 2
Gerrit-Owner: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Dan Burkert <da...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Tidy Bot
Gerrit-Comment-Date: Fri, 04 May 2018 02:06:05 +0000
Gerrit-HasComments: No
[kudu-CR] KUDU-2191: HMS inconsistent metadata fix tool
Posted by "Dan Burkert (Code Review)" <ge...@cloudera.org>.
Dan Burkert has posted comments on this change. ( http://gerrit.cloudera.org:8080/10303 )
Change subject: KUDU-2191: HMS inconsistent metadata fix tool
......................................................................
Patch Set 4:
(7 comments)
http://gerrit.cloudera.org:8080/#/c/10303/4//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/10303/4//COMMIT_MSG@11
PS4, Line 11: Ant
And
http://gerrit.cloudera.org:8080/#/c/10303/2/src/kudu/tools/tool_action_hms.cc
File src/kudu/tools/tool_action_hms.cc:
http://gerrit.cloudera.org:8080/#/c/10303/2/src/kudu/tools/tool_action_hms.cc@237
PS2, Line 237:
nit: present
http://gerrit.cloudera.org:8080/#/c/10303/4/src/kudu/tools/tool_action_hms.cc
File src/kudu/tools/tool_action_hms.cc:
http://gerrit.cloudera.org:8080/#/c/10303/4/src/kudu/tools/tool_action_hms.cc@142
PS4, Line 142: Warning
'Warn'
http://gerrit.cloudera.org:8080/#/c/10303/4/src/kudu/tools/tool_action_hms.cc@448
PS4, Line 448: // There must be one Kudu table in the entry.
This is hard to follow, could the constraint instead be encoded in the type itself? E.g. change tables_map to be an
unordered_map<string, pair<shared_ptr<KuduTable>, vector<hive::Table>>>
http://gerrit.cloudera.org:8080/#/c/10303/4/src/kudu/tools/tool_action_hms.cc@474
PS4, Line 474: // 3. Correct the table name.
Picking an HMS entry to rename arbitrarily makes me pretty nervous, since HMS entries come associated with other metadata like sentry privileges. I'm not sure we should have an automated fix for this situation, since it's not clear what the right course is. It should also be an extraordinary situation, since we try very hard in the master not to allow it to happen.
http://gerrit.cloudera.org:8080/#/c/10303/4/src/kudu/tools/tool_action_hms.cc@485
PS4, Line 485: RETURN_NOT_OK(AlterKuduTable(kudu_client, kudu_table_name, new_table_name));
Shouldn't this be a 'kudu only' alter?
http://gerrit.cloudera.org:8080/#/c/10303/4/src/kudu/tools/tool_action_hms.cc@502
PS4, Line 502: RETURN_NOT_OK(alterer->RenameTo(kudu_table_name)
Why does this do a rename to the same table name?
--
To view, visit http://gerrit.cloudera.org:8080/10303
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I63c694b5d9877cfbd218139f2e9ecf05fd69c997
Gerrit-Change-Number: 10303
Gerrit-PatchSet: 4
Gerrit-Owner: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Dan Burkert <da...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Tidy Bot
Gerrit-Comment-Date: Wed, 20 Jun 2018 18:06:09 +0000
Gerrit-HasComments: Yes