You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@kudu.apache.org by "Abhishek Chennaka (Code Review)" <ge...@cloudera.org> on 2022/02/28 16:30:08 UTC
[kudu-CR] [tools] KUDU-3333 Table Counts in kudu hms dryrun
Abhishek Chennaka has uploaded this change for review. ( http://gerrit.cloudera.org:8080/18280
Change subject: [tools] KUDU-3333 Table Counts in kudu hms dryrun
......................................................................
[tools] KUDU-3333 Table Counts in kudu hms dryrun
In cases where the user running the Kudu CLI tool, kudu hms
fix, doesn't have permissions from Ranger/Sentry to access
the tables, these tables would be treated as non-existant tables
in Kudu. In such scenarios, there might be situations where the
tables could be dropped from HMS inspite of them being present
in Kudu when run with -drop_orphan_tables flag.
This patch adds additional logging which reports the total
table counts from HMS and Kudu master catalogs and warns the
user if there are no tables in Kudu when kudu hms fix command is
run.
Change-Id: Idf26141d2a3fd6cbb7249b3492fc6a50a0c0aa2d
---
M src/kudu/tools/tool_action_hms.cc
1 file changed, 19 insertions(+), 4 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/80/18280/1
--
To view, visit http://gerrit.cloudera.org:8080/18280
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: Idf26141d2a3fd6cbb7249b3492fc6a50a0c0aa2d
Gerrit-Change-Number: 18280
Gerrit-PatchSet: 1
Gerrit-Owner: Abhishek Chennaka <ac...@cloudera.com>
[kudu-CR] [tools] KUDU-3333 Table Counts in kudu hms dryrun
Posted by "Andrew Wong (Code Review)" <ge...@cloudera.org>.
Andrew Wong has posted comments on this change. ( http://gerrit.cloudera.org:8080/18280 )
Change subject: [tools] KUDU-3333 Table Counts in kudu hms dryrun
......................................................................
Patch Set 1: Code-Review+1
We chatted about this a bit offline. Would be nice to add snippets of the before and after output to the commit message, since we're not adding a test in this patch.
--
To view, visit http://gerrit.cloudera.org:8080/18280
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Idf26141d2a3fd6cbb7249b3492fc6a50a0c0aa2d
Gerrit-Change-Number: 18280
Gerrit-PatchSet: 1
Gerrit-Owner: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Mon, 14 Mar 2022 22:16:48 +0000
Gerrit-HasComments: No
[kudu-CR] [tools] KUDU-3333 Include Table Counts in kudu hms Dryrun
Posted by "Andrew Wong (Code Review)" <ge...@cloudera.org>.
Andrew Wong has posted comments on this change. ( http://gerrit.cloudera.org:8080/18280 )
Change subject: [tools] KUDU-3333 Include Table Counts in kudu hms Dryrun
......................................................................
Patch Set 3: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/18280
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Idf26141d2a3fd6cbb7249b3492fc6a50a0c0aa2d
Gerrit-Change-Number: 18280
Gerrit-PatchSet: 3
Gerrit-Owner: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Thu, 17 Mar 2022 20:10:47 +0000
Gerrit-HasComments: No
[kudu-CR] [tools] KUDU-3333 Include Table Counts in kudu hms Dryrun
Posted by "Abhishek Chennaka (Code Review)" <ge...@cloudera.org>.
Hello Kudu Jenkins, Andrew Wong,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/18280
to look at the new patch set (#3).
Change subject: [tools] KUDU-3333 Include Table Counts in kudu hms Dryrun
......................................................................
[tools] KUDU-3333 Include Table Counts in kudu hms Dryrun
In cases where the user running the Kudu CLI tool, kudu hms
fix, doesn't have permissions from Ranger/Sentry to access
the tables, these tables would be treated as non-existant tables
in Kudu. In such scenarios, there might be situations where the
tables could be dropped from HMS inspite of them being present
in Kudu when run with -drop_orphan_tables flag.
This patch adds additional logging which reports the total
table counts from HMS and Kudu master catalogs and warns the
user if there are no tables in Kudu when kudu hms fix command is
run.
Sample runs of the tool before and after the change:
In case of an empty cluster no output is seen without the code
change. After the code change we see the below:
$ ./kudu hms fix `hostname -f`
I0315 16:16:36.039008 351197 tool_action_hms.cc:867] Number of Kudu tables found in Kudu master catalog: 0
I0315 16:16:36.039080 351197 tool_action_hms.cc:868] Number of Kudu tables found in HMS catalog: 0
$ ./kudu hms fix --dryrun `hostname -f`
I0315 16:16:55.158463 351291 tool_action_hms.cc:642] NOTE: There are zero kudu tables listed. If the cluster indeed has kudu tables please re-run the command with right credentials.
I0315 16:16:55.158546 351291 tool_action_hms.cc:867] Number of Kudu tables found in Kudu master catalog: 0
I0315 16:16:55.158555 351291 tool_action_hms.cc:868] Number of Kudu tables found in HMS catalog: 0
In case of a non-empty cluster without the change:
$ kudu hms fix --dryrun `hostname -f` --ignore_other_clusters=false
I0315 16:57:55.329049 365038 tool_action_hms.cc:757] [dryrun] Refreshing HMS table metadata for Kudu table default.my_first_table [id=408e5696e51c462c86a6d9a84bb95583]
Non-empty cluster after the change:
$ ./kudu hms fix --dryrun `hostname -f`
I0315 16:19:20.885208 352393 tool_action_hms.cc:822] [dryrun] Changing owner of default.my_first_table [id=408e5696e51c462c86a6d9a84bb95583] to admin in Kudu catalog.
I0315 16:19:20.885274 352393 tool_action_hms.cc:853] [dryrun] Refreshing HMS table metadata for Kudu table default.my_first_table [id=408e5696e51c462c86a6d9a84bb95583]
I0315 16:19:20.885285 352393 tool_action_hms.cc:867] Number of Kudu tables found in Kudu master catalog: 1
I0315 16:19:20.885325 352393 tool_action_hms.cc:868] Number of Kudu tables found in HMS catalog: 1
Change-Id: Idf26141d2a3fd6cbb7249b3492fc6a50a0c0aa2d
---
M src/kudu/tools/tool_action_hms.cc
1 file changed, 21 insertions(+), 4 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/80/18280/3
--
To view, visit http://gerrit.cloudera.org:8080/18280
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Idf26141d2a3fd6cbb7249b3492fc6a50a0c0aa2d
Gerrit-Change-Number: 18280
Gerrit-PatchSet: 3
Gerrit-Owner: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
[kudu-CR] [tools] KUDU-3333 Include Table Counts in kudu hms Dryrun
Posted by "Abhishek Chennaka (Code Review)" <ge...@cloudera.org>.
Hello Kudu Jenkins, Andrew Wong,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/18280
to look at the new patch set (#2).
Change subject: [tools] KUDU-3333 Include Table Counts in kudu hms Dryrun
......................................................................
[tools] KUDU-3333 Include Table Counts in kudu hms Dryrun
In cases where the user running the Kudu CLI tool, kudu hms
fix, doesn't have permissions from Ranger/Sentry to access
the tables, these tables would be treated as non-existant tables
in Kudu. In such scenarios, there might be situations where the
tables could be dropped from HMS inspite of them being present
in Kudu when run with -drop_orphan_tables flag.
This patch adds additional logging which reports the total
table counts from HMS and Kudu master catalogs and warns the
user if there are no tables in Kudu when kudu hms fix command is
run.
Sample runs of the tool after the change:
I0315 16:16:36.039008 351197 tool_action_hms.cc:867] Number of Kudu tables found in Kudu master catalog: 0
I0315 16:16:36.039080 351197 tool_action_hms.cc:868] Number of Kudu tables found in HMS catalog: 0
I0315 16:16:55.158463 351291 tool_action_hms.cc:642] NOTE: There are zero kudu tables listed. If the cluster indeed has kudu tables please re-run the command with right credentials.
I0315 16:16:55.158546 351291 tool_action_hms.cc:867] Number of Kudu tables found in Kudu master catalog: 0
I0315 16:16:55.158555 351291 tool_action_hms.cc:868] Number of Kudu tables found in HMS catalog: 0
I0315 16:19:20.885208 352393 tool_action_hms.cc:822] [dryrun] Changing owner of default.my_first_table [id=408e5696e51c462c86a6d9a84bb95583] to admin in Kudu catalog.
I0315 16:19:20.885274 352393 tool_action_hms.cc:853] [dryrun] Refreshing HMS table metadata for Kudu table default.my_first_table [id=408e5696e51c462c86a6d9a84bb95583]
I0315 16:19:20.885285 352393 tool_action_hms.cc:867] Number of Kudu tables found in Kudu master catalog: 1
I0315 16:19:20.885325 352393 tool_action_hms.cc:868] Number of Kudu tables found in HMS catalog: 1
Change-Id: Idf26141d2a3fd6cbb7249b3492fc6a50a0c0aa2d
---
M src/kudu/tools/tool_action_hms.cc
1 file changed, 19 insertions(+), 4 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/80/18280/2
--
To view, visit http://gerrit.cloudera.org:8080/18280
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Idf26141d2a3fd6cbb7249b3492fc6a50a0c0aa2d
Gerrit-Change-Number: 18280
Gerrit-PatchSet: 2
Gerrit-Owner: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
[kudu-CR] [tools] KUDU-3333 Include Table Counts in kudu hms Dryrun
Posted by "Andrew Wong (Code Review)" <ge...@cloudera.org>.
Andrew Wong has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/18280 )
Change subject: [tools] KUDU-3333 Include Table Counts in kudu hms Dryrun
......................................................................
[tools] KUDU-3333 Include Table Counts in kudu hms Dryrun
In cases where the user running the Kudu CLI tool, kudu hms
fix, doesn't have permissions from Ranger/Sentry to access
the tables, these tables would be treated as non-existant tables
in Kudu. In such scenarios, there might be situations where the
tables could be dropped from HMS inspite of them being present
in Kudu when run with -drop_orphan_tables flag.
This patch adds additional logging which reports the total
table counts from HMS and Kudu master catalogs and warns the
user if there are no tables in Kudu when kudu hms fix command is
run.
Sample runs of the tool before and after the change:
In case of an empty cluster no output is seen without the code
change. After the code change we see the below:
$ ./kudu hms fix `hostname -f`
I0315 16:16:36.039008 351197 tool_action_hms.cc:867] Number of Kudu tables found in Kudu master catalog: 0
I0315 16:16:36.039080 351197 tool_action_hms.cc:868] Number of Kudu tables found in HMS catalog: 0
$ ./kudu hms fix --dryrun `hostname -f`
I0315 16:16:55.158463 351291 tool_action_hms.cc:642] NOTE: There are zero kudu tables listed. If the cluster indeed has kudu tables please re-run the command with right credentials.
I0315 16:16:55.158546 351291 tool_action_hms.cc:867] Number of Kudu tables found in Kudu master catalog: 0
I0315 16:16:55.158555 351291 tool_action_hms.cc:868] Number of Kudu tables found in HMS catalog: 0
In case of a non-empty cluster without the change:
$ kudu hms fix --dryrun `hostname -f` --ignore_other_clusters=false
I0315 16:57:55.329049 365038 tool_action_hms.cc:757] [dryrun] Refreshing HMS table metadata for Kudu table default.my_first_table [id=408e5696e51c462c86a6d9a84bb95583]
Non-empty cluster after the change:
$ ./kudu hms fix --dryrun `hostname -f`
I0315 16:19:20.885208 352393 tool_action_hms.cc:822] [dryrun] Changing owner of default.my_first_table [id=408e5696e51c462c86a6d9a84bb95583] to admin in Kudu catalog.
I0315 16:19:20.885274 352393 tool_action_hms.cc:853] [dryrun] Refreshing HMS table metadata for Kudu table default.my_first_table [id=408e5696e51c462c86a6d9a84bb95583]
I0315 16:19:20.885285 352393 tool_action_hms.cc:867] Number of Kudu tables found in Kudu master catalog: 1
I0315 16:19:20.885325 352393 tool_action_hms.cc:868] Number of Kudu tables found in HMS catalog: 1
Change-Id: Idf26141d2a3fd6cbb7249b3492fc6a50a0c0aa2d
Reviewed-on: http://gerrit.cloudera.org:8080/18280
Tested-by: Kudu Jenkins
Reviewed-by: Andrew Wong <aw...@cloudera.com>
---
M src/kudu/tools/tool_action_hms.cc
1 file changed, 21 insertions(+), 4 deletions(-)
Approvals:
Kudu Jenkins: Verified
Andrew Wong: Looks good to me, approved
--
To view, visit http://gerrit.cloudera.org:8080/18280
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: Idf26141d2a3fd6cbb7249b3492fc6a50a0c0aa2d
Gerrit-Change-Number: 18280
Gerrit-PatchSet: 4
Gerrit-Owner: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)