You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Fang-Yu Rao (Code Review)" <ge...@cloudera.org> on 2022/12/18 10:31:07 UTC
[Impala-ASF-CR] IMPALA-10399, IMPALA-11788: Reset Ranger policy repository in an E2E test
Fang-Yu Rao has uploaded this change for review. ( http://gerrit.cloudera.org:8080/19373
Change subject: IMPALA-10399, IMPALA-11788: Reset Ranger policy repository in an E2E test
......................................................................
IMPALA-10399, IMPALA-11788: Reset Ranger policy repository in an E2E test
test_show_grant_hive_privilege() uses Ranger's REST API to get all the
existing policies from the Ranger server after creating a policy that
grants the LOCK and SELECT privileges on all the tables and columns in
the unique database in order to verify the granted privileges indeed
exist in Ranger's policy repository.
The way we downloaded all the policies from the Ranger server in
test_show_grant_hive_privilege(), however, did not always work.
Specifically, when there were already a lot of existing policies in
Ranger, the policy that granted the LOCK and SELECT privileges would not
be included in the result returned via one single GET request. We found
that to reproduce the issue it suffices to add 300 Ranger policies
before adding the policy granting those 2 privileges.
Moreover, we found that even we set the argument 'stream' of
requests.get() to True and used iter_content() to read the response in
chunks, we still could not retrieve the policy added in
test_show_grant_hive_privilege().
As a workaround, this patch resets Ranger's policy repository for Impala
before we create the policy granting those 2 privileges so that this
test will be more resilient to the number of existing policies in the
repository.
Change-Id: Iff56ec03ceeb2912039241ea302f4bb8948d61f8
---
M testdata/bin/create-load-data.sh
A testdata/bin/setup-ranger.sh
M tests/authorization/test_ranger.py
3 files changed, 111 insertions(+), 71 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/73/19373/2
--
To view, visit http://gerrit.cloudera.org:8080/19373
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: Iff56ec03ceeb2912039241ea302f4bb8948d61f8
Gerrit-Change-Number: 19373
Gerrit-PatchSet: 2
Gerrit-Owner: Fang-Yu Rao <fa...@cloudera.com>
[Impala-ASF-CR] IMPALA-10399, IMPALA-11060, IMPALA-11788: Reset Ranger policy repository in an E2E test
Posted by "Fang-Yu Rao (Code Review)" <ge...@cloudera.org>.
Hello Quanlong Huang, Andrew Sherman, Csaba Ringhofer, Michael Smith, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/19373
to look at the new patch set (#5).
Change subject: IMPALA-10399, IMPALA-11060, IMPALA-11788: Reset Ranger policy repository in an E2E test
......................................................................
IMPALA-10399, IMPALA-11060, IMPALA-11788: Reset Ranger policy repository in an E2E test
test_show_grant_hive_privilege() uses Ranger's REST API to get all the
existing policies from the Ranger server after creating a policy that
grants the LOCK and SELECT privileges on all the tables and columns in
the unique database in order to verify the granted privileges indeed
exist in Ranger's policy repository.
The way we download all the policies from the Ranger server in
test_show_grant_hive_privilege(), however, did not
always work. Specifically, when there were already a lot of existing
policies in Ranger, the policy that granted the LOCK and SELECT
privileges would not be included in the result returned via one single
GET request. We found that to reproduce the issue it suffices to add 300
Ranger policies before adding the policy granting those 2 privileges.
Moreover, we found that even we set the argument 'stream' of
requests.get() to True and used iter_content() to read the response in
chunks, we still could not retrieve the policy added in
test_show_grant_hive_privilege().
As a workaround, instead of changing how we download all the policies
from the Ranger server, this patch resets Ranger's policy repository for
Impala before we create the policy granting those 2 privileges so that
this test will be more resilient to the number of existing policies in
the repository.
Change-Id: Iff56ec03ceeb2912039241ea302f4bb8948d61f8
---
M testdata/bin/create-load-data.sh
A testdata/bin/setup-ranger.sh
M tests/authorization/test_ranger.py
M tests/common/custom_cluster_test_suite.py
4 files changed, 115 insertions(+), 73 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/73/19373/5
--
To view, visit http://gerrit.cloudera.org:8080/19373
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Iff56ec03ceeb2912039241ea302f4bb8948d61f8
Gerrit-Change-Number: 19373
Gerrit-PatchSet: 5
Gerrit-Owner: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Andrew Sherman <as...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
[Impala-ASF-CR] IMPALA-10399, IMPALA-11788: Reset Ranger policy repository in an E2E test
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19373 )
Change subject: IMPALA-10399, IMPALA-11788: Reset Ranger policy repository in an E2E test
......................................................................
Patch Set 2:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/8927/ DRY_RUN=true
--
To view, visit http://gerrit.cloudera.org:8080/19373
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iff56ec03ceeb2912039241ea302f4bb8948d61f8
Gerrit-Change-Number: 19373
Gerrit-PatchSet: 2
Gerrit-Owner: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Sun, 18 Dec 2022 10:31:41 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10399, IMPALA-11788: Reset Ranger policy repository in an E2E test
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19373 )
Change subject: IMPALA-10399, IMPALA-11788: Reset Ranger policy repository in an E2E test
......................................................................
Patch Set 2:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/12040/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/19373
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iff56ec03ceeb2912039241ea302f4bb8948d61f8
Gerrit-Change-Number: 19373
Gerrit-PatchSet: 2
Gerrit-Owner: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Sun, 18 Dec 2022 10:50:53 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10399, IMPALA-11060, IMPALA-11788: Reset Ranger policy repository in an E2E test
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19373 )
Change subject: IMPALA-10399, IMPALA-11060, IMPALA-11788: Reset Ranger policy repository in an E2E test
......................................................................
Patch Set 3:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/12052/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/19373
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iff56ec03ceeb2912039241ea302f4bb8948d61f8
Gerrit-Change-Number: 19373
Gerrit-PatchSet: 3
Gerrit-Owner: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Wed, 21 Dec 2022 05:31:11 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10399, IMPALA-11060, IMPALA-11788: Reset Ranger policy repository in an E2E test
Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/19373 )
Change subject: IMPALA-10399, IMPALA-11060, IMPALA-11788: Reset Ranger policy repository in an E2E test
......................................................................
Patch Set 5: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/19373
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iff56ec03ceeb2912039241ea302f4bb8948d61f8
Gerrit-Change-Number: 19373
Gerrit-PatchSet: 5
Gerrit-Owner: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Andrew Sherman <as...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Wed, 28 Dec 2022 01:47:51 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10399, IMPALA-11788: Reset Ranger policy repository in an E2E test
Posted by "Fang-Yu Rao (Code Review)" <ge...@cloudera.org>.
Fang-Yu Rao has posted comments on this change. ( http://gerrit.cloudera.org:8080/19373 )
Change subject: IMPALA-10399, IMPALA-11788: Reset Ranger policy repository in an E2E test
......................................................................
Patch Set 2:
Hi all, please let me know if you have any comment on the patch. It passed the dryrun tests according to https://jenkins.impala.io/job/gerrit-verify-dryrun/8927/.
I just found that part of this patch also implements what we would like to do in IMPALA-11060. Hence I will update the subject of the patch in the next patch set.
Thanks!
--
To view, visit http://gerrit.cloudera.org:8080/19373
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iff56ec03ceeb2912039241ea302f4bb8948d61f8
Gerrit-Change-Number: 19373
Gerrit-PatchSet: 2
Gerrit-Owner: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Mon, 19 Dec 2022 01:55:17 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10399, IMPALA-11060, IMPALA-11788: Reset Ranger policy repository in an E2E test
Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/19373 )
Change subject: IMPALA-10399, IMPALA-11060, IMPALA-11788: Reset Ranger policy repository in an E2E test
......................................................................
IMPALA-10399, IMPALA-11060, IMPALA-11788: Reset Ranger policy repository in an E2E test
test_show_grant_hive_privilege() uses Ranger's REST API to get all the
existing policies from the Ranger server after creating a policy that
grants the LOCK and SELECT privileges on all the tables and columns in
the unique database in order to verify the granted privileges indeed
exist in Ranger's policy repository.
The way we download all the policies from the Ranger server in
test_show_grant_hive_privilege(), however, did not
always work. Specifically, when there were already a lot of existing
policies in Ranger, the policy that granted the LOCK and SELECT
privileges would not be included in the result returned via one single
GET request. We found that to reproduce the issue it suffices to add 300
Ranger policies before adding the policy granting those 2 privileges.
Moreover, we found that even we set the argument 'stream' of
requests.get() to True and used iter_content() to read the response in
chunks, we still could not retrieve the policy added in
test_show_grant_hive_privilege().
As a workaround, instead of changing how we download all the policies
from the Ranger server, this patch resets Ranger's policy repository for
Impala before we create the policy granting those 2 privileges so that
this test will be more resilient to the number of existing policies in
the repository.
Change-Id: Iff56ec03ceeb2912039241ea302f4bb8948d61f8
Reviewed-on: http://gerrit.cloudera.org:8080/19373
Tested-by: Impala Public Jenkins <im...@cloudera.com>
Reviewed-by: Quanlong Huang <hu...@gmail.com>
---
M testdata/bin/create-load-data.sh
A testdata/bin/setup-ranger.sh
M tests/authorization/test_ranger.py
M tests/common/custom_cluster_test_suite.py
4 files changed, 115 insertions(+), 73 deletions(-)
Approvals:
Impala Public Jenkins: Verified
Quanlong Huang: Looks good to me, approved
--
To view, visit http://gerrit.cloudera.org:8080/19373
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: Iff56ec03ceeb2912039241ea302f4bb8948d61f8
Gerrit-Change-Number: 19373
Gerrit-PatchSet: 6
Gerrit-Owner: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Andrew Sherman <as...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
[Impala-ASF-CR] IMPALA-10399, IMPALA-11060, IMPALA-11788: Reset Ranger policy repository in an E2E test
Posted by "Fang-Yu Rao (Code Review)" <ge...@cloudera.org>.
Hello Quanlong Huang, Csaba Ringhofer, Michael Smith, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/19373
to look at the new patch set (#4).
Change subject: IMPALA-10399, IMPALA-11060, IMPALA-11788: Reset Ranger policy repository in an E2E test
......................................................................
IMPALA-10399, IMPALA-11060, IMPALA-11788: Reset Ranger policy repository in an E2E test
test_show_grant_hive_privilege() uses Ranger's REST API to get all the
existing policies from the Ranger server after creating a policy that
grants the LOCK and SELECT privileges on all the tables and columns in
the unique database in order to verify the granted privileges indeed
exist in Ranger's policy repository.
The way we download all the policies from the Ranger server in
test_show_grant_hive_privilege(), however, did not
always work. Specifically, when there were already a lot of existing
policies in Ranger, the policy that granted the LOCK and SELECT
privileges would not be included in the result returned via one single
GET request. We found that to reproduce the issue it suffices to add 300
Ranger policies before adding the policy granting those 2 privileges.
Moreover, we found that even we set the argument 'stream' of
requests.get() to True and used iter_content() to read the response in
chunks, we still could not retrieve the policy added in
test_show_grant_hive_privilege().
As a workaround, instead of changing how we download all the policies
from the Ranger server, this patch resets Ranger's policy repository for
Impala before we create the policy granting those 2 privileges so that
this test will be more resilient to the number of existing policies in
the repository.
Change-Id: Iff56ec03ceeb2912039241ea302f4bb8948d61f8
---
M testdata/bin/create-load-data.sh
A testdata/bin/setup-ranger.sh
M tests/authorization/test_ranger.py
M tests/common/custom_cluster_test_suite.py
4 files changed, 116 insertions(+), 73 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/73/19373/4
--
To view, visit http://gerrit.cloudera.org:8080/19373
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Iff56ec03ceeb2912039241ea302f4bb8948d61f8
Gerrit-Change-Number: 19373
Gerrit-PatchSet: 4
Gerrit-Owner: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
[Impala-ASF-CR] IMPALA-10399, IMPALA-11788: Reset Ranger policy repository in an E2E test
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19373 )
Change subject: IMPALA-10399, IMPALA-11788: Reset Ranger policy repository in an E2E test
......................................................................
Patch Set 2: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/19373
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iff56ec03ceeb2912039241ea302f4bb8948d61f8
Gerrit-Change-Number: 19373
Gerrit-PatchSet: 2
Gerrit-Owner: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Sun, 18 Dec 2022 15:36:19 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10399, IMPALA-11060, IMPALA-11788: Reset Ranger policy repository in an E2E test
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19373 )
Change subject: IMPALA-10399, IMPALA-11060, IMPALA-11788: Reset Ranger policy repository in an E2E test
......................................................................
Patch Set 3:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/8928/ DRY_RUN=true
--
To view, visit http://gerrit.cloudera.org:8080/19373
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iff56ec03ceeb2912039241ea302f4bb8948d61f8
Gerrit-Change-Number: 19373
Gerrit-PatchSet: 3
Gerrit-Owner: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Wed, 21 Dec 2022 05:13:22 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10399, IMPALA-11060, IMPALA-11788: Reset Ranger policy repository in an E2E test
Posted by "Fang-Yu Rao (Code Review)" <ge...@cloudera.org>.
Hello Quanlong Huang, Csaba Ringhofer, Michael Smith, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/19373
to look at the new patch set (#3).
Change subject: IMPALA-10399, IMPALA-11060, IMPALA-11788: Reset Ranger policy repository in an E2E test
......................................................................
IMPALA-10399, IMPALA-11060, IMPALA-11788: Reset Ranger policy repository in an E2E test
test_show_grant_hive_privilege() uses Ranger's REST API to get all the
existing policies from the Ranger server after creating a policy that
grants the LOCK and SELECT privileges on all the tables and columns in
the unique database in order to verify the granted privileges indeed
exist in Ranger's policy repository.
Before this patch, the way we downloaded all the policies from the
Ranger server in test_show_grant_hive_privilege(), however, did not
always work. Specifically, when there were already a lot of existing
policies in Ranger, the policy that granted the LOCK and SELECT
privileges would not be included in the result returned via one single
GET request. We found that to reproduce the issue it suffices to add 300
Ranger policies before adding the policy granting those 2 privileges.
Moreover, we found that even we set the argument 'stream' of
requests.get() to True and used iter_content() to read the response in
chunks, we still could not retrieve the policy added in
test_show_grant_hive_privilege().
As a workaround, this patch resets Ranger's policy repository for Impala
before we create the policy granting those 2 privileges so that this
test will be more resilient to the number of existing policies in the
repository.
Change-Id: Iff56ec03ceeb2912039241ea302f4bb8948d61f8
---
M testdata/bin/create-load-data.sh
A testdata/bin/setup-ranger.sh
M tests/authorization/test_ranger.py
M tests/common/custom_cluster_test_suite.py
4 files changed, 116 insertions(+), 73 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/73/19373/3
--
To view, visit http://gerrit.cloudera.org:8080/19373
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Iff56ec03ceeb2912039241ea302f4bb8948d61f8
Gerrit-Change-Number: 19373
Gerrit-PatchSet: 3
Gerrit-Owner: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
[Impala-ASF-CR] IMPALA-10399, IMPALA-11060, IMPALA-11788: Reset Ranger policy repository in an E2E test
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19373 )
Change subject: IMPALA-10399, IMPALA-11060, IMPALA-11788: Reset Ranger policy repository in an E2E test
......................................................................
Patch Set 4:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/8931/ DRY_RUN=true
--
To view, visit http://gerrit.cloudera.org:8080/19373
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iff56ec03ceeb2912039241ea302f4bb8948d61f8
Gerrit-Change-Number: 19373
Gerrit-PatchSet: 4
Gerrit-Owner: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Andrew Sherman <as...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Sat, 24 Dec 2022 20:12:26 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10399, IMPALA-11060, IMPALA-11788: Reset Ranger policy repository in an E2E test
Posted by "Andrew Sherman (Code Review)" <ge...@cloudera.org>.
Andrew Sherman has posted comments on this change. ( http://gerrit.cloudera.org:8080/19373 )
Change subject: IMPALA-10399, IMPALA-11060, IMPALA-11788: Reset Ranger policy repository in an E2E test
......................................................................
Patch Set 4: Code-Review+2
LGTM - thanks for fixing this
--
To view, visit http://gerrit.cloudera.org:8080/19373
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iff56ec03ceeb2912039241ea302f4bb8948d61f8
Gerrit-Change-Number: 19373
Gerrit-PatchSet: 4
Gerrit-Owner: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Andrew Sherman <as...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Sat, 24 Dec 2022 17:00:41 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10399, IMPALA-11060, IMPALA-11788: Reset Ranger policy repository in an E2E test
Posted by "Fang-Yu Rao (Code Review)" <ge...@cloudera.org>.
Fang-Yu Rao has posted comments on this change. ( http://gerrit.cloudera.org:8080/19373 )
Change subject: IMPALA-10399, IMPALA-11060, IMPALA-11788: Reset Ranger policy repository in an E2E test
......................................................................
Patch Set 4:
Only the commit message was slightly updated in patch set 4 according to Quanlong's suggestion previously.
Let me know if there are additional comments. Thanks!
--
To view, visit http://gerrit.cloudera.org:8080/19373
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iff56ec03ceeb2912039241ea302f4bb8948d61f8
Gerrit-Change-Number: 19373
Gerrit-PatchSet: 4
Gerrit-Owner: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Wed, 21 Dec 2022 18:14:39 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10399, IMPALA-11060, IMPALA-11788: Reset Ranger policy repository in an E2E test
Posted by "Fang-Yu Rao (Code Review)" <ge...@cloudera.org>.
Fang-Yu Rao has posted comments on this change. ( http://gerrit.cloudera.org:8080/19373 )
Change subject: IMPALA-10399, IMPALA-11060, IMPALA-11788: Reset Ranger policy repository in an E2E test
......................................................................
Patch Set 3:
(2 comments)
Hi all, I have addressed Quanlong's comments in the previous iteration.
Let me know if there is any additional suggestion.
Thanks very much for the help!
http://gerrit.cloudera.org:8080/#/c/19373/2//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/19373/2//COMMIT_MSG@15
PS2, Line 15: patch, th
> nit: download
Thanks Quanlong!
To emphasize this was the behavior prior to this patch, in the updated commit message, I will add "Before this patch" in front the this sentence.
http://gerrit.cloudera.org:8080/#/c/19373/2/testdata/bin/setup-ranger.sh
File testdata/bin/setup-ranger.sh:
http://gerrit.cloudera.org:8080/#/c/19373/2/testdata/bin/setup-ranger.sh@25
PS2, Line 25: function setup-ranger {
> Thanks for extracting this! It also fixes IMPALA-11060.
Ack
--
To view, visit http://gerrit.cloudera.org:8080/19373
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iff56ec03ceeb2912039241ea302f4bb8948d61f8
Gerrit-Change-Number: 19373
Gerrit-PatchSet: 3
Gerrit-Owner: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Wed, 21 Dec 2022 05:12:54 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10399, IMPALA-11788: Reset Ranger policy repository in an E2E test
Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/19373 )
Change subject: IMPALA-10399, IMPALA-11788: Reset Ranger policy repository in an E2E test
......................................................................
Patch Set 2: Code-Review+1
(3 comments)
Thanks for working on this! The workaround LGTM.
http://gerrit.cloudera.org:8080/#/c/19373/2//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/19373/2//COMMIT_MSG@15
PS2, Line 15: downloaded
nit: download
http://gerrit.cloudera.org:8080/#/c/19373/2/testdata/bin/setup-ranger.sh
File testdata/bin/setup-ranger.sh:
http://gerrit.cloudera.org:8080/#/c/19373/2/testdata/bin/setup-ranger.sh@25
PS2, Line 25: function setup-ranger {
Thanks for extracting this! It also fixes IMPALA-11060.
http://gerrit.cloudera.org:8080/#/c/19373/2/tests/authorization/test_ranger.py
File tests/authorization/test_ranger.py:
http://gerrit.cloudera.org:8080/#/c/19373/2/tests/authorization/test_ranger.py@754
PS2, Line 754: check_call([script_kill_ranger])
: check_call([script_create_test_config, '-create_ranger_policy_db'])
: check_call([script_run_ranger])
: check_call([script_setup_ranger])
: # We need to wait for long enough for the policies set up by setup-ranger.sh to
: # take effect. Executing "refresh authorization" as the user 'admin' does not work
: # because the user 'admin' has not been granted the required privileges to execute
: # the statement before the Ranger plugin is updated.
: time.sleep(60)
It might be better to do this in CustomClusterTestSuite.setup_method() before starting the Impala cluster. So we don't need to wait for 60s.
https://github.com/apache/impala/blob/390a932064ba48c44e70e431f2c20e53beddd970/tests/common/custom_cluster_test_suite.py#L159
We can add an argument like reset_ranger (default to False) in with_args():
https://github.com/apache/impala/blob/390a932064ba48c44e70e431f2c20e53beddd970/tests/common/custom_cluster_test_suite.py#L105
and use it later in setup_method().
--
To view, visit http://gerrit.cloudera.org:8080/19373
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iff56ec03ceeb2912039241ea302f4bb8948d61f8
Gerrit-Change-Number: 19373
Gerrit-PatchSet: 2
Gerrit-Owner: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Mon, 19 Dec 2022 01:58:45 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10399, IMPALA-11060, IMPALA-11788: Reset Ranger policy repository in an E2E test
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19373 )
Change subject: IMPALA-10399, IMPALA-11060, IMPALA-11788: Reset Ranger policy repository in an E2E test
......................................................................
Patch Set 5:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/8932/ DRY_RUN=true
--
To view, visit http://gerrit.cloudera.org:8080/19373
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iff56ec03ceeb2912039241ea302f4bb8948d61f8
Gerrit-Change-Number: 19373
Gerrit-PatchSet: 5
Gerrit-Owner: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Andrew Sherman <as...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Mon, 26 Dec 2022 23:12:39 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10399, IMPALA-11060, IMPALA-11788: Reset Ranger policy repository in an E2E test
Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/19373 )
Change subject: IMPALA-10399, IMPALA-11060, IMPALA-11788: Reset Ranger policy repository in an E2E test
......................................................................
Patch Set 5:
Merging this. Thank Fang-Yu and Andrew!
--
To view, visit http://gerrit.cloudera.org:8080/19373
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iff56ec03ceeb2912039241ea302f4bb8948d61f8
Gerrit-Change-Number: 19373
Gerrit-PatchSet: 5
Gerrit-Owner: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Andrew Sherman <as...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Wed, 28 Dec 2022 01:48:24 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10399, IMPALA-11060, IMPALA-11788: Reset Ranger policy repository in an E2E test
Posted by "Fang-Yu Rao (Code Review)" <ge...@cloudera.org>.
Fang-Yu Rao has posted comments on this change. ( http://gerrit.cloudera.org:8080/19373 )
Change subject: IMPALA-10399, IMPALA-11060, IMPALA-11788: Reset Ranger policy repository in an E2E test
......................................................................
Patch Set 5:
Thanks for the +2 Andrew! :-)
I just found that there was a redundant space in patch set 4 at https://gerrit.cloudera.org/c/19373/4/tests/authorization/test_ranger.py#744 and have removed it in the next patch.
--
To view, visit http://gerrit.cloudera.org:8080/19373
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iff56ec03ceeb2912039241ea302f4bb8948d61f8
Gerrit-Change-Number: 19373
Gerrit-PatchSet: 5
Gerrit-Owner: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Andrew Sherman <as...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Mon, 26 Dec 2022 22:57:39 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10399, IMPALA-11060, IMPALA-11788: Reset Ranger policy repository in an E2E test
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19373 )
Change subject: IMPALA-10399, IMPALA-11060, IMPALA-11788: Reset Ranger policy repository in an E2E test
......................................................................
Patch Set 5:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/12066/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/19373
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iff56ec03ceeb2912039241ea302f4bb8948d61f8
Gerrit-Change-Number: 19373
Gerrit-PatchSet: 5
Gerrit-Owner: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Andrew Sherman <as...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Mon, 26 Dec 2022 23:16:40 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10399, IMPALA-11060, IMPALA-11788: Reset Ranger policy repository in an E2E test
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19373 )
Change subject: IMPALA-10399, IMPALA-11060, IMPALA-11788: Reset Ranger policy repository in an E2E test
......................................................................
Patch Set 4: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/19373
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iff56ec03ceeb2912039241ea302f4bb8948d61f8
Gerrit-Change-Number: 19373
Gerrit-PatchSet: 4
Gerrit-Owner: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Andrew Sherman <as...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Sun, 25 Dec 2022 01:17:58 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10399, IMPALA-11060, IMPALA-11788: Reset Ranger policy repository in an E2E test
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19373 )
Change subject: IMPALA-10399, IMPALA-11060, IMPALA-11788: Reset Ranger policy repository in an E2E test
......................................................................
Patch Set 5: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/19373
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iff56ec03ceeb2912039241ea302f4bb8948d61f8
Gerrit-Change-Number: 19373
Gerrit-PatchSet: 5
Gerrit-Owner: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Andrew Sherman <as...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Tue, 27 Dec 2022 04:27:29 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10399, IMPALA-11060, IMPALA-11788: Reset Ranger policy repository in an E2E test
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19373 )
Change subject: IMPALA-10399, IMPALA-11060, IMPALA-11788: Reset Ranger policy repository in an E2E test
......................................................................
Patch Set 3: Verified-1
Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/8928/
--
To view, visit http://gerrit.cloudera.org:8080/19373
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iff56ec03ceeb2912039241ea302f4bb8948d61f8
Gerrit-Change-Number: 19373
Gerrit-PatchSet: 3
Gerrit-Owner: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Wed, 21 Dec 2022 10:18:40 +0000
Gerrit-HasComments: No