You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Fang-Yu Rao (Code Review)" <ge...@cloudera.org> on 2020/01/21 23:39:30 UTC

[Impala-ASF-CR] IMPALA-9149: part 2: Re-enable Ranger-related EE tests 

Fang-Yu Rao has uploaded this change for review. ( http://gerrit.cloudera.org:8080/15088


Change subject: IMPALA-9149: part 2: Re-enable Ranger-related EE tests
......................................................................

IMPALA-9149: part 2: Re-enable Ranger-related EE tests

In IMPALA-9047, we disabled some Ranger-related FE and BE tests due to
changes in Ranger's behavior after upgrading Ranger from 1.2 to 2.0.
This patch aims to re-enable those disabled EE tests in
tests/authorization/test_authorized_proxy.py and
tests/authorization/test_ranger.py to increase Impala's test coverage of
authorization via Ranger.

The Ranger-related tests in test_authorized_proxy.py test Impala's
delegation for clients. Two types of delegation are supported in Impala,
i.e., a user can delegate the execution of a query to either 1) another
user, or 2) a group of users. In the former case, Ranger will check
whether or not the delegated user specified in the option
'authorized_proxy_user_config' possesses sufficient privileges to access
the resources, whereas in the latter case, before checking the delegated
group is granted sufficient privileges, Ranger will check with the help
of Impala whether or not the delegated user specified in
'authorized_proxy_user_config' belongs to the delegated group specified
in 'authorized_proxy_group_config' in the underlying OS. This type of
delegation requires Impala to retrieve the groups the delegated user
belongs to from the underlying OS and thus if the delegated user does
not exist in the underlying OS, Impala would inform Ranger that the
delegated user does not belong to any group, which in turn would fail
the authorization even though in the policies on the Ranger server, the
delegated user belongs to the delegated group and the delegated group is
granted sufficient privileges. The re-enabled tests here involve queries
in which the delegated user does not exist in the underlying OS. To
avoid the need for creating an actual delegated user and its
corresponding delegated groups in the underlying OS when running the EE
tests, we added to 'impalad_args' an additional option, i.e.,
'use_customized_user_group_mapper_for_ranger', which, when set to true,
allows Impala to use a customized user-to-groups mapping.

On the other hand, to re-enable test_legacy_catalog_ownership() in
test_ranger.py, we removed in _test_ownership() a test query that was
expected to fail the authorization in Ranger 1.2 but passes the
authorization in Ranger 2.0. This is due to the fact that in Ranger 2.0,
a user does not have to be explicitly granted the privileges of
accessing a resource as long as the user is the owner of the resource.

Testing:
- Passed FE tests.
- Passed the tests in test_authorized_proxy.py.
- Passed the tests in test_ranger.py.

Change-Id: I17420d7ff9beacd1b4d2ad72b68b8b54983e60cb
---
M be/src/common/global-flags.cc
M be/src/util/backend-gflag-util.cc
M common/thrift/BackendGflags.thrift
M fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationChecker.java
M fe/src/main/java/org/apache/impala/service/BackendConfig.java
M tests/authorization/test_authorized_proxy.py
M tests/authorization/test_ranger.py
7 files changed, 81 insertions(+), 96 deletions(-)



  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/88/15088/1
-- 
To view, visit http://gerrit.cloudera.org:8080/15088
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I17420d7ff9beacd1b4d2ad72b68b8b54983e60cb
Gerrit-Change-Number: 15088
Gerrit-PatchSet: 1
Gerrit-Owner: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>

[Impala-ASF-CR] IMPALA-9149: part 2: Re-enable Ranger-related EE tests

Posted by "Fang-Yu Rao (Code Review)" <ge...@cloudera.org>.
Fang-Yu Rao has uploaded a new patch set (#2). ( http://gerrit.cloudera.org:8080/15088 )

Change subject: IMPALA-9149: part 2: Re-enable Ranger-related EE tests
......................................................................

IMPALA-9149: part 2: Re-enable Ranger-related EE tests

In IMPALA-9047, we disabled some Ranger-related FE and BE tests due to
changes in Ranger's behavior after upgrading Ranger from 1.2 to 2.0.
This patch aims to re-enable those disabled EE tests in
tests/authorization/test_authorized_proxy.py and
tests/authorization/test_ranger.py to increase Impala's test coverage of
authorization via Ranger.

The Ranger-related tests in test_authorized_proxy.py test Impala's
delegation for clients. Two types of delegation are supported in Impala,
i.e., a user can delegate the execution of a query to either 1) another
user, or 2) a group of users. In the former case, Ranger will check
whether or not the delegated user specified in the option
'authorized_proxy_user_config' possesses sufficient privileges to access
the resources, whereas in the latter case, before checking the delegated
group is granted sufficient privileges, Ranger will check with the help
of Impala whether or not the delegated user specified in
'authorized_proxy_user_config' belongs to the delegated group specified
in 'authorized_proxy_group_config' in the underlying OS. This type of
delegation requires Impala to retrieve the groups the delegated user
belongs to from the underlying OS and thus if the delegated user does
not exist in the underlying OS, Impala would inform Ranger that the
delegated user does not belong to any group, which in turn would fail
the authorization even though in the policies on the Ranger server, the
delegated user belongs to the delegated group and the delegated group is
granted sufficient privileges. The re-enabled tests here involve queries
in which the delegated user does not exist in the underlying OS. To
avoid the need for creating an actual delegated user and its
corresponding delegated groups in the underlying OS when running the EE
tests, we added to 'impalad_args' an additional option, i.e.,
'use_customized_user_group_mapper_for_ranger', which, when set to true,
allows Impala to use a customized user-to-groups mapping.

On the other hand, to re-enable test_legacy_catalog_ownership() in
test_ranger.py, we removed in _test_ownership() a test query that was
expected to fail the authorization in Ranger 1.2 but passes the
authorization in Ranger 2.0. This is due to the fact that in Ranger 2.0,
a user does not have to be explicitly granted the privileges of
accessing a resource as long as the user is the owner of the resource.

Testing:
- Passed FE tests.
- Passed the tests in test_authorized_proxy.py.
- Passed the tests in test_ranger.py.

Change-Id: I17420d7ff9beacd1b4d2ad72b68b8b54983e60cb
---
M be/src/common/global-flags.cc
M be/src/util/backend-gflag-util.cc
M common/thrift/BackendGflags.thrift
M fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationChecker.java
M fe/src/main/java/org/apache/impala/service/BackendConfig.java
M tests/authorization/test_authorized_proxy.py
M tests/authorization/test_ranger.py
7 files changed, 81 insertions(+), 96 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/88/15088/2
-- 
To view, visit http://gerrit.cloudera.org:8080/15088
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I17420d7ff9beacd1b4d2ad72b68b8b54983e60cb
Gerrit-Change-Number: 15088
Gerrit-PatchSet: 2
Gerrit-Owner: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>

[Impala-ASF-CR] IMPALA-9149: part 2: Re-enable Ranger-related EE tests

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15088 )

Change subject: IMPALA-9149: part 2: Re-enable Ranger-related EE tests
......................................................................


Patch Set 5: Code-Review+2


-- 
To view, visit http://gerrit.cloudera.org:8080/15088
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I17420d7ff9beacd1b4d2ad72b68b8b54983e60cb
Gerrit-Change-Number: 15088
Gerrit-PatchSet: 5
Gerrit-Owner: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Thu, 30 Jan 2020 14:23:31 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-9149: part 2: Re-enable Ranger-related EE tests

Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/15088 )

Change subject: IMPALA-9149: part 2: Re-enable Ranger-related EE tests
......................................................................


Patch Set 2: Code-Review+1

(2 comments)

Good job! LGTM.

http://gerrit.cloudera.org:8080/#/c/15088/2/tests/authorization/test_authorized_proxy.py
File tests/authorization/test_authorized_proxy.py:

http://gerrit.cloudera.org:8080/#/c/15088/2/tests/authorization/test_authorized_proxy.py@119
PS2, Line 119: "non_owner"
nit: As it's a constant now, we can put it in the string. The same in other places that change getuser() to "non_owner".


http://gerrit.cloudera.org:8080/#/c/15088/2/tests/authorization/test_authorized_proxy.py@139
PS2, Line 139: --use_customized_user_group_mapper_for_ranger=true
nit: Can the boolean type flag be used without "=true"?



-- 
To view, visit http://gerrit.cloudera.org:8080/15088
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I17420d7ff9beacd1b4d2ad72b68b8b54983e60cb
Gerrit-Change-Number: 15088
Gerrit-PatchSet: 2
Gerrit-Owner: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Wed, 22 Jan 2020 02:52:56 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-9149: part 2: Re-enable Ranger-related EE tests

Posted by "Fang-Yu Rao (Code Review)" <ge...@cloudera.org>.
Fang-Yu Rao has posted comments on this change. ( http://gerrit.cloudera.org:8080/15088 )

Change subject: IMPALA-9149: part 2: Re-enable Ranger-related EE tests
......................................................................


Patch Set 4:

Added Csaba as a reviewer. Thanks to Csaba for the help!


-- 
To view, visit http://gerrit.cloudera.org:8080/15088
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I17420d7ff9beacd1b4d2ad72b68b8b54983e60cb
Gerrit-Change-Number: 15088
Gerrit-PatchSet: 4
Gerrit-Owner: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Wed, 29 Jan 2020 18:39:16 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-9149: part 2: Re-enable Ranger-related EE tests

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15088 )

Change subject: IMPALA-9149: part 2: Re-enable Ranger-related EE tests
......................................................................


Patch Set 4:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/5501/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/15088
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I17420d7ff9beacd1b4d2ad72b68b8b54983e60cb
Gerrit-Change-Number: 15088
Gerrit-PatchSet: 4
Gerrit-Owner: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Thu, 23 Jan 2020 01:34:04 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-9149: part 2: Re-enable Ranger-related EE tests

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15088 )

Change subject: IMPALA-9149: part 2: Re-enable Ranger-related EE tests
......................................................................


Patch Set 5:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/5476/ DRY_RUN=false


-- 
To view, visit http://gerrit.cloudera.org:8080/15088
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I17420d7ff9beacd1b4d2ad72b68b8b54983e60cb
Gerrit-Change-Number: 15088
Gerrit-PatchSet: 5
Gerrit-Owner: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Thu, 30 Jan 2020 14:23:32 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-9149: part 2: Re-enable Ranger-related EE tests

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15088 )

Change subject: IMPALA-9149: part 2: Re-enable Ranger-related EE tests
......................................................................


Patch Set 1:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/5487/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/15088
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I17420d7ff9beacd1b4d2ad72b68b8b54983e60cb
Gerrit-Change-Number: 15088
Gerrit-PatchSet: 1
Gerrit-Owner: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Wed, 22 Jan 2020 00:24:28 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-9149: part 2: Re-enable Ranger-related EE tests

Posted by "Fang-Yu Rao (Code Review)" <ge...@cloudera.org>.
Fang-Yu Rao has posted comments on this change. ( http://gerrit.cloudera.org:8080/15088 )

Change subject: IMPALA-9149: part 2: Re-enable Ranger-related EE tests
......................................................................


Patch Set 5:

> Patch Set 4: Code-Review+2
> 
> (1 comment)

Thanks Csaba for the pointer! I will take note of this. :-)


-- 
To view, visit http://gerrit.cloudera.org:8080/15088
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I17420d7ff9beacd1b4d2ad72b68b8b54983e60cb
Gerrit-Change-Number: 15088
Gerrit-PatchSet: 5
Gerrit-Owner: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Thu, 30 Jan 2020 16:09:14 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-9149: part 2: Re-enable Ranger-related EE tests

Posted by "Fang-Yu Rao (Code Review)" <ge...@cloudera.org>.
Fang-Yu Rao has uploaded a new patch set (#4). ( http://gerrit.cloudera.org:8080/15088 )

Change subject: IMPALA-9149: part 2: Re-enable Ranger-related EE tests
......................................................................

IMPALA-9149: part 2: Re-enable Ranger-related EE tests

In IMPALA-9047, we disabled some Ranger-related FE and BE tests due to
changes in Ranger's behavior after upgrading Ranger from 1.2 to 2.0.
This patch aims to re-enable those disabled EE tests in
tests/authorization/test_authorized_proxy.py and
tests/authorization/test_ranger.py to increase Impala's test coverage of
authorization via Ranger.

The Ranger-related tests in test_authorized_proxy.py test Impala's
delegation for clients. Two types of delegation are supported in Impala,
i.e., a user can delegate the execution of a query to either 1) another
user, or 2) a group of users. In the former case, Ranger will check
whether or not the delegated user specified in the option
'authorized_proxy_user_config' possesses sufficient privileges to access
the resources, whereas in the latter case, before checking the delegated
group is granted sufficient privileges, Ranger will check with the help
of Impala whether or not the delegated user specified in
'authorized_proxy_user_config' belongs to the delegated group specified
in 'authorized_proxy_group_config' in the underlying OS. This type of
delegation requires Impala to retrieve the groups the delegated user
belongs to from the underlying OS and thus if the delegated user does
not exist in the underlying OS, Impala would inform Ranger that the
delegated user does not belong to any group, which in turn would fail
the authorization even though in the policies on the Ranger server, the
delegated user belongs to the delegated group and the delegated group is
granted sufficient privileges.

The re-enabled Ranger tests in test_authorized_proxy.py involve queries
in which the delegated user, i.e., 'non_owner', does not exist in the
underlying OS. We use 'non_owner' as the delegated user instead of
getuser() so that we will have to explicitly grant 'non_owner'
sufficient privileges of accessing the resources. To avoid the need for
creating an actual delegated user and its corresponding delegated groups
in the underlying OS when running the EE tests, we added to
'impalad_args' an additional option, i.e.,
'use_customized_user_groups_mapper_for_ranger', which, when set to true,
allows Impala to use a customized user-to-groups mapping when performing
authorization via Ranger. On the other hand, we set the delegated user
to getuser() when running the respective Sentry related tests to avoid
the need for having to provide Sentry with a customized user-to-groups
mapping.

To re-enable test_legacy_catalog_ownership() in test_ranger.py, we
removed in _test_ownership() a test query that was expected to fail the
authorization in Ranger 1.2 but passes the authorization in Ranger 2.0.
This is due to the fact that in Ranger 2.0, a user does not have to be
explicitly granted the privileges of accessing a resource as long as the
user is the owner of the resource.

Testing:
- Passed FE tests.
- Passed the tests in test_authorized_proxy.py.
- Passed the tests in test_ranger.py.

Change-Id: I17420d7ff9beacd1b4d2ad72b68b8b54983e60cb
---
M be/src/common/global-flags.cc
M be/src/util/backend-gflag-util.cc
M common/thrift/BackendGflags.thrift
M fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationChecker.java
M fe/src/main/java/org/apache/impala/service/BackendConfig.java
M tests/authorization/test_authorized_proxy.py
M tests/authorization/test_ranger.py
7 files changed, 90 insertions(+), 103 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/88/15088/4
-- 
To view, visit http://gerrit.cloudera.org:8080/15088
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I17420d7ff9beacd1b4d2ad72b68b8b54983e60cb
Gerrit-Change-Number: 15088
Gerrit-PatchSet: 4
Gerrit-Owner: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>

[Impala-ASF-CR] IMPALA-9149: part 2: Re-enable Ranger-related EE tests

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15088 )

Change subject: IMPALA-9149: part 2: Re-enable Ranger-related EE tests
......................................................................


Patch Set 5: Verified+1


-- 
To view, visit http://gerrit.cloudera.org:8080/15088
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I17420d7ff9beacd1b4d2ad72b68b8b54983e60cb
Gerrit-Change-Number: 15088
Gerrit-PatchSet: 5
Gerrit-Owner: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Thu, 30 Jan 2020 19:13:18 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-9149: part 2: Re-enable Ranger-related EE tests

Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/15088 )

Change subject: IMPALA-9149: part 2: Re-enable Ranger-related EE tests
......................................................................


Patch Set 4: Code-Review+2

(1 comment)

http://gerrit.cloudera.org:8080/#/c/15088/4//COMMIT_MSG
Commit Message:

http://gerrit.cloudera.org:8080/#/c/15088/4//COMMIT_MSG@47
PS4, Line 47:  provide Sentry with a customized user-to-groups
            : mapping.
fyi Sentry tests have their own mechanism of to add custom user->group mappings if needed, see:

https://github.com/apache/impala/blob/79c5f87565467074697a7d98e01c9742f7228991/fe/src/test/resources/sentry-site.xml.py#L61

https://github.com/apache/impala/blob/master/fe/src/test/java/org/apache/impala/testutil/TestSentryGroupMapper.java



-- 
To view, visit http://gerrit.cloudera.org:8080/15088
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I17420d7ff9beacd1b4d2ad72b68b8b54983e60cb
Gerrit-Change-Number: 15088
Gerrit-PatchSet: 4
Gerrit-Owner: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Thu, 30 Jan 2020 12:53:55 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-9149: part 2: Re-enable Ranger-related EE tests

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15088 )

Change subject: IMPALA-9149: part 2: Re-enable Ranger-related EE tests
......................................................................


Patch Set 2:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/5490/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/15088
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I17420d7ff9beacd1b4d2ad72b68b8b54983e60cb
Gerrit-Change-Number: 15088
Gerrit-PatchSet: 2
Gerrit-Owner: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Wed, 22 Jan 2020 01:26:08 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-9149: part 2: Re-enable Ranger-related EE tests

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/15088 )

Change subject: IMPALA-9149: part 2: Re-enable Ranger-related EE tests
......................................................................

IMPALA-9149: part 2: Re-enable Ranger-related EE tests

In IMPALA-9047, we disabled some Ranger-related FE and BE tests due to
changes in Ranger's behavior after upgrading Ranger from 1.2 to 2.0.
This patch aims to re-enable those disabled EE tests in
tests/authorization/test_authorized_proxy.py and
tests/authorization/test_ranger.py to increase Impala's test coverage of
authorization via Ranger.

The Ranger-related tests in test_authorized_proxy.py test Impala's
delegation for clients. Two types of delegation are supported in Impala,
i.e., a user can delegate the execution of a query to either 1) another
user, or 2) a group of users. In the former case, Ranger will check
whether or not the delegated user specified in the option
'authorized_proxy_user_config' possesses sufficient privileges to access
the resources, whereas in the latter case, before checking the delegated
group is granted sufficient privileges, Ranger will check with the help
of Impala whether or not the delegated user specified in
'authorized_proxy_user_config' belongs to the delegated group specified
in 'authorized_proxy_group_config' in the underlying OS. This type of
delegation requires Impala to retrieve the groups the delegated user
belongs to from the underlying OS and thus if the delegated user does
not exist in the underlying OS, Impala would inform Ranger that the
delegated user does not belong to any group, which in turn would fail
the authorization even though in the policies on the Ranger server, the
delegated user belongs to the delegated group and the delegated group is
granted sufficient privileges.

The re-enabled Ranger tests in test_authorized_proxy.py involve queries
in which the delegated user, i.e., 'non_owner', does not exist in the
underlying OS. We use 'non_owner' as the delegated user instead of
getuser() so that we will have to explicitly grant 'non_owner'
sufficient privileges of accessing the resources. To avoid the need for
creating an actual delegated user and its corresponding delegated groups
in the underlying OS when running the EE tests, we added to
'impalad_args' an additional option, i.e.,
'use_customized_user_groups_mapper_for_ranger', which, when set to true,
allows Impala to use a customized user-to-groups mapping when performing
authorization via Ranger. On the other hand, we set the delegated user
to getuser() when running the respective Sentry related tests to avoid
the need for having to provide Sentry with a customized user-to-groups
mapping.

To re-enable test_legacy_catalog_ownership() in test_ranger.py, we
removed in _test_ownership() a test query that was expected to fail the
authorization in Ranger 1.2 but passes the authorization in Ranger 2.0.
This is due to the fact that in Ranger 2.0, a user does not have to be
explicitly granted the privileges of accessing a resource as long as the
user is the owner of the resource.

Testing:
- Passed FE tests.
- Passed the tests in test_authorized_proxy.py.
- Passed the tests in test_ranger.py.

Change-Id: I17420d7ff9beacd1b4d2ad72b68b8b54983e60cb
Reviewed-on: http://gerrit.cloudera.org:8080/15088
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
M be/src/common/global-flags.cc
M be/src/util/backend-gflag-util.cc
M common/thrift/BackendGflags.thrift
M fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationChecker.java
M fe/src/main/java/org/apache/impala/service/BackendConfig.java
M tests/authorization/test_authorized_proxy.py
M tests/authorization/test_ranger.py
7 files changed, 90 insertions(+), 103 deletions(-)

Approvals:
  Impala Public Jenkins: Looks good to me, approved; Verified

-- 
To view, visit http://gerrit.cloudera.org:8080/15088
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I17420d7ff9beacd1b4d2ad72b68b8b54983e60cb
Gerrit-Change-Number: 15088
Gerrit-PatchSet: 6
Gerrit-Owner: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>