You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Fredy Wijaya (Code Review)" <ge...@cloudera.org> on 2019/02/28 03:45:22 UTC

[Impala-ASF-CR] IMPALA-8100 (WIP): Add initial support for Ranger

Fredy Wijaya has uploaded this change for review. ( http://gerrit.cloudera.org:8080/12632


Change subject: IMPALA-8100 (WIP): Add initial support for Ranger
......................................................................

IMPALA-8100 (WIP): Add initial support for Ranger

This patch adds an initial support for Ranger that can be enabled via
the following flags to do enforcement.
- ranger_service_type=impala
- ranger_app_id=some_app_id
- authorization_factory_class=\
    org.apache.impala.authorization.ranger.RangerAuthorizationFactory

Change-Id: I8cad9e609d20aae1ff645c84fd58a02afee70276
---
M be/src/service/frontend.cc
M be/src/util/backend-gflag-util.cc
M common/thrift/BackendGflags.thrift
M fe/pom.xml
M fe/src/main/java/org/apache/impala/authorization/AuthorizationProvider.java
A fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationChecker.java
A fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationConfig.java
A fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationFactory.java
A fe/src/main/java/org/apache/impala/authorization/ranger/RangerImpalaPlugin.java
A fe/src/main/java/org/apache/impala/authorization/ranger/RangerImpalaResourceBuilder.java
M fe/src/main/java/org/apache/impala/authorization/sentry/SentryAuthorizationFactory.java
M fe/src/main/java/org/apache/impala/service/BackendConfig.java
A fe/src/test/resources/ranger-impala-audit.xml
A fe/src/test/resources/ranger-impala-security.xml
M impala-parent/pom.xml
M testdata/bin/create-load-data.sh
M testdata/cluster/.gitignore
A testdata/cluster/ranger/setup/impala_service.json
A testdata/cluster/ranger/setup/impala_servicedef.json
A testdata/cluster/ranger/setup/impala_user.json.template
20 files changed, 616 insertions(+), 1 deletion(-)



  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/32/12632/2
-- 
To view, visit http://gerrit.cloudera.org:8080/12632
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I8cad9e609d20aae1ff645c84fd58a02afee70276
Gerrit-Change-Number: 12632
Gerrit-PatchSet: 2
Gerrit-Owner: Fredy Wijaya <fw...@cloudera.com>

[Impala-ASF-CR] IMPALA-8100: Add initial support for Ranger

Posted by "Fredy Wijaya (Code Review)" <ge...@cloudera.org>.
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/12632 )

Change subject: IMPALA-8100: Add initial support for Ranger
......................................................................


Patch Set 8:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/12632/7/fe/src/test/java/org/apache/impala/common/FrontendTestBase.java
File fe/src/test/java/org/apache/impala/common/FrontendTestBase.java:

http://gerrit.cloudera.org:8080/#/c/12632/7/fe/src/test/java/org/apache/impala/common/FrontendTestBase.java@325
PS7, Line 325:       public AuthorizationChecker newAuthorizationChecker(
> line too long (92 > 90)
Done



-- 
To view, visit http://gerrit.cloudera.org:8080/12632
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I8cad9e609d20aae1ff645c84fd58a02afee70276
Gerrit-Change-Number: 12632
Gerrit-PatchSet: 8
Gerrit-Owner: Fredy Wijaya <fw...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Fredy Wijaya <fw...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Paul Rogers <pr...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Comment-Date: Wed, 06 Mar 2019 02:27:30 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-8100: Add initial support for Ranger

Posted by "Fredy Wijaya (Code Review)" <ge...@cloudera.org>.
Fredy Wijaya has uploaded a new patch set (#6). ( http://gerrit.cloudera.org:8080/12632 )

Change subject: IMPALA-8100: Add initial support for Ranger
......................................................................

IMPALA-8100: Add initial support for Ranger

This patch adds an initial support for Ranger that can be enabled via
the following flags in both impalad and catalogd to do enforcement.
- ranger_service_type=impala
- ranger_app_id=some_app_id
- authorization_factory_class=\
    org.apache.impala.authorization.ranger.RangerAuthorizationFactory

This patch updates AuthorizationStmtTest with a parameterized test that
runs the tests against Sentry and Ranger.

Testing:
- Updated AuthorizationStmtTest with Ranger
- Ran all FE tests
- Ran all E2E authorization tests

Change-Id: I8cad9e609d20aae1ff645c84fd58a02afee70276
---
M be/src/service/frontend.cc
M be/src/util/backend-gflag-util.cc
M bin/rat_exclude_files.txt
M common/thrift/BackendGflags.thrift
M fe/pom.xml
M fe/src/main/java/org/apache/impala/authorization/AuthorizableFactory.java
M fe/src/main/java/org/apache/impala/authorization/AuthorizationProvider.java
M fe/src/main/java/org/apache/impala/authorization/DefaultAuthorizableFactory.java
M fe/src/main/java/org/apache/impala/authorization/PrivilegeRequestBuilder.java
A fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationChecker.java
A fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationConfig.java
A fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationFactory.java
A fe/src/main/java/org/apache/impala/authorization/ranger/RangerImpalaPlugin.java
A fe/src/main/java/org/apache/impala/authorization/ranger/RangerImpalaResourceBuilder.java
M fe/src/main/java/org/apache/impala/authorization/sentry/SentryAuthorizableFactory.java
M fe/src/main/java/org/apache/impala/authorization/sentry/SentryAuthorizationChecker.java
M fe/src/main/java/org/apache/impala/authorization/sentry/SentryAuthorizationFactory.java
M fe/src/main/java/org/apache/impala/service/BackendConfig.java
M fe/src/test/java/org/apache/impala/analysis/AuthorizationStmtTest.java
M fe/src/test/java/org/apache/impala/authorization/sentry/SentryProxyTest.java
M fe/src/test/java/org/apache/impala/testutil/CatalogServiceTestCatalog.java
M fe/src/test/java/org/apache/impala/testutil/ImpaladTestCatalog.java
A fe/src/test/resources/ranger-impala-audit.xml
A fe/src/test/resources/ranger-impala-security.xml
M impala-parent/pom.xml
M testdata/bin/create-load-data.sh
M testdata/cluster/.gitignore
A testdata/cluster/ranger/setup/impala_policy1.json
A testdata/cluster/ranger/setup/impala_policy2.json
A testdata/cluster/ranger/setup/impala_policy3.json
A testdata/cluster/ranger/setup/impala_service.json
A testdata/cluster/ranger/setup/impala_servicedef.json
A testdata/cluster/ranger/setup/impala_user.json.template
33 files changed, 1,293 insertions(+), 246 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/32/12632/6
-- 
To view, visit http://gerrit.cloudera.org:8080/12632
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I8cad9e609d20aae1ff645c84fd58a02afee70276
Gerrit-Change-Number: 12632
Gerrit-PatchSet: 6
Gerrit-Owner: Fredy Wijaya <fw...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Fredy Wijaya <fw...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Paul Rogers <pr...@cloudera.com>

[Impala-ASF-CR] IMPALA-8100: Add initial support for Ranger

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12632 )

Change subject: IMPALA-8100: Add initial support for Ranger
......................................................................


Patch Set 15:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/2477/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/12632
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I8cad9e609d20aae1ff645c84fd58a02afee70276
Gerrit-Change-Number: 12632
Gerrit-PatchSet: 15
Gerrit-Owner: Fredy Wijaya <fw...@cloudera.com>
Gerrit-Reviewer: Austin Nobis <an...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Fredy Wijaya <fw...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Paul Rogers <pr...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Comment-Date: Wed, 20 Mar 2019 06:05:57 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-8100: Add initial support for Ranger

Posted by "Fredy Wijaya (Code Review)" <ge...@cloudera.org>.
Fredy Wijaya has uploaded a new patch set (#5). ( http://gerrit.cloudera.org:8080/12632 )

Change subject: IMPALA-8100: Add initial support for Ranger
......................................................................

IMPALA-8100: Add initial support for Ranger

This patch adds an initial support for Ranger that can be enabled via
the following flags in both impalad and catalogd to do enforcement.
- ranger_service_type=impala
- ranger_app_id=some_app_id
- authorization_factory_class=\
    org.apache.impala.authorization.ranger.RangerAuthorizationFactory

This patch updates AuthorizationStmtTest with a parameterized test that
runs the tests against Sentry and Ranger.

Testing:
- Updated AuthorizationStmtTest with Ranger
- Ran all FE tests
- Ran all E2E authorization tests

Change-Id: I8cad9e609d20aae1ff645c84fd58a02afee70276
---
M be/src/service/frontend.cc
M be/src/util/backend-gflag-util.cc
M bin/rat_exclude_files.txt
M common/thrift/BackendGflags.thrift
M fe/pom.xml
M fe/src/main/java/org/apache/impala/authorization/AuthorizableFactory.java
M fe/src/main/java/org/apache/impala/authorization/AuthorizationProvider.java
M fe/src/main/java/org/apache/impala/authorization/DefaultAuthorizableFactory.java
M fe/src/main/java/org/apache/impala/authorization/PrivilegeRequestBuilder.java
A fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationChecker.java
A fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationConfig.java
A fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationFactory.java
A fe/src/main/java/org/apache/impala/authorization/ranger/RangerImpalaPlugin.java
A fe/src/main/java/org/apache/impala/authorization/ranger/RangerImpalaResourceBuilder.java
M fe/src/main/java/org/apache/impala/authorization/sentry/SentryAuthorizableFactory.java
M fe/src/main/java/org/apache/impala/authorization/sentry/SentryAuthorizationChecker.java
M fe/src/main/java/org/apache/impala/authorization/sentry/SentryAuthorizationFactory.java
M fe/src/main/java/org/apache/impala/service/BackendConfig.java
M fe/src/test/java/org/apache/impala/analysis/AuthorizationStmtTest.java
M fe/src/test/java/org/apache/impala/authorization/sentry/SentryProxyTest.java
M fe/src/test/java/org/apache/impala/testutil/CatalogServiceTestCatalog.java
M fe/src/test/java/org/apache/impala/testutil/ImpaladTestCatalog.java
A fe/src/test/resources/ranger-impala-audit.xml
A fe/src/test/resources/ranger-impala-security.xml
M impala-parent/pom.xml
M testdata/bin/create-load-data.sh
M testdata/cluster/.gitignore
A testdata/cluster/ranger/setup/impala_policy1.json
A testdata/cluster/ranger/setup/impala_policy2.json
A testdata/cluster/ranger/setup/impala_policy3.json
A testdata/cluster/ranger/setup/impala_service.json
A testdata/cluster/ranger/setup/impala_servicedef.json
A testdata/cluster/ranger/setup/impala_user.json.template
33 files changed, 1,291 insertions(+), 246 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/32/12632/5
-- 
To view, visit http://gerrit.cloudera.org:8080/12632
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I8cad9e609d20aae1ff645c84fd58a02afee70276
Gerrit-Change-Number: 12632
Gerrit-PatchSet: 5
Gerrit-Owner: Fredy Wijaya <fw...@cloudera.com>
Gerrit-Reviewer: Fredy Wijaya <fw...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>

[Impala-ASF-CR] IMPALA-8100: Add initial support for Ranger

Posted by "Fredy Wijaya (Code Review)" <ge...@cloudera.org>.
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/12632 )

Change subject: IMPALA-8100: Add initial support for Ranger
......................................................................


Patch Set 6:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/12632/6/be/src/service/frontend.cc
File be/src/service/frontend.cc:

http://gerrit.cloudera.org:8080/#/c/12632/6/be/src/service/frontend.cc@41
PS6, Line 41:     "org.apache.impala.authorization.sentry.SentryAuthorizationFactory",
> It's a minor issue, but given that we enumerate AuthorizationProvider to be
I'm actually having a second thought whether it's a good idea to make AuthorizationProvider as an enum as opposed to a string. With a string, anyone can technically implement AuthorizationFactory without having making any modification to Impala. Hence, making the authorization truly pluggable.



-- 
To view, visit http://gerrit.cloudera.org:8080/12632
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I8cad9e609d20aae1ff645c84fd58a02afee70276
Gerrit-Change-Number: 12632
Gerrit-PatchSet: 6
Gerrit-Owner: Fredy Wijaya <fw...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Fredy Wijaya <fw...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Paul Rogers <pr...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Comment-Date: Tue, 05 Mar 2019 04:49:08 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-8100: Add initial support for Ranger

Posted by "Todd Lipcon (Code Review)" <ge...@cloudera.org>.
Todd Lipcon has posted comments on this change. ( http://gerrit.cloudera.org:8080/12632 )

Change subject: IMPALA-8100: Add initial support for Ranger
......................................................................


Patch Set 9:

(25 comments)

http://gerrit.cloudera.org:8080/#/c/12632/9//COMMIT_MSG
Commit Message:

http://gerrit.cloudera.org:8080/#/c/12632/9//COMMIT_MSG@30
PS9, Line 30: - Ran all FE tests
            : - Ran all E2E authorization tests
did you run these with ranger configured? I'm surprised they would pass since best I can tell this patch doesn't attempt to support GRANT/REVOKE statements etc, right?


http://gerrit.cloudera.org:8080/#/c/12632/9/be/src/service/frontend.cc
File be/src/service/frontend.cc:

http://gerrit.cloudera.org:8080/#/c/12632/9/be/src/service/frontend.cc@41
PS9, Line 41:     "org.apache.impala.authorization.sentry.SentryAuthorizationFactory",
I wonder if we should switch this to be a more human-friendly flag instead of a class name, before we ship it?


http://gerrit.cloudera.org:8080/#/c/12632/9/be/src/service/frontend.cc@46
PS9, Line 46:     "identify the application that communicates with Ranger.");
Worth noting whether this is required, considering it has an empty default


http://gerrit.cloudera.org:8080/#/c/12632/9/be/src/service/frontend.cc@47
PS9, Line 47: DEFINE_string(server_name, "", "The name to use for securing this impalad "
we probably can't rename the existing sentry-specific flags (for compat reasons) but perhaps we can amend the help text for each of these flags to say something like "only relevant if sentry is enabled" or something?

It might be worth hard-coding some checks to make sure GetCommandLineFlagInfoOrDie("server_name").is_default for these flags if ranger is set, and vice-versa for ranger-specific flags, so that people don't accidentally cross-pollinate their configs


http://gerrit.cloudera.org:8080/#/c/12632/9/fe/pom.xml
File fe/pom.xml:

http://gerrit.cloudera.org:8080/#/c/12632/9/fe/pom.xml@112
PS9, Line 112:       <exclusions>
it seems that ranger-plugins-common has a pom dependency on mysql-connector-java. Is that really necessary? I thought ranger used a "service" model rather than the plugin directly talking to any database?


http://gerrit.cloudera.org:8080/#/c/12632/9/fe/src/main/java/org/apache/impala/analysis/DescribeTableStmt.java
File fe/src/main/java/org/apache/impala/analysis/DescribeTableStmt.java:

http://gerrit.cloudera.org:8080/#/c/12632/9/fe/src/main/java/org/apache/impala/analysis/DescribeTableStmt.java@126
PS9, Line 126:         Privilege.ANY);
I'm not 100% sure about this part of the change. If I don't have the 'VIEW_METADATA' privilege but I do have permissions on one or more columns, should I still be able to use 'DESCRIBE' in all cases? For example, 'DESCRIBE EXTENDED' on a view will show the view text and other metadata, but maybe I'm not supposed to have that capability if I don't have VIEW_METADATA privileges?

Do we have some good documentation on the intended semantics here?


http://gerrit.cloudera.org:8080/#/c/12632/9/fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationChecker.java
File fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationChecker.java:

http://gerrit.cloudera.org:8080/#/c/12632/9/fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationChecker.java@77
PS9, Line 77:         // Any column access is special in Ranger. For example if we want to check if
            :         // we have access to any column on a particular table, we need to build a resource
            :         // without the column resource.
            :         // For example:
            :         // access type: RangerPolicyEngine.ANY_ACCESS
            :         // resources: [server=server1, database=foo, table=bar]
I'm not understanding this. When you are saying "Any column access" do you mean "authorization checks for wildcard columns" or something? I think expanding upon the example would be useful.


http://gerrit.cloudera.org:8080/#/c/12632/9/fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationChecker.java@106
PS9, Line 106:     if (request.getPrivilege() == Privilege.VIEW_METADATA) {
does this need to be special cased? It seems all the other privileges already have an "implied" set which is equal to just the privilege itself. (ANY probably does need to be special cased still)


http://gerrit.cloudera.org:8080/#/c/12632/9/fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationChecker.java@117
PS9, Line 117:       if (authorize(plugin, resource, user, privilege)) {
does this result in multiple audit events, one per implied privilege?


http://gerrit.cloudera.org:8080/#/c/12632/9/fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationChecker.java@131
PS9, Line 131: new HashSet
can you use Collections.emptySet?

Also is this a TODO? Should we be doing group resolution here? Or will Ranger fill in its own groups?


http://gerrit.cloudera.org:8080/#/c/12632/9/fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationConfig.java
File fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationConfig.java:

http://gerrit.cloudera.org:8080/#/c/12632/9/fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationConfig.java@38
PS9, Line 38: serviceType
nit: Preconditions.checkNotNull(foo) returns 'foo', so you can combine these assignments with the checks


http://gerrit.cloudera.org:8080/#/c/12632/9/fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationFactory.java
File fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationFactory.java:

http://gerrit.cloudera.org:8080/#/c/12632/9/fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationFactory.java@55
PS9, Line 55:     Preconditions.checkArgument(!Strings.isNullOrEmpty(backendConfig.getRangerAppId()));
is this the only validation that happens on this config? If so, we should make sure the exception has a clear message pointing back to the flag so the user doesn't need to go pull up the source to track down which config is throwing IllegalArgumentException


http://gerrit.cloudera.org:8080/#/c/12632/9/fe/src/main/java/org/apache/impala/authorization/ranger/RangerImpalaResourceBuilder.java
File fe/src/main/java/org/apache/impala/authorization/ranger/RangerImpalaResourceBuilder.java:

http://gerrit.cloudera.org:8080/#/c/12632/9/fe/src/main/java/org/apache/impala/authorization/ranger/RangerImpalaResourceBuilder.java@37
PS9, Line 37: serverName
can we add Preconditions.checkNotNull() for all parameters in this file? Is it valid to ever pass null?


http://gerrit.cloudera.org:8080/#/c/12632/9/fe/src/main/java/org/apache/impala/authorization/ranger/RangerImpalaResourceBuilder.java@52
PS9, Line 52:     rangerAccessResource.setValue(COLUMN, columnName);
general question about ranger semantics -- are resources case sensitive? should we be canonicalizing to lower case anywhere? Or should we be Preconditioning that its' already canonicalized? Or does that get handled on the ranger side?


http://gerrit.cloudera.org:8080/#/c/12632/9/fe/src/test/java/org/apache/impala/analysis/AuthorizationStmtTest.java
File fe/src/test/java/org/apache/impala/analysis/AuthorizationStmtTest.java:

http://gerrit.cloudera.org:8080/#/c/12632/9/fe/src/test/java/org/apache/impala/analysis/AuthorizationStmtTest.java@90
PS9, Line 90:   private static final String RANGER_SERVICE_TYPE = "impala";
            :   private static final String RANGER_APP_ID = "impala";
can these be pulled from the config instead to avoid duplication?


http://gerrit.cloudera.org:8080/#/c/12632/9/fe/src/test/java/org/apache/impala/analysis/AuthorizationStmtTest.java@110
PS9, Line 110: System.getenv("IMPALA_HOME") + "/fe/src/test/resources/sentry-site.
this is a little sketchy. Shouldn't this resource be on the classpath and thus just loadable by its name?


http://gerrit.cloudera.org:8080/#/c/12632/9/fe/src/test/java/org/apache/impala/analysis/AuthorizationTest.java
File fe/src/test/java/org/apache/impala/analysis/AuthorizationTest.java:

http://gerrit.cloudera.org:8080/#/c/12632/9/fe/src/test/java/org/apache/impala/analysis/AuthorizationTest.java@178
PS9, Line 178:     SentryAuthorizationFactory authzFactory = new SentryAuthorizationFactory(
should we be parametrizing this test to be able to run against ranger as well? Or is it too specific to Sentry?


http://gerrit.cloudera.org:8080/#/c/12632/9/fe/src/test/java/org/apache/impala/common/FrontendTestBase.java
File fe/src/test/java/org/apache/impala/common/FrontendTestBase.java:

http://gerrit.cloudera.org:8080/#/c/12632/9/fe/src/test/java/org/apache/impala/common/FrontendTestBase.java@308
PS9, Line 308:           @Override
             :           public boolean isEnabled() { return true; }
             :           @Override
             :           public AuthorizationProvider getProvider() {
             :             return AuthorizationProvider.NONE;
             :           }
I'm not sure what this means, semantically -- authorization enabled but NONE?


http://gerrit.cloudera.org:8080/#/c/12632/9/fe/src/test/resources/ranger-impala-security.xml
File fe/src/test/resources/ranger-impala-security.xml:

http://gerrit.cloudera.org:8080/#/c/12632/9/fe/src/test/resources/ranger-impala-security.xml@21
PS9, Line 21:     <value>test_impala</value>
how does this interact with the gflag for "service id"?


http://gerrit.cloudera.org:8080/#/c/12632/9/fe/src/test/resources/ranger-impala-security.xml@37
PS9, Line 37:     <value>/tmp</value>
can we auto-configure this based on other existing scratch dir configs, at least as a default?


http://gerrit.cloudera.org:8080/#/c/12632/9/fe/src/test/resources/ranger-impala-security.xml@48
PS9, Line 48:       How often to poll for changes in policies?
are these strings copied from somewhere? It's weird to see a question mark in a description.


http://gerrit.cloudera.org:8080/#/c/12632/9/fe/src/test/resources/ranger-impala-security.xml@56
PS9, Line 56:       RangerRestClient Connection Timeout in Milli Seconds.
milliseconds


http://gerrit.cloudera.org:8080/#/c/12632/9/fe/src/test/resources/ranger-impala-security.xml@64
PS9, Line 64:       RangerRestClient read Timeout in Milli Seconds.
milliseconds


http://gerrit.cloudera.org:8080/#/c/12632/9/testdata/bin/create-load-data.sh
File testdata/bin/create-load-data.sh:

http://gerrit.cloudera.org:8080/#/c/12632/9/testdata/bin/create-load-data.sh@308
PS9, Line 308:   perl -wpl -e 's/\$\{([^}]+)\}/defined $ENV{$1} ? $ENV{$1} : $&/eg' \
you don't want to die() if $ENV{$1} is not defined here? leaving it unsubstituted seems like it'll just make a harder-to-troubleshoot error later


http://gerrit.cloudera.org:8080/#/c/12632/9/testdata/cluster/ranger/setup/impala_servicedef.json
File testdata/cluster/ranger/setup/impala_servicedef.json:

http://gerrit.cloudera.org:8080/#/c/12632/9/testdata/cluster/ranger/setup/impala_servicedef.json@1
PS9, Line 1: {
I'm a little curious, at a high level, how this interacts with any ranger polciies defined for hive. Shouldn't we have a single service definition which applies to both hive and impala, rather than our own service def?

Also, what's the eventual deployment model? This is in our test data, but in real life we'd expect these files to be deployed onto the ranger server? Maybe a README file in this directory would be useful.



-- 
To view, visit http://gerrit.cloudera.org:8080/12632
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I8cad9e609d20aae1ff645c84fd58a02afee70276
Gerrit-Change-Number: 12632
Gerrit-PatchSet: 9
Gerrit-Owner: Fredy Wijaya <fw...@cloudera.com>
Gerrit-Reviewer: Austin Nobis <an...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Fredy Wijaya <fw...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Paul Rogers <pr...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Comment-Date: Wed, 13 Mar 2019 22:23:10 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-8100: Add initial support for Ranger

Posted by "Fredy Wijaya (Code Review)" <ge...@cloudera.org>.
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/12632 )

Change subject: IMPALA-8100: Add initial support for Ranger
......................................................................


Patch Set 15:

Todd, as discussed offline, I updated the CR to use Hive service definition instead of creating a new one for Impala to allow sharing policies between Hive and Impala. Please take a look at this CR again.


-- 
To view, visit http://gerrit.cloudera.org:8080/12632
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I8cad9e609d20aae1ff645c84fd58a02afee70276
Gerrit-Change-Number: 12632
Gerrit-PatchSet: 15
Gerrit-Owner: Fredy Wijaya <fw...@cloudera.com>
Gerrit-Reviewer: Austin Nobis <an...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Fredy Wijaya <fw...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Paul Rogers <pr...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Comment-Date: Wed, 20 Mar 2019 05:22:31 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-8100: Add initial support for Ranger

Posted by "Fredy Wijaya (Code Review)" <ge...@cloudera.org>.
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/12632 )

Change subject: IMPALA-8100: Add initial support for Ranger
......................................................................


Patch Set 9:

Rebased. Paul/Bharath, this is ready for review. Can you take a look at it?


-- 
To view, visit http://gerrit.cloudera.org:8080/12632
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I8cad9e609d20aae1ff645c84fd58a02afee70276
Gerrit-Change-Number: 12632
Gerrit-PatchSet: 9
Gerrit-Owner: Fredy Wijaya <fw...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Fredy Wijaya <fw...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Paul Rogers <pr...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Comment-Date: Thu, 07 Mar 2019 16:25:19 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-8100: Add initial support for Ranger

Posted by "Philip Zeyliger (Code Review)" <ge...@cloudera.org>.
Philip Zeyliger has posted comments on this change. ( http://gerrit.cloudera.org:8080/12632 )

Change subject: IMPALA-8100: Add initial support for Ranger
......................................................................


Patch Set 6:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/12632/6/be/src/service/frontend.cc
File be/src/service/frontend.cc:

http://gerrit.cloudera.org:8080/#/c/12632/6/be/src/service/frontend.cc@41
PS6, Line 41:     "org.apache.impala.authorization.sentry.SentryAuthorizationFactory",
It's a minor issue, but given that we enumerate AuthorizationProvider to be either SENTRY, RANGER, or NONE, does this really need to be a class? Could it be "--athorization_provider [sentry|ranger|none]".



-- 
To view, visit http://gerrit.cloudera.org:8080/12632
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I8cad9e609d20aae1ff645c84fd58a02afee70276
Gerrit-Change-Number: 12632
Gerrit-PatchSet: 6
Gerrit-Owner: Fredy Wijaya <fw...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Fredy Wijaya <fw...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Paul Rogers <pr...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Comment-Date: Tue, 05 Mar 2019 03:57:49 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-8100: Add initial support for Ranger

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12632 )

Change subject: IMPALA-8100: Add initial support for Ranger
......................................................................


Patch Set 18:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/3936/ DRY_RUN=false


-- 
To view, visit http://gerrit.cloudera.org:8080/12632
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I8cad9e609d20aae1ff645c84fd58a02afee70276
Gerrit-Change-Number: 12632
Gerrit-PatchSet: 18
Gerrit-Owner: Fredy Wijaya <fw...@cloudera.com>
Gerrit-Reviewer: Austin Nobis <an...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Fredy Wijaya <fw...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Paul Rogers <pr...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Comment-Date: Thu, 21 Mar 2019 05:38:34 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-8100: Add initial support for Ranger

Posted by "Fredy Wijaya (Code Review)" <ge...@cloudera.org>.
Fredy Wijaya has uploaded a new patch set (#9). ( http://gerrit.cloudera.org:8080/12632 )

Change subject: IMPALA-8100: Add initial support for Ranger
......................................................................

IMPALA-8100: Add initial support for Ranger

This patch adds an initial support for Ranger that can be enabled via
the following flags in both impalad and catalogd to do enforcement.
- ranger_service_type=impala
- ranger_app_id=some_app_id
- authorization_factory_class=\
    org.apache.impala.authorization.ranger.RangerAuthorizationFactory

There's a change in DESCRIBE <table> privilege requirement to use ANY
privilege instead of VIEW_METADATA privilege as the first-level check
to play nicely with Ranger. This is not a security risk since the
column-level filtering logic after the first-level check will use
VIEW_METADATA privilege to filter out unauthorized column access. In
other words, DESCRIBE <table> may return an empty result instead of
an authorization error as long as there exists any privilege in the
given table.

This patch updates AuthorizationStmtTest with a parameterized test that
runs the tests against Sentry and Ranger.

Testing:
- Updated AuthorizationStmtTest with Ranger
- Ran all FE tests
- Ran all E2E authorization tests

Change-Id: I8cad9e609d20aae1ff645c84fd58a02afee70276
---
M be/src/service/frontend.cc
M be/src/util/backend-gflag-util.cc
M bin/rat_exclude_files.txt
M common/thrift/BackendGflags.thrift
M fe/pom.xml
M fe/src/main/java/org/apache/impala/analysis/DescribeTableStmt.java
M fe/src/main/java/org/apache/impala/authorization/AuthorizationProvider.java
M fe/src/main/java/org/apache/impala/authorization/DefaultAuthorizableFactory.java
M fe/src/main/java/org/apache/impala/authorization/Privilege.java
A fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationChecker.java
A fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationConfig.java
A fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationFactory.java
A fe/src/main/java/org/apache/impala/authorization/ranger/RangerImpalaPlugin.java
A fe/src/main/java/org/apache/impala/authorization/ranger/RangerImpalaResourceBuilder.java
M fe/src/main/java/org/apache/impala/authorization/sentry/SentryAuthorizationChecker.java
M fe/src/main/java/org/apache/impala/authorization/sentry/SentryAuthorizationFactory.java
M fe/src/main/java/org/apache/impala/service/BackendConfig.java
M fe/src/test/java/org/apache/impala/analysis/AnalyzerTest.java
M fe/src/test/java/org/apache/impala/analysis/AuditingTest.java
M fe/src/test/java/org/apache/impala/analysis/AuthorizationStmtTest.java
M fe/src/test/java/org/apache/impala/analysis/AuthorizationTest.java
M fe/src/test/java/org/apache/impala/analysis/ToSqlTest.java
M fe/src/test/java/org/apache/impala/authorization/sentry/SentryProxyTest.java
M fe/src/test/java/org/apache/impala/common/FrontendFixture.java
M fe/src/test/java/org/apache/impala/common/FrontendTestBase.java
M fe/src/test/java/org/apache/impala/testutil/CatalogServiceTestCatalog.java
M fe/src/test/java/org/apache/impala/testutil/ImpaladTestCatalog.java
A fe/src/test/resources/ranger-impala-audit.xml
A fe/src/test/resources/ranger-impala-security.xml
M impala-parent/pom.xml
M testdata/bin/create-load-data.sh
M testdata/cluster/.gitignore
A testdata/cluster/ranger/setup/impala_policy1.json
A testdata/cluster/ranger/setup/impala_policy2.json
A testdata/cluster/ranger/setup/impala_policy3.json
A testdata/cluster/ranger/setup/impala_service.json
A testdata/cluster/ranger/setup/impala_servicedef.json
A testdata/cluster/ranger/setup/impala_user.json.template
38 files changed, 1,378 insertions(+), 223 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/32/12632/9
-- 
To view, visit http://gerrit.cloudera.org:8080/12632
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I8cad9e609d20aae1ff645c84fd58a02afee70276
Gerrit-Change-Number: 12632
Gerrit-PatchSet: 9
Gerrit-Owner: Fredy Wijaya <fw...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Fredy Wijaya <fw...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Paul Rogers <pr...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>

[Impala-ASF-CR] IMPALA-8100: Add initial support for Ranger

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12632 )

Change subject: IMPALA-8100: Add initial support for Ranger
......................................................................


Patch Set 8:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/2370/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/12632
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I8cad9e609d20aae1ff645c84fd58a02afee70276
Gerrit-Change-Number: 12632
Gerrit-PatchSet: 8
Gerrit-Owner: Fredy Wijaya <fw...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Fredy Wijaya <fw...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Paul Rogers <pr...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Comment-Date: Wed, 06 Mar 2019 03:03:05 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-8100: Add initial support for Ranger

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12632 )

Change subject: IMPALA-8100: Add initial support for Ranger
......................................................................


Patch Set 7:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/2369/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/12632
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I8cad9e609d20aae1ff645c84fd58a02afee70276
Gerrit-Change-Number: 12632
Gerrit-PatchSet: 7
Gerrit-Owner: Fredy Wijaya <fw...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Fredy Wijaya <fw...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Paul Rogers <pr...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Comment-Date: Wed, 06 Mar 2019 02:51:59 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-8100: Add initial support for Ranger

Posted by "Paul Rogers (Code Review)" <ge...@cloudera.org>.
Paul Rogers has posted comments on this change. ( http://gerrit.cloudera.org:8080/12632 )

Change subject: IMPALA-8100: Add initial support for Ranger
......................................................................


Patch Set 9: Code-Review+1

Really like how this turned out. Very clean. Would like Bharath to look at the details of semantics.


-- 
To view, visit http://gerrit.cloudera.org:8080/12632
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I8cad9e609d20aae1ff645c84fd58a02afee70276
Gerrit-Change-Number: 12632
Gerrit-PatchSet: 9
Gerrit-Owner: Fredy Wijaya <fw...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Fredy Wijaya <fw...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Paul Rogers <pr...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Comment-Date: Thu, 07 Mar 2019 19:57:57 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-8100: Add initial support for Ranger

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12632 )

Change subject: IMPALA-8100: Add initial support for Ranger
......................................................................


Patch Set 20:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/3939/ DRY_RUN=false


-- 
To view, visit http://gerrit.cloudera.org:8080/12632
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I8cad9e609d20aae1ff645c84fd58a02afee70276
Gerrit-Change-Number: 12632
Gerrit-PatchSet: 20
Gerrit-Owner: Fredy Wijaya <fw...@cloudera.com>
Gerrit-Reviewer: Austin Nobis <an...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Fredy Wijaya <fw...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Paul Rogers <pr...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Comment-Date: Thu, 21 Mar 2019 14:26:57 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-8100: Add initial support for Ranger

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12632 )

Change subject: IMPALA-8100: Add initial support for Ranger
......................................................................


Patch Set 18: Verified-1

Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/3936/


-- 
To view, visit http://gerrit.cloudera.org:8080/12632
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I8cad9e609d20aae1ff645c84fd58a02afee70276
Gerrit-Change-Number: 12632
Gerrit-PatchSet: 18
Gerrit-Owner: Fredy Wijaya <fw...@cloudera.com>
Gerrit-Reviewer: Austin Nobis <an...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Fredy Wijaya <fw...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Paul Rogers <pr...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Comment-Date: Thu, 21 Mar 2019 06:59:09 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-8100: Add initial support for Ranger

Posted by "Fredy Wijaya (Code Review)" <ge...@cloudera.org>.
Fredy Wijaya has uploaded a new patch set (#17). ( http://gerrit.cloudera.org:8080/12632 )

Change subject: IMPALA-8100: Add initial support for Ranger
......................................................................

IMPALA-8100: Add initial support for Ranger

This patch adds an initial support for Ranger that can be enabled via
the following flags in both impalad and catalogd to do enforcement.
- ranger_service_type=hive
- ranger_app_id=some_app_id
- authorization_factory_class=\
    org.apache.impala.authorization.ranger.RangerAuthorizationFactory

The Ranger plugin for Impala uses Hive service definition to allow
sharing Ranger policies between Hive and Impala. Temporarily the REFRESH
privilege uses "read" access type and it will be updated in the later
patch once Ranger supports "refresh" access type.

There's a change in DESCRIBE <table> privilege requirement to use ANY
privilege instead of VIEW_METADATA privilege as the first-level check
to play nicely with Ranger. This is not a security risk since the
column-level filtering logic after the first-level check will use
VIEW_METADATA privilege to filter out unauthorized column access. In
other words, DESCRIBE <table> may return an empty result instead of
an authorization error as long as there exists any privilege in the
given table.

This patch updates AuthorizationStmtTest with a parameterized test that
runs the tests against Sentry and Ranger.

Testing:
- Updated AuthorizationStmtTest with Ranger
- Ran all FE tests
- Ran all E2E authorization tests

Change-Id: I8cad9e609d20aae1ff645c84fd58a02afee70276
---
M be/src/service/frontend.cc
M be/src/util/backend-gflag-util.cc
M bin/rat_exclude_files.txt
M common/thrift/BackendGflags.thrift
M fe/pom.xml
M fe/src/main/java/org/apache/impala/analysis/DescribeTableStmt.java
M fe/src/main/java/org/apache/impala/authorization/AuthorizationChecker.java
M fe/src/main/java/org/apache/impala/authorization/AuthorizationPolicy.java
M fe/src/main/java/org/apache/impala/authorization/AuthorizationProvider.java
M fe/src/main/java/org/apache/impala/authorization/DefaultAuthorizableFactory.java
M fe/src/main/java/org/apache/impala/authorization/NoneAuthorizationFactory.java
M fe/src/main/java/org/apache/impala/authorization/Privilege.java
A fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationChecker.java
A fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationConfig.java
A fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationFactory.java
A fe/src/main/java/org/apache/impala/authorization/ranger/RangerImpalaPlugin.java
A fe/src/main/java/org/apache/impala/authorization/ranger/RangerImpalaResourceBuilder.java
M fe/src/main/java/org/apache/impala/authorization/sentry/SentryAuthorizationChecker.java
M fe/src/main/java/org/apache/impala/authorization/sentry/SentryAuthorizationFactory.java
M fe/src/main/java/org/apache/impala/service/BackendConfig.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M fe/src/test/java/org/apache/impala/analysis/AnalyzerTest.java
M fe/src/test/java/org/apache/impala/analysis/AuditingTest.java
M fe/src/test/java/org/apache/impala/analysis/AuthorizationStmtTest.java
M fe/src/test/java/org/apache/impala/analysis/AuthorizationTest.java
M fe/src/test/java/org/apache/impala/analysis/ToSqlTest.java
M fe/src/test/java/org/apache/impala/authorization/sentry/SentryProxyTest.java
M fe/src/test/java/org/apache/impala/common/FrontendFixture.java
M fe/src/test/java/org/apache/impala/common/FrontendTestBase.java
M fe/src/test/java/org/apache/impala/testutil/CatalogServiceTestCatalog.java
M fe/src/test/java/org/apache/impala/testutil/ImpaladTestCatalog.java
A fe/src/test/resources/ranger-hive-audit.xml
A fe/src/test/resources/ranger-hive-security.xml
M impala-parent/pom.xml
M testdata/bin/create-load-data.sh
M testdata/cluster/.gitignore
A testdata/cluster/ranger/setup/impala_service.json
A testdata/cluster/ranger/setup/impala_user.json.template
38 files changed, 1,131 insertions(+), 237 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/32/12632/17
-- 
To view, visit http://gerrit.cloudera.org:8080/12632
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I8cad9e609d20aae1ff645c84fd58a02afee70276
Gerrit-Change-Number: 12632
Gerrit-PatchSet: 17
Gerrit-Owner: Fredy Wijaya <fw...@cloudera.com>
Gerrit-Reviewer: Austin Nobis <an...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Fredy Wijaya <fw...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Paul Rogers <pr...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>

[Impala-ASF-CR] IMPALA-8100: Add initial support for Ranger

Posted by "Fredy Wijaya (Code Review)" <ge...@cloudera.org>.
Fredy Wijaya has uploaded a new patch set (#11). ( http://gerrit.cloudera.org:8080/12632 )

Change subject: IMPALA-8100: Add initial support for Ranger
......................................................................

IMPALA-8100: Add initial support for Ranger

This patch adds an initial support for Ranger that can be enabled via
the following flags in both impalad and catalogd to do enforcement.
- ranger_service_type=impala
- ranger_app_id=some_app_id
- authorization_factory_class=\
    org.apache.impala.authorization.ranger.RangerAuthorizationFactory

There's a change in DESCRIBE <table> privilege requirement to use ANY
privilege instead of VIEW_METADATA privilege as the first-level check
to play nicely with Ranger. This is not a security risk since the
column-level filtering logic after the first-level check will use
VIEW_METADATA privilege to filter out unauthorized column access. In
other words, DESCRIBE <table> may return an empty result instead of
an authorization error as long as there exists any privilege in the
given table.

This patch updates AuthorizationStmtTest with a parameterized test that
runs the tests against Sentry and Ranger.

Testing:
- Updated AuthorizationStmtTest with Ranger
- Ran all FE tests
- Ran all E2E authorization tests

Change-Id: I8cad9e609d20aae1ff645c84fd58a02afee70276
---
M be/src/service/frontend.cc
M be/src/util/backend-gflag-util.cc
M bin/rat_exclude_files.txt
M common/thrift/BackendGflags.thrift
M fe/pom.xml
M fe/src/main/java/org/apache/impala/analysis/DescribeTableStmt.java
M fe/src/main/java/org/apache/impala/authorization/AuthorizationChecker.java
M fe/src/main/java/org/apache/impala/authorization/AuthorizationPolicy.java
M fe/src/main/java/org/apache/impala/authorization/AuthorizationProvider.java
M fe/src/main/java/org/apache/impala/authorization/DefaultAuthorizableFactory.java
M fe/src/main/java/org/apache/impala/authorization/NoneAuthorizationFactory.java
M fe/src/main/java/org/apache/impala/authorization/Privilege.java
A fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationChecker.java
A fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationConfig.java
A fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationFactory.java
A fe/src/main/java/org/apache/impala/authorization/ranger/RangerImpalaPlugin.java
A fe/src/main/java/org/apache/impala/authorization/ranger/RangerImpalaResourceBuilder.java
M fe/src/main/java/org/apache/impala/authorization/sentry/SentryAuthorizationChecker.java
M fe/src/main/java/org/apache/impala/authorization/sentry/SentryAuthorizationFactory.java
M fe/src/main/java/org/apache/impala/service/BackendConfig.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M fe/src/test/java/org/apache/impala/analysis/AnalyzerTest.java
M fe/src/test/java/org/apache/impala/analysis/AuditingTest.java
M fe/src/test/java/org/apache/impala/analysis/AuthorizationStmtTest.java
M fe/src/test/java/org/apache/impala/analysis/AuthorizationTest.java
M fe/src/test/java/org/apache/impala/analysis/ToSqlTest.java
M fe/src/test/java/org/apache/impala/authorization/sentry/SentryProxyTest.java
M fe/src/test/java/org/apache/impala/common/FrontendFixture.java
M fe/src/test/java/org/apache/impala/common/FrontendTestBase.java
M fe/src/test/java/org/apache/impala/testutil/CatalogServiceTestCatalog.java
M fe/src/test/java/org/apache/impala/testutil/ImpaladTestCatalog.java
A fe/src/test/resources/ranger-impala-audit.xml
A fe/src/test/resources/ranger-impala-security.xml
M impala-parent/pom.xml
M testdata/bin/create-load-data.sh
M testdata/cluster/.gitignore
A testdata/cluster/ranger/setup/impala_policy1.json
A testdata/cluster/ranger/setup/impala_policy2.json
A testdata/cluster/ranger/setup/impala_policy3.json
A testdata/cluster/ranger/setup/impala_service.json
A testdata/cluster/ranger/setup/impala_servicedef.json
A testdata/cluster/ranger/setup/impala_user.json.template
42 files changed, 1,461 insertions(+), 237 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/32/12632/11
-- 
To view, visit http://gerrit.cloudera.org:8080/12632
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I8cad9e609d20aae1ff645c84fd58a02afee70276
Gerrit-Change-Number: 12632
Gerrit-PatchSet: 11
Gerrit-Owner: Fredy Wijaya <fw...@cloudera.com>
Gerrit-Reviewer: Austin Nobis <an...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Fredy Wijaya <fw...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Paul Rogers <pr...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>

[Impala-ASF-CR] IMPALA-8100: Add initial support for Ranger

Posted by "Fredy Wijaya (Code Review)" <ge...@cloudera.org>.
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/12632 )

Change subject: IMPALA-8100: Add initial support for Ranger
......................................................................


Patch Set 13:

(3 comments)

http://gerrit.cloudera.org:8080/#/c/12632/9/fe/src/main/java/org/apache/impala/analysis/DescribeTableStmt.java
File fe/src/main/java/org/apache/impala/analysis/DescribeTableStmt.java:

http://gerrit.cloudera.org:8080/#/c/12632/9/fe/src/main/java/org/apache/impala/analysis/DescribeTableStmt.java@126
PS9, Line 126:         Privilege.ANY);
> got it. I missed that that JIRA had already handled making sure that view t
We don't have E2E test for that, but we have a test that basically tests the output of describe: https://gerrit.cloudera.org/c/12632/12/fe/src/test/java/org/apache/impala/analysis/AuthorizationStmtTest.java#1384


http://gerrit.cloudera.org:8080/#/c/12632/9/fe/src/test/resources/ranger-impala-security.xml
File fe/src/test/resources/ranger-impala-security.xml:

http://gerrit.cloudera.org:8080/#/c/12632/9/fe/src/test/resources/ranger-impala-security.xml@37
PS9, Line 37:     <!-- TODO: use scratch dir -->
> I think picking the first directory is a reasonable default. Feel free to a
Sure. TODO added.


http://gerrit.cloudera.org:8080/#/c/12632/9/testdata/cluster/ranger/setup/impala_servicedef.json
File testdata/cluster/ranger/setup/impala_servicedef.json:

http://gerrit.cloudera.org:8080/#/c/12632/9/testdata/cluster/ranger/setup/impala_servicedef.json@1
PS9, Line 1: {
> OK. That's ifne to start with this for testing, but I strongly believe we d
Yup, I will definitely take note of it.



-- 
To view, visit http://gerrit.cloudera.org:8080/12632
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I8cad9e609d20aae1ff645c84fd58a02afee70276
Gerrit-Change-Number: 12632
Gerrit-PatchSet: 13
Gerrit-Owner: Fredy Wijaya <fw...@cloudera.com>
Gerrit-Reviewer: Austin Nobis <an...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Fredy Wijaya <fw...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Paul Rogers <pr...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Comment-Date: Fri, 15 Mar 2019 15:33:38 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-8100: Add initial support for Ranger

Posted by "Fredy Wijaya (Code Review)" <ge...@cloudera.org>.
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/12632 )

Change subject: IMPALA-8100: Add initial support for Ranger
......................................................................


Patch Set 5:

This is ready for review now. It depends on this CR: https://gerrit.cloudera.org/c/12542/


-- 
To view, visit http://gerrit.cloudera.org:8080/12632
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I8cad9e609d20aae1ff645c84fd58a02afee70276
Gerrit-Change-Number: 12632
Gerrit-PatchSet: 5
Gerrit-Owner: Fredy Wijaya <fw...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Fredy Wijaya <fw...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Paul Rogers <pr...@cloudera.com>
Gerrit-Comment-Date: Tue, 05 Mar 2019 03:27:17 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-8100: Add initial support for Ranger

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12632 )

Change subject: IMPALA-8100: Add initial support for Ranger
......................................................................


Patch Set 19:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/2501/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/12632
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I8cad9e609d20aae1ff645c84fd58a02afee70276
Gerrit-Change-Number: 12632
Gerrit-PatchSet: 19
Gerrit-Owner: Fredy Wijaya <fw...@cloudera.com>
Gerrit-Reviewer: Austin Nobis <an...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Fredy Wijaya <fw...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Paul Rogers <pr...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Comment-Date: Thu, 21 Mar 2019 15:06:46 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-8100: Add initial support for Ranger

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12632 )

Change subject: IMPALA-8100: Add initial support for Ranger
......................................................................


Patch Set 5:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/2350/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/12632
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I8cad9e609d20aae1ff645c84fd58a02afee70276
Gerrit-Change-Number: 12632
Gerrit-PatchSet: 5
Gerrit-Owner: Fredy Wijaya <fw...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Fredy Wijaya <fw...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Paul Rogers <pr...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Comment-Date: Tue, 05 Mar 2019 04:07:52 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-8100: Add initial support for Ranger

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12632 )

Change subject: IMPALA-8100: Add initial support for Ranger
......................................................................


Patch Set 16:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/2478/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/12632
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I8cad9e609d20aae1ff645c84fd58a02afee70276
Gerrit-Change-Number: 12632
Gerrit-PatchSet: 16
Gerrit-Owner: Fredy Wijaya <fw...@cloudera.com>
Gerrit-Reviewer: Austin Nobis <an...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Fredy Wijaya <fw...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Paul Rogers <pr...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Comment-Date: Wed, 20 Mar 2019 07:27:52 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-8100: Add initial support for Ranger

Posted by "Fredy Wijaya (Code Review)" <ge...@cloudera.org>.
Fredy Wijaya has uploaded a new patch set (#15). ( http://gerrit.cloudera.org:8080/12632 )

Change subject: IMPALA-8100: Add initial support for Ranger
......................................................................

IMPALA-8100: Add initial support for Ranger

This patch adds an initial support for Ranger that can be enabled via
the following flags in both impalad and catalogd to do enforcement.
- ranger_service_type=hive
- ranger_app_id=some_app_id
- authorization_factory_class=\
    org.apache.impala.authorization.ranger.RangerAuthorizationFactory

The Ranger plugin for Impala uses Hive service definition to allow
sharing Ranger policies between Hive and Impala. Temporarily the REFRESH
privilege uses "read" access type and it will be updated in the later
patch once Ranger supports "refresh" access type.

There's a change in DESCRIBE <table> privilege requirement to use ANY
privilege instead of VIEW_METADATA privilege as the first-level check
to play nicely with Ranger. This is not a security risk since the
column-level filtering logic after the first-level check will use
VIEW_METADATA privilege to filter out unauthorized column access. In
other words, DESCRIBE <table> may return an empty result instead of
an authorization error as long as there exists any privilege in the
given table.

This patch updates AuthorizationStmtTest with a parameterized test that
runs the tests against Sentry and Ranger.

Testing:
- Updated AuthorizationStmtTest with Ranger
- Ran all FE tests
- Ran all E2E authorization tests

Change-Id: I8cad9e609d20aae1ff645c84fd58a02afee70276
---
M be/src/service/frontend.cc
M be/src/util/backend-gflag-util.cc
M bin/rat_exclude_files.txt
M common/thrift/BackendGflags.thrift
M fe/pom.xml
M fe/src/main/java/org/apache/impala/analysis/DescribeTableStmt.java
M fe/src/main/java/org/apache/impala/authorization/AuthorizationChecker.java
M fe/src/main/java/org/apache/impala/authorization/AuthorizationPolicy.java
M fe/src/main/java/org/apache/impala/authorization/AuthorizationProvider.java
M fe/src/main/java/org/apache/impala/authorization/DefaultAuthorizableFactory.java
M fe/src/main/java/org/apache/impala/authorization/NoneAuthorizationFactory.java
M fe/src/main/java/org/apache/impala/authorization/Privilege.java
A fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationChecker.java
A fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationConfig.java
A fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationFactory.java
A fe/src/main/java/org/apache/impala/authorization/ranger/RangerImpalaPlugin.java
A fe/src/main/java/org/apache/impala/authorization/ranger/RangerImpalaResourceBuilder.java
M fe/src/main/java/org/apache/impala/authorization/sentry/SentryAuthorizationChecker.java
M fe/src/main/java/org/apache/impala/authorization/sentry/SentryAuthorizationFactory.java
M fe/src/main/java/org/apache/impala/service/BackendConfig.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M fe/src/test/java/org/apache/impala/analysis/AnalyzerTest.java
M fe/src/test/java/org/apache/impala/analysis/AuditingTest.java
M fe/src/test/java/org/apache/impala/analysis/AuthorizationStmtTest.java
M fe/src/test/java/org/apache/impala/analysis/AuthorizationTest.java
M fe/src/test/java/org/apache/impala/analysis/ToSqlTest.java
M fe/src/test/java/org/apache/impala/authorization/sentry/SentryProxyTest.java
M fe/src/test/java/org/apache/impala/common/FrontendFixture.java
M fe/src/test/java/org/apache/impala/common/FrontendTestBase.java
M fe/src/test/java/org/apache/impala/testutil/CatalogServiceTestCatalog.java
M fe/src/test/java/org/apache/impala/testutil/ImpaladTestCatalog.java
A fe/src/test/resources/ranger-hive-audit.xml
A fe/src/test/resources/ranger-hive-security.xml
M impala-parent/pom.xml
M testdata/bin/create-load-data.sh
M testdata/cluster/.gitignore
A testdata/cluster/ranger/setup/impala_service.json
A testdata/cluster/ranger/setup/impala_user.json.template
38 files changed, 1,131 insertions(+), 238 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/32/12632/15
-- 
To view, visit http://gerrit.cloudera.org:8080/12632
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I8cad9e609d20aae1ff645c84fd58a02afee70276
Gerrit-Change-Number: 12632
Gerrit-PatchSet: 15
Gerrit-Owner: Fredy Wijaya <fw...@cloudera.com>
Gerrit-Reviewer: Austin Nobis <an...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Fredy Wijaya <fw...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Paul Rogers <pr...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>

[Impala-ASF-CR] IMPALA-8100: Add initial support for Ranger

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12632 )

Change subject: IMPALA-8100: Add initial support for Ranger
......................................................................


Patch Set 5:

(4 comments)

http://gerrit.cloudera.org:8080/#/c/12632/5/fe/src/test/java/org/apache/impala/analysis/AuthorizationStmtTest.java
File fe/src/test/java/org/apache/impala/analysis/AuthorizationStmtTest.java:

http://gerrit.cloudera.org:8080/#/c/12632/5/fe/src/test/java/org/apache/impala/analysis/AuthorizationStmtTest.java@2404
PS5, Line 2404:               .error(accessError(true, "functional.alltypes_view"), onDatabase("functional",
line too long (92 > 90)


http://gerrit.cloudera.org:8080/#/c/12632/5/fe/src/test/java/org/apache/impala/analysis/AuthorizationStmtTest.java@2411
PS5, Line 2411:               .error(accessError(true, "functional.alltypes_view"), onDatabase("functional",
line too long (92 > 90)


http://gerrit.cloudera.org:8080/#/c/12632/5/fe/src/test/java/org/apache/impala/analysis/AuthorizationStmtTest.java@2414
PS5, Line 2414:                   "alltypes_view", allExcept(TPrivilegeLevel.ALL, TPrivilegeLevel.OWNER)));
line too long (91 > 90)


http://gerrit.cloudera.org:8080/#/c/12632/5/fe/src/test/java/org/apache/impala/analysis/AuthorizationStmtTest.java@2445
PS5, Line 2445:           authorize(String.format("alter database functional set owner %s foo", ownerType))
line too long (91 > 90)



-- 
To view, visit http://gerrit.cloudera.org:8080/12632
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I8cad9e609d20aae1ff645c84fd58a02afee70276
Gerrit-Change-Number: 12632
Gerrit-PatchSet: 5
Gerrit-Owner: Fredy Wijaya <fw...@cloudera.com>
Gerrit-Reviewer: Fredy Wijaya <fw...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Tue, 05 Mar 2019 03:27:02 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-8100 (WIP): Add initial support for Ranger

Posted by "Fredy Wijaya (Code Review)" <ge...@cloudera.org>.
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/12632 )

Change subject: IMPALA-8100 (WIP): Add initial support for Ranger
......................................................................


Patch Set 3:

Make the rat happy.


-- 
To view, visit http://gerrit.cloudera.org:8080/12632
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I8cad9e609d20aae1ff645c84fd58a02afee70276
Gerrit-Change-Number: 12632
Gerrit-PatchSet: 3
Gerrit-Owner: Fredy Wijaya <fw...@cloudera.com>
Gerrit-Reviewer: Fredy Wijaya <fw...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Thu, 28 Feb 2019 04:41:52 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-8100: Add initial support for Ranger

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12632 )

Change subject: IMPALA-8100: Add initial support for Ranger
......................................................................


Patch Set 6:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/2351/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/12632
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I8cad9e609d20aae1ff645c84fd58a02afee70276
Gerrit-Change-Number: 12632
Gerrit-PatchSet: 6
Gerrit-Owner: Fredy Wijaya <fw...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Fredy Wijaya <fw...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Paul Rogers <pr...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Comment-Date: Tue, 05 Mar 2019 04:09:58 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-8100: Add initial support for Ranger

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12632 )

Change subject: IMPALA-8100: Add initial support for Ranger
......................................................................


Patch Set 20: Code-Review+2


-- 
To view, visit http://gerrit.cloudera.org:8080/12632
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I8cad9e609d20aae1ff645c84fd58a02afee70276
Gerrit-Change-Number: 12632
Gerrit-PatchSet: 20
Gerrit-Owner: Fredy Wijaya <fw...@cloudera.com>
Gerrit-Reviewer: Austin Nobis <an...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Fredy Wijaya <fw...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Paul Rogers <pr...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Comment-Date: Thu, 21 Mar 2019 14:26:56 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-8100: Add initial support for Ranger

Posted by "Fredy Wijaya (Code Review)" <ge...@cloudera.org>.
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/12632 )

Change subject: IMPALA-8100: Add initial support for Ranger
......................................................................


Patch Set 6:

(4 comments)

http://gerrit.cloudera.org:8080/#/c/12632/5/fe/src/test/java/org/apache/impala/analysis/AuthorizationStmtTest.java
File fe/src/test/java/org/apache/impala/analysis/AuthorizationStmtTest.java:

http://gerrit.cloudera.org:8080/#/c/12632/5/fe/src/test/java/org/apache/impala/analysis/AuthorizationStmtTest.java@2404
PS5, Line 2404:               .error(accessError(true, "functional.alltypes_view"), onDatabase(
> line too long (92 > 90)
Done


http://gerrit.cloudera.org:8080/#/c/12632/5/fe/src/test/java/org/apache/impala/analysis/AuthorizationStmtTest.java@2411
PS5, Line 2411:               .error(accessError(true, "functional.alltypes_view"), onDatabase(
> line too long (92 > 90)
Done


http://gerrit.cloudera.org:8080/#/c/12632/5/fe/src/test/java/org/apache/impala/analysis/AuthorizationStmtTest.java@2414
PS5, Line 2414:                   "alltypes_view", allExcept(TPrivilegeLevel.ALL,
> line too long (91 > 90)
Done


http://gerrit.cloudera.org:8080/#/c/12632/5/fe/src/test/java/org/apache/impala/analysis/AuthorizationStmtTest.java@2445
PS5, Line 2445:         for (String ownerType : new String[]{"user", "role"}) {
> line too long (91 > 90)
Done



-- 
To view, visit http://gerrit.cloudera.org:8080/12632
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I8cad9e609d20aae1ff645c84fd58a02afee70276
Gerrit-Change-Number: 12632
Gerrit-PatchSet: 6
Gerrit-Owner: Fredy Wijaya <fw...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Fredy Wijaya <fw...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Paul Rogers <pr...@cloudera.com>
Gerrit-Comment-Date: Tue, 05 Mar 2019 03:29:26 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-8100: Add initial support for Ranger

Posted by "Fredy Wijaya (Code Review)" <ge...@cloudera.org>.
Fredy Wijaya has uploaded a new patch set (#16). ( http://gerrit.cloudera.org:8080/12632 )

Change subject: IMPALA-8100: Add initial support for Ranger
......................................................................

IMPALA-8100: Add initial support for Ranger

This patch adds an initial support for Ranger that can be enabled via
the following flags in both impalad and catalogd to do enforcement.
- ranger_service_type=hive
- ranger_app_id=some_app_id
- authorization_factory_class=\
    org.apache.impala.authorization.ranger.RangerAuthorizationFactory

The Ranger plugin for Impala uses Hive service definition to allow
sharing Ranger policies between Hive and Impala. Temporarily the REFRESH
privilege uses "read" access type and it will be updated in the later
patch once Ranger supports "refresh" access type.

There's a change in DESCRIBE <table> privilege requirement to use ANY
privilege instead of VIEW_METADATA privilege as the first-level check
to play nicely with Ranger. This is not a security risk since the
column-level filtering logic after the first-level check will use
VIEW_METADATA privilege to filter out unauthorized column access. In
other words, DESCRIBE <table> may return an empty result instead of
an authorization error as long as there exists any privilege in the
given table.

This patch updates AuthorizationStmtTest with a parameterized test that
runs the tests against Sentry and Ranger.

Testing:
- Updated AuthorizationStmtTest with Ranger
- Ran all FE tests
- Ran all E2E authorization tests

Change-Id: I8cad9e609d20aae1ff645c84fd58a02afee70276
---
M be/src/service/frontend.cc
M be/src/util/backend-gflag-util.cc
M bin/rat_exclude_files.txt
M common/thrift/BackendGflags.thrift
M fe/pom.xml
M fe/src/main/java/org/apache/impala/analysis/DescribeTableStmt.java
M fe/src/main/java/org/apache/impala/authorization/AuthorizationChecker.java
M fe/src/main/java/org/apache/impala/authorization/AuthorizationPolicy.java
M fe/src/main/java/org/apache/impala/authorization/AuthorizationProvider.java
M fe/src/main/java/org/apache/impala/authorization/DefaultAuthorizableFactory.java
M fe/src/main/java/org/apache/impala/authorization/NoneAuthorizationFactory.java
M fe/src/main/java/org/apache/impala/authorization/Privilege.java
A fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationChecker.java
A fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationConfig.java
A fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationFactory.java
A fe/src/main/java/org/apache/impala/authorization/ranger/RangerImpalaPlugin.java
A fe/src/main/java/org/apache/impala/authorization/ranger/RangerImpalaResourceBuilder.java
M fe/src/main/java/org/apache/impala/authorization/sentry/SentryAuthorizationChecker.java
M fe/src/main/java/org/apache/impala/authorization/sentry/SentryAuthorizationFactory.java
M fe/src/main/java/org/apache/impala/service/BackendConfig.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M fe/src/test/java/org/apache/impala/analysis/AnalyzerTest.java
M fe/src/test/java/org/apache/impala/analysis/AuditingTest.java
M fe/src/test/java/org/apache/impala/analysis/AuthorizationStmtTest.java
M fe/src/test/java/org/apache/impala/analysis/AuthorizationTest.java
M fe/src/test/java/org/apache/impala/analysis/ToSqlTest.java
M fe/src/test/java/org/apache/impala/authorization/sentry/SentryProxyTest.java
M fe/src/test/java/org/apache/impala/common/FrontendFixture.java
M fe/src/test/java/org/apache/impala/common/FrontendTestBase.java
M fe/src/test/java/org/apache/impala/testutil/CatalogServiceTestCatalog.java
M fe/src/test/java/org/apache/impala/testutil/ImpaladTestCatalog.java
A fe/src/test/resources/ranger-hive-audit.xml
A fe/src/test/resources/ranger-hive-security.xml
M impala-parent/pom.xml
M testdata/bin/create-load-data.sh
M testdata/cluster/.gitignore
A testdata/cluster/ranger/setup/impala_service.json
A testdata/cluster/ranger/setup/impala_user.json.template
38 files changed, 1,131 insertions(+), 238 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/32/12632/16
-- 
To view, visit http://gerrit.cloudera.org:8080/12632
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I8cad9e609d20aae1ff645c84fd58a02afee70276
Gerrit-Change-Number: 12632
Gerrit-PatchSet: 16
Gerrit-Owner: Fredy Wijaya <fw...@cloudera.com>
Gerrit-Reviewer: Austin Nobis <an...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Fredy Wijaya <fw...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Paul Rogers <pr...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>

[Impala-ASF-CR] IMPALA-8100: Add initial support for Ranger

Posted by "Fredy Wijaya (Code Review)" <ge...@cloudera.org>.
Fredy Wijaya has uploaded a new patch set (#19). ( http://gerrit.cloudera.org:8080/12632 )

Change subject: IMPALA-8100: Add initial support for Ranger
......................................................................

IMPALA-8100: Add initial support for Ranger

This patch adds an initial support for Ranger that can be enabled via
the following flags in both impalad and catalogd to do enforcement.
- ranger_service_type=hive
- ranger_app_id=some_app_id
- authorization_factory_class=\
    org.apache.impala.authorization.ranger.RangerAuthorizationFactory

The Ranger plugin for Impala uses Hive service definition to allow
sharing Ranger policies between Hive and Impala. Temporarily the REFRESH
privilege uses "read" access type and it will be updated in the later
patch once Ranger supports "refresh" access type.

There's a change in DESCRIBE <table> privilege requirement to use ANY
privilege instead of VIEW_METADATA privilege as the first-level check
to play nicely with Ranger. This is not a security risk since the
column-level filtering logic after the first-level check will use
VIEW_METADATA privilege to filter out unauthorized column access. In
other words, DESCRIBE <table> may return an empty result instead of
an authorization error as long as there exists any privilege in the
given table.

This patch updates AuthorizationStmtTest with a parameterized test that
runs the tests against Sentry and Ranger.

Testing:
- Updated AuthorizationStmtTest with Ranger
- Ran all FE tests
- Ran all E2E authorization tests

Change-Id: I8cad9e609d20aae1ff645c84fd58a02afee70276
---
M be/src/service/frontend.cc
M be/src/util/backend-gflag-util.cc
M bin/rat_exclude_files.txt
M common/thrift/BackendGflags.thrift
M fe/pom.xml
M fe/src/main/java/org/apache/impala/analysis/DescribeTableStmt.java
M fe/src/main/java/org/apache/impala/authorization/AuthorizationChecker.java
M fe/src/main/java/org/apache/impala/authorization/AuthorizationPolicy.java
M fe/src/main/java/org/apache/impala/authorization/AuthorizationProvider.java
M fe/src/main/java/org/apache/impala/authorization/DefaultAuthorizableFactory.java
M fe/src/main/java/org/apache/impala/authorization/NoneAuthorizationFactory.java
M fe/src/main/java/org/apache/impala/authorization/Privilege.java
A fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationChecker.java
A fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationConfig.java
A fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationFactory.java
A fe/src/main/java/org/apache/impala/authorization/ranger/RangerImpalaPlugin.java
A fe/src/main/java/org/apache/impala/authorization/ranger/RangerImpalaResourceBuilder.java
M fe/src/main/java/org/apache/impala/authorization/sentry/SentryAuthorizationChecker.java
M fe/src/main/java/org/apache/impala/authorization/sentry/SentryAuthorizationFactory.java
M fe/src/main/java/org/apache/impala/service/BackendConfig.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M fe/src/test/java/org/apache/impala/analysis/AnalyzerTest.java
M fe/src/test/java/org/apache/impala/analysis/AuditingTest.java
M fe/src/test/java/org/apache/impala/analysis/AuthorizationStmtTest.java
M fe/src/test/java/org/apache/impala/analysis/AuthorizationTest.java
M fe/src/test/java/org/apache/impala/analysis/ToSqlTest.java
M fe/src/test/java/org/apache/impala/authorization/sentry/SentryProxyTest.java
M fe/src/test/java/org/apache/impala/common/FrontendFixture.java
M fe/src/test/java/org/apache/impala/common/FrontendTestBase.java
M fe/src/test/java/org/apache/impala/testutil/CatalogServiceTestCatalog.java
M fe/src/test/java/org/apache/impala/testutil/ImpaladTestCatalog.java
A fe/src/test/resources/ranger-hive-audit.xml
A fe/src/test/resources/ranger-hive-security.xml
M impala-parent/pom.xml
M testdata/bin/create-load-data.sh
M testdata/cluster/.gitignore
A testdata/cluster/ranger/setup/impala_service.json
A testdata/cluster/ranger/setup/impala_user.json.template
38 files changed, 1,132 insertions(+), 238 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/32/12632/19
-- 
To view, visit http://gerrit.cloudera.org:8080/12632
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I8cad9e609d20aae1ff645c84fd58a02afee70276
Gerrit-Change-Number: 12632
Gerrit-PatchSet: 19
Gerrit-Owner: Fredy Wijaya <fw...@cloudera.com>
Gerrit-Reviewer: Austin Nobis <an...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Fredy Wijaya <fw...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Paul Rogers <pr...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>

[Impala-ASF-CR] IMPALA-8100: Add initial support for Ranger

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12632 )

Change subject: IMPALA-8100: Add initial support for Ranger
......................................................................


Patch Set 18: Code-Review+2


-- 
To view, visit http://gerrit.cloudera.org:8080/12632
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I8cad9e609d20aae1ff645c84fd58a02afee70276
Gerrit-Change-Number: 12632
Gerrit-PatchSet: 18
Gerrit-Owner: Fredy Wijaya <fw...@cloudera.com>
Gerrit-Reviewer: Austin Nobis <an...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Fredy Wijaya <fw...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Paul Rogers <pr...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Comment-Date: Thu, 21 Mar 2019 05:38:33 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-8100: Add initial support for Ranger

Posted by "Fredy Wijaya (Code Review)" <ge...@cloudera.org>.
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/12632 )

Change subject: IMPALA-8100: Add initial support for Ranger
......................................................................


Patch Set 19: Code-Review+2

Trivial rebase fix due to CR that recently got merged: https://gerrit.cloudera.org/c/12813/. Carrying +2.


-- 
To view, visit http://gerrit.cloudera.org:8080/12632
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I8cad9e609d20aae1ff645c84fd58a02afee70276
Gerrit-Change-Number: 12632
Gerrit-PatchSet: 19
Gerrit-Owner: Fredy Wijaya <fw...@cloudera.com>
Gerrit-Reviewer: Austin Nobis <an...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Fredy Wijaya <fw...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Paul Rogers <pr...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Comment-Date: Thu, 21 Mar 2019 14:26:25 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-8100: Add initial support for Ranger

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12632 )

Change subject: IMPALA-8100: Add initial support for Ranger
......................................................................


Patch Set 17:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/2497/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/12632
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I8cad9e609d20aae1ff645c84fd58a02afee70276
Gerrit-Change-Number: 12632
Gerrit-PatchSet: 17
Gerrit-Owner: Fredy Wijaya <fw...@cloudera.com>
Gerrit-Reviewer: Austin Nobis <an...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Fredy Wijaya <fw...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Paul Rogers <pr...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Comment-Date: Thu, 21 Mar 2019 01:35:00 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-8100: Add initial support for Ranger

Posted by "Fredy Wijaya (Code Review)" <ge...@cloudera.org>.
Fredy Wijaya has uploaded a new patch set (#13). ( http://gerrit.cloudera.org:8080/12632 )

Change subject: IMPALA-8100: Add initial support for Ranger
......................................................................

IMPALA-8100: Add initial support for Ranger

This patch adds an initial support for Ranger that can be enabled via
the following flags in both impalad and catalogd to do enforcement.
- ranger_service_type=impala
- ranger_app_id=some_app_id
- authorization_factory_class=\
    org.apache.impala.authorization.ranger.RangerAuthorizationFactory

There's a change in DESCRIBE <table> privilege requirement to use ANY
privilege instead of VIEW_METADATA privilege as the first-level check
to play nicely with Ranger. This is not a security risk since the
column-level filtering logic after the first-level check will use
VIEW_METADATA privilege to filter out unauthorized column access. In
other words, DESCRIBE <table> may return an empty result instead of
an authorization error as long as there exists any privilege in the
given table.

This patch updates AuthorizationStmtTest with a parameterized test that
runs the tests against Sentry and Ranger.

Testing:
- Updated AuthorizationStmtTest with Ranger
- Ran all FE tests
- Ran all E2E authorization tests

Change-Id: I8cad9e609d20aae1ff645c84fd58a02afee70276
---
M be/src/service/frontend.cc
M be/src/util/backend-gflag-util.cc
M bin/rat_exclude_files.txt
M common/thrift/BackendGflags.thrift
M fe/pom.xml
M fe/src/main/java/org/apache/impala/analysis/DescribeTableStmt.java
M fe/src/main/java/org/apache/impala/authorization/AuthorizationChecker.java
M fe/src/main/java/org/apache/impala/authorization/AuthorizationPolicy.java
M fe/src/main/java/org/apache/impala/authorization/AuthorizationProvider.java
M fe/src/main/java/org/apache/impala/authorization/DefaultAuthorizableFactory.java
M fe/src/main/java/org/apache/impala/authorization/NoneAuthorizationFactory.java
M fe/src/main/java/org/apache/impala/authorization/Privilege.java
A fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationChecker.java
A fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationConfig.java
A fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationFactory.java
A fe/src/main/java/org/apache/impala/authorization/ranger/RangerImpalaPlugin.java
A fe/src/main/java/org/apache/impala/authorization/ranger/RangerImpalaResourceBuilder.java
M fe/src/main/java/org/apache/impala/authorization/sentry/SentryAuthorizationChecker.java
M fe/src/main/java/org/apache/impala/authorization/sentry/SentryAuthorizationFactory.java
M fe/src/main/java/org/apache/impala/service/BackendConfig.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M fe/src/test/java/org/apache/impala/analysis/AnalyzerTest.java
M fe/src/test/java/org/apache/impala/analysis/AuditingTest.java
M fe/src/test/java/org/apache/impala/analysis/AuthorizationStmtTest.java
M fe/src/test/java/org/apache/impala/analysis/AuthorizationTest.java
M fe/src/test/java/org/apache/impala/analysis/ToSqlTest.java
M fe/src/test/java/org/apache/impala/authorization/sentry/SentryProxyTest.java
M fe/src/test/java/org/apache/impala/common/FrontendFixture.java
M fe/src/test/java/org/apache/impala/common/FrontendTestBase.java
M fe/src/test/java/org/apache/impala/testutil/CatalogServiceTestCatalog.java
M fe/src/test/java/org/apache/impala/testutil/ImpaladTestCatalog.java
A fe/src/test/resources/ranger-impala-audit.xml
A fe/src/test/resources/ranger-impala-security.xml
M impala-parent/pom.xml
M testdata/bin/create-load-data.sh
M testdata/cluster/.gitignore
A testdata/cluster/ranger/setup/impala_policy1.json
A testdata/cluster/ranger/setup/impala_policy2.json
A testdata/cluster/ranger/setup/impala_policy3.json
A testdata/cluster/ranger/setup/impala_service.json
A testdata/cluster/ranger/setup/impala_servicedef.json
A testdata/cluster/ranger/setup/impala_user.json.template
42 files changed, 1,462 insertions(+), 237 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/32/12632/13
-- 
To view, visit http://gerrit.cloudera.org:8080/12632
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I8cad9e609d20aae1ff645c84fd58a02afee70276
Gerrit-Change-Number: 12632
Gerrit-PatchSet: 13
Gerrit-Owner: Fredy Wijaya <fw...@cloudera.com>
Gerrit-Reviewer: Austin Nobis <an...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Fredy Wijaya <fw...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Paul Rogers <pr...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>

[Impala-ASF-CR] IMPALA-8100: Add initial support for Ranger

Posted by "Fredy Wijaya (Code Review)" <ge...@cloudera.org>.
Fredy Wijaya has uploaded a new patch set (#8). ( http://gerrit.cloudera.org:8080/12632 )

Change subject: IMPALA-8100: Add initial support for Ranger
......................................................................

IMPALA-8100: Add initial support for Ranger

This patch adds an initial support for Ranger that can be enabled via
the following flags in both impalad and catalogd to do enforcement.
- ranger_service_type=impala
- ranger_app_id=some_app_id
- authorization_factory_class=\
    org.apache.impala.authorization.ranger.RangerAuthorizationFactory

There's a change in DESCRIBE <table> privilege requirement to use ANY
privilege instead of VIEW_METADATA privilege as the first-level check
to play nicely with Ranger. This is not a security risk since the
column-level filtering logic after the first-level check will use
VIEW_METADATA privilege to filter out unauthorized column access. In
other words, DESCRIBE <table> may return an empty result instead of
an authorization error as long as there exists any privilege in the
given table.

This patch updates AuthorizationStmtTest with a parameterized test that
runs the tests against Sentry and Ranger.

Testing:
- Updated AuthorizationStmtTest with Ranger
- Ran all FE tests
- Ran all E2E authorization tests

Change-Id: I8cad9e609d20aae1ff645c84fd58a02afee70276
---
M be/src/service/frontend.cc
M be/src/util/backend-gflag-util.cc
M bin/rat_exclude_files.txt
M common/thrift/BackendGflags.thrift
M fe/pom.xml
M fe/src/main/java/org/apache/impala/analysis/DescribeTableStmt.java
M fe/src/main/java/org/apache/impala/authorization/AuthorizationProvider.java
M fe/src/main/java/org/apache/impala/authorization/DefaultAuthorizableFactory.java
M fe/src/main/java/org/apache/impala/authorization/Privilege.java
A fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationChecker.java
A fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationConfig.java
A fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationFactory.java
A fe/src/main/java/org/apache/impala/authorization/ranger/RangerImpalaPlugin.java
A fe/src/main/java/org/apache/impala/authorization/ranger/RangerImpalaResourceBuilder.java
M fe/src/main/java/org/apache/impala/authorization/sentry/SentryAuthorizationChecker.java
M fe/src/main/java/org/apache/impala/authorization/sentry/SentryAuthorizationFactory.java
M fe/src/main/java/org/apache/impala/service/BackendConfig.java
M fe/src/test/java/org/apache/impala/analysis/AnalyzerTest.java
M fe/src/test/java/org/apache/impala/analysis/AuditingTest.java
M fe/src/test/java/org/apache/impala/analysis/AuthorizationStmtTest.java
M fe/src/test/java/org/apache/impala/analysis/AuthorizationTest.java
M fe/src/test/java/org/apache/impala/analysis/ToSqlTest.java
M fe/src/test/java/org/apache/impala/authorization/sentry/SentryProxyTest.java
M fe/src/test/java/org/apache/impala/common/FrontendFixture.java
M fe/src/test/java/org/apache/impala/common/FrontendTestBase.java
M fe/src/test/java/org/apache/impala/testutil/CatalogServiceTestCatalog.java
M fe/src/test/java/org/apache/impala/testutil/ImpaladTestCatalog.java
A fe/src/test/resources/ranger-impala-audit.xml
A fe/src/test/resources/ranger-impala-security.xml
M impala-parent/pom.xml
M testdata/bin/create-load-data.sh
M testdata/cluster/.gitignore
A testdata/cluster/ranger/setup/impala_policy1.json
A testdata/cluster/ranger/setup/impala_policy2.json
A testdata/cluster/ranger/setup/impala_policy3.json
A testdata/cluster/ranger/setup/impala_service.json
A testdata/cluster/ranger/setup/impala_servicedef.json
A testdata/cluster/ranger/setup/impala_user.json.template
38 files changed, 1,378 insertions(+), 223 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/32/12632/8
-- 
To view, visit http://gerrit.cloudera.org:8080/12632
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I8cad9e609d20aae1ff645c84fd58a02afee70276
Gerrit-Change-Number: 12632
Gerrit-PatchSet: 8
Gerrit-Owner: Fredy Wijaya <fw...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Fredy Wijaya <fw...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Paul Rogers <pr...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>

[Impala-ASF-CR] IMPALA-8100: Add initial support for Ranger

Posted by "Austin Nobis (Code Review)" <ge...@cloudera.org>.
Austin Nobis has posted comments on this change. ( http://gerrit.cloudera.org:8080/12632 )

Change subject: IMPALA-8100: Add initial support for Ranger
......................................................................


Patch Set 13: Code-Review+1


-- 
To view, visit http://gerrit.cloudera.org:8080/12632
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I8cad9e609d20aae1ff645c84fd58a02afee70276
Gerrit-Change-Number: 12632
Gerrit-PatchSet: 13
Gerrit-Owner: Fredy Wijaya <fw...@cloudera.com>
Gerrit-Reviewer: Austin Nobis <an...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Fredy Wijaya <fw...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Paul Rogers <pr...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Comment-Date: Fri, 15 Mar 2019 21:01:14 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-8100: Add initial support for Ranger

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12632 )

Change subject: IMPALA-8100: Add initial support for Ranger
......................................................................


Patch Set 20: Verified+1


-- 
To view, visit http://gerrit.cloudera.org:8080/12632
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I8cad9e609d20aae1ff645c84fd58a02afee70276
Gerrit-Change-Number: 12632
Gerrit-PatchSet: 20
Gerrit-Owner: Fredy Wijaya <fw...@cloudera.com>
Gerrit-Reviewer: Austin Nobis <an...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Fredy Wijaya <fw...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Paul Rogers <pr...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Comment-Date: Thu, 21 Mar 2019 19:08:10 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-8100: Add initial support for Ranger

Posted by "Fredy Wijaya (Code Review)" <ge...@cloudera.org>.
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/12632 )

Change subject: IMPALA-8100: Add initial support for Ranger
......................................................................


Patch Set 11:

(25 comments)

http://gerrit.cloudera.org:8080/#/c/12632/9//COMMIT_MSG
Commit Message:

http://gerrit.cloudera.org:8080/#/c/12632/9//COMMIT_MSG@30
PS9, Line 30: - Ran all FE tests
            : - Ran all E2E authorization tests
> did you run these with ranger configured? I'm surprised they would pass sin
For FE, it runs with both Ranger and Sentry via JUnit parameterized test. I ran E2E authorization tests to make sure no regression in the Sentry implementation.


http://gerrit.cloudera.org:8080/#/c/12632/9/be/src/service/frontend.cc
File be/src/service/frontend.cc:

http://gerrit.cloudera.org:8080/#/c/12632/9/be/src/service/frontend.cc@41
PS9, Line 41:     "org.apache.impala.authorization.sentry.SentryAuthorizationFactory",
> I wonder if we should switch this to be a more human-friendly flag instead 
Phil brought up the same thing. I have a plan to make it more human-friendly in a different JIRA: https://issues.apache.org/jira/browse/IMPALA-8309


http://gerrit.cloudera.org:8080/#/c/12632/9/be/src/service/frontend.cc@46
PS9, Line 46:     " uniquely identify the application that communicates with Ranger.");
> Worth noting whether this is required, considering it has an empty default
Done


http://gerrit.cloudera.org:8080/#/c/12632/9/be/src/service/frontend.cc@47
PS9, Line 47: DEFINE_string(server_name, "", "(Required) The name to use for securing this impalad "
> we probably can't rename the existing sentry-specific flags (for compat rea
Done. server_name is also required for Ranger.


http://gerrit.cloudera.org:8080/#/c/12632/9/fe/pom.xml
File fe/pom.xml:

http://gerrit.cloudera.org:8080/#/c/12632/9/fe/pom.xml@112
PS9, Line 112:       <exclusions>
> it seems that ranger-plugins-common has a pom dependency on mysql-connector
Yes it's a service model. Running mvn dependency:tree on Ranger gave me this output. I think it's pretty light after excluding some of its transitive dependencies.

[INFO] +- org.apache.ranger:ranger-plugins-common:jar:1.2.0:compile                                                                                                                                                                                                                                                          
[INFO] |  \- org.codehaus.jackson:jackson-jaxrs:jar:1.9.13:compile                        
[INFO] +- org.apache.ranger:ranger-plugins-audit:jar:1.2.0:compile


http://gerrit.cloudera.org:8080/#/c/12632/9/fe/src/main/java/org/apache/impala/analysis/DescribeTableStmt.java
File fe/src/main/java/org/apache/impala/analysis/DescribeTableStmt.java:

http://gerrit.cloudera.org:8080/#/c/12632/9/fe/src/main/java/org/apache/impala/analysis/DescribeTableStmt.java@126
PS9, Line 126:         Privilege.ANY);
> I'm not 100% sure about this part of the change. If I don't have the 'VIEW_
If I don't have the 'VIEW_METADATA' privilege but I do have permissions on one or more columns, should I still be able to use 'DESCRIBE' in all cases?

Yes, however the output of DESCRIBE will vary depending on whether or not the user has VIEW_METADATA privilege (the second level check). Filtering out the DESCRIBE output is part of this JIRA: https://issues.apache.org/jira/browse/IMPALA-6479.

This behavior is pretty similar to "show databases/tables". A user can always execute "show databases/tables", however only databases/tables that the user has access to (with VIEW_METADATA privilege) will be displayed.


http://gerrit.cloudera.org:8080/#/c/12632/9/fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationChecker.java
File fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationChecker.java:

http://gerrit.cloudera.org:8080/#/c/12632/9/fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationChecker.java@77
PS9, Line 77:             .table(authorizable.getTableName());
            :         break;
            :       case COLUMN:
            :         builder.server(config_.getServerName())
            :             .database(authorizable.getDbName());
            :         // * in Ranger means "all". For example to check access
> I'm not understanding this. When you are saying "Any column access" do you 
Done. I agree it was confusing.


http://gerrit.cloudera.org:8080/#/c/12632/9/fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationChecker.java@106
PS9, Line 106:             .function(authorizable.getFnName());
> does this need to be special cased? It seems all the other privileges alrea
Ah yeah you're right.


http://gerrit.cloudera.org:8080/#/c/12632/9/fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationChecker.java@117
PS9, Line 117:     if (request.getPrivilege() == Privilege.ANY) {
> does this result in multiple audit events, one per implied privilege?
Yes.


http://gerrit.cloudera.org:8080/#/c/12632/9/fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationChecker.java@131
PS9, Line 131: 
> can you use Collections.emptySet?
Good catch, meant to leave it as a todo, but it's easy enough that i'll just do it i this patch.


http://gerrit.cloudera.org:8080/#/c/12632/9/fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationConfig.java
File fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationConfig.java:

http://gerrit.cloudera.org:8080/#/c/12632/9/fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationConfig.java@38
PS9, Line 38:  RangerConf
> nit: Preconditions.checkNotNull(foo) returns 'foo', so you can combine thes
Done


http://gerrit.cloudera.org:8080/#/c/12632/9/fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationFactory.java
File fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationFactory.java:

http://gerrit.cloudera.org:8080/#/c/12632/9/fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationFactory.java@55
PS9, Line 55:     Preconditions.checkArgument(!Strings.isNullOrEmpty(
> is this the only validation that happens on this config? If so, we should m
Ah yeah, good point. Done.


http://gerrit.cloudera.org:8080/#/c/12632/9/fe/src/main/java/org/apache/impala/authorization/ranger/RangerImpalaResourceBuilder.java
File fe/src/main/java/org/apache/impala/authorization/ranger/RangerImpalaResourceBuilder.java:

http://gerrit.cloudera.org:8080/#/c/12632/9/fe/src/main/java/org/apache/impala/authorization/ranger/RangerImpalaResourceBuilder.java@37
PS9, Line 37: r(String s
> can we add Preconditions.checkNotNull() for all parameters in this file? Is
Done


http://gerrit.cloudera.org:8080/#/c/12632/9/fe/src/main/java/org/apache/impala/authorization/ranger/RangerImpalaResourceBuilder.java@52
PS9, Line 52:   public RangerImpalaResourceBuilder column(String columnName) {
> general question about ranger semantics -- are resources case sensitive? sh
That'll be handled in Ranger side, more specifically the service definition: https://gerrit.cloudera.org/c/12632/9/testdata/cluster/ranger/setup/impala_servicedef.json#142


http://gerrit.cloudera.org:8080/#/c/12632/9/fe/src/test/java/org/apache/impala/analysis/AuthorizationStmtTest.java
File fe/src/test/java/org/apache/impala/analysis/AuthorizationStmtTest.java:

http://gerrit.cloudera.org:8080/#/c/12632/9/fe/src/test/java/org/apache/impala/analysis/AuthorizationStmtTest.java@90
PS9, Line 90:   private static final String RANGER_SERVICE_TYPE = "impala";
            :   private static final String RANGER_APP_ID = "impala";
> can these be pulled from the config instead to avoid duplication?
Ranger needs service type and app ID to create a Ranger config.


http://gerrit.cloudera.org:8080/#/c/12632/9/fe/src/test/java/org/apache/impala/analysis/AuthorizationStmtTest.java@110
PS9, Line 110: System.getenv("IMPALA_HOME") + "/fe/src/test/resources/sentry-site.
> this is a little sketchy. Shouldn't this resource be on the classpath and t
Yes, we could. This is some remnant of the old stuff. The createHadoopGroupAuthConfig doesn't understand reading a resource from a classpath. I plan to clean up a lot of authorization tests in this JIRA: https://issues.apache.org/jira/browse/IMPALA-8248


http://gerrit.cloudera.org:8080/#/c/12632/9/fe/src/test/java/org/apache/impala/analysis/AuthorizationTest.java
File fe/src/test/java/org/apache/impala/analysis/AuthorizationTest.java:

http://gerrit.cloudera.org:8080/#/c/12632/9/fe/src/test/java/org/apache/impala/analysis/AuthorizationTest.java@178
PS9, Line 178:     SentryAuthorizationFactory authzFactory = new SentryAuthorizationFactory(
> should we be parametrizing this test to be able to run against ranger as we
There's a separate JIRA to refactor all the authorization tests to basically separate Sentry and Ranger specific tests and put the tests that are common to both: https://issues.apache.org/jira/browse/IMPALA-8248. I don't want this patch to get too big.


http://gerrit.cloudera.org:8080/#/c/12632/9/fe/src/test/java/org/apache/impala/common/FrontendTestBase.java
File fe/src/test/java/org/apache/impala/common/FrontendTestBase.java:

http://gerrit.cloudera.org:8080/#/c/12632/9/fe/src/test/java/org/apache/impala/common/FrontendTestBase.java@308
PS9, Line 308:           @Override
             :           public boolean isEnabled() { return true; }
             :           @Override
             :           public AuthorizationProvider getProvider() {
             :             return AuthorizationProvider.NONE;
             :           }
> I'm not sure what this means, semantically -- authorization enabled but NON
Yes, it should be DUMMY or ALWAYS_TRUE provider, but I don't think this warrants for an additional enum value :) This is pretty much a hack to make non-authorization unit tests (see AnalyzerTest and ToSqlTest) easier because some statements, such as REFRESH AUTHORIZATION can only be executed when authorization is enabled, but we also don't want to run it with Sentry or Ranger.


http://gerrit.cloudera.org:8080/#/c/12632/9/fe/src/test/resources/ranger-impala-security.xml
File fe/src/test/resources/ranger-impala-security.xml:

http://gerrit.cloudera.org:8080/#/c/12632/9/fe/src/test/resources/ranger-impala-security.xml@21
PS9, Line 21:     <value>test_impala</value>
> how does this interact with the gflag for "service id"?
service ID is defined in service definition: https://gerrit.cloudera.org/c/12632/9/testdata/cluster/ranger/setup/impala_servicedef.json#50


http://gerrit.cloudera.org:8080/#/c/12632/9/fe/src/test/resources/ranger-impala-security.xml@37
PS9, Line 37:     <value>/tmp</value>
> can we auto-configure this based on other existing scratch dir configs, at 
The scratch_dirs flag takes multiple directories, do we just pick any scratch directory randomly?


http://gerrit.cloudera.org:8080/#/c/12632/9/fe/src/test/resources/ranger-impala-security.xml@48
PS9, Line 48:       Polling interval in milliseconds to poll for changes in policies.
> are these strings copied from somewhere? It's weird to see a question mark 
Yeah I copied this config file from Ranger. I will update all the descriptions.


http://gerrit.cloudera.org:8080/#/c/12632/9/fe/src/test/resources/ranger-impala-security.xml@56
PS9, Line 56:       RangerRestClient connection timeout in milliseconds.
> milliseconds
Done


http://gerrit.cloudera.org:8080/#/c/12632/9/fe/src/test/resources/ranger-impala-security.xml@64
PS9, Line 64:       RangerRestClient read timeout in milliseconds.
> milliseconds
Done


http://gerrit.cloudera.org:8080/#/c/12632/9/testdata/bin/create-load-data.sh
File testdata/bin/create-load-data.sh:

http://gerrit.cloudera.org:8080/#/c/12632/9/testdata/bin/create-load-data.sh@308
PS9, Line 308:   perl -wpl -e 's/\$\{([^}]+)\}/defined $ENV{$1} ? $ENV{$1} : $&/eg' \
> you don't want to die() if $ENV{$1} is not defined here? leaving it unsubst
Done


http://gerrit.cloudera.org:8080/#/c/12632/9/testdata/cluster/ranger/setup/impala_servicedef.json
File testdata/cluster/ranger/setup/impala_servicedef.json:

http://gerrit.cloudera.org:8080/#/c/12632/9/testdata/cluster/ranger/setup/impala_servicedef.json@1
PS9, Line 1: {
> I'm a little curious, at a high level, how this interacts with any ranger p
This is something that I don't quite know the answer whether Impala should have the same service definition as Hive. We can't reuse the existing Hive service definition because it contains specific things to Hive, such as Hive JDBC driver, different access types/privileges, etc. Unlike with Sentry, the Ranger model prefers for each service to have their own service definition. IOW, there will not be shared policies between Impala and Hive.

If we need to have shared policies between Impala and Hive, we need to come up with a new service definition, but that also means the existing Hive Ranger plugin may need to be updated. Let me know what you think?

I also haven't spent much time on the deployment. I believe only the service definition will need to be deployed. All other files here are purely for testing. I'll look into this more once everything is in a better shape. For now, we can consider all these files here for testing.



-- 
To view, visit http://gerrit.cloudera.org:8080/12632
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I8cad9e609d20aae1ff645c84fd58a02afee70276
Gerrit-Change-Number: 12632
Gerrit-PatchSet: 11
Gerrit-Owner: Fredy Wijaya <fw...@cloudera.com>
Gerrit-Reviewer: Austin Nobis <an...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Fredy Wijaya <fw...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Paul Rogers <pr...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Comment-Date: Thu, 14 Mar 2019 23:54:28 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-8100: Add initial support for Ranger

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12632 )

Change subject: IMPALA-8100: Add initial support for Ranger
......................................................................


Patch Set 7:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/12632/7/fe/src/test/java/org/apache/impala/common/FrontendTestBase.java
File fe/src/test/java/org/apache/impala/common/FrontendTestBase.java:

http://gerrit.cloudera.org:8080/#/c/12632/7/fe/src/test/java/org/apache/impala/common/FrontendTestBase.java@325
PS7, Line 325:       public AuthorizationChecker newAuthorizationChecker(AuthorizationPolicy authzPolicy) {
line too long (92 > 90)



-- 
To view, visit http://gerrit.cloudera.org:8080/12632
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I8cad9e609d20aae1ff645c84fd58a02afee70276
Gerrit-Change-Number: 12632
Gerrit-PatchSet: 7
Gerrit-Owner: Fredy Wijaya <fw...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Fredy Wijaya <fw...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Paul Rogers <pr...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Comment-Date: Wed, 06 Mar 2019 02:22:39 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-8100: Add initial support for Ranger

Posted by "Fredy Wijaya (Code Review)" <ge...@cloudera.org>.
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/12632 )

Change subject: IMPALA-8100: Add initial support for Ranger
......................................................................


Patch Set 17:

(2 comments)

http://gerrit.cloudera.org:8080/#/c/12632/16/be/src/service/frontend.cc
File be/src/service/frontend.cc:

http://gerrit.cloudera.org:8080/#/c/12632/16/be/src/service/frontend.cc@47
PS16, Line 47:  with Range
> this is only required if authorization is enabled, right?
Yes. I guess instead of putting (Required) here can be confusing. Done.


http://gerrit.cloudera.org:8080/#/c/12632/16/fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationChecker.java
File fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationChecker.java:

http://gerrit.cloudera.org:8080/#/c/12632/16/fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationChecker.java@72
PS16, Line 72:        // Hive service definition does not have a concept of server. So we define
             :         // server to mean all access to all resource sets.
> Is this something we need to address? I can't recall all the cases in which
It would be good if Hive could support it, but that might actually be a big change from the existing Ranger plugin for Hive.

Technically having server as a root is better since we only need to make a single call instead of having 3 calls. We have certain statements that require server-level authorization, such as shutdown command, invalidate metadata (the global), and there are more I think.



-- 
To view, visit http://gerrit.cloudera.org:8080/12632
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I8cad9e609d20aae1ff645c84fd58a02afee70276
Gerrit-Change-Number: 12632
Gerrit-PatchSet: 17
Gerrit-Owner: Fredy Wijaya <fw...@cloudera.com>
Gerrit-Reviewer: Austin Nobis <an...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Fredy Wijaya <fw...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Paul Rogers <pr...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Comment-Date: Thu, 21 Mar 2019 01:05:33 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-8100 (WIP): Add initial support for Ranger

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12632 )

Change subject: IMPALA-8100 (WIP): Add initial support for Ranger
......................................................................


Patch Set 2:

Build Failed 

https://jenkins.impala.io/job/gerrit-code-review-checks/2294/ : Initial code review checks failed. See linked job for details on the failure.


-- 
To view, visit http://gerrit.cloudera.org:8080/12632
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I8cad9e609d20aae1ff645c84fd58a02afee70276
Gerrit-Change-Number: 12632
Gerrit-PatchSet: 2
Gerrit-Owner: Fredy Wijaya <fw...@cloudera.com>
Gerrit-Reviewer: Fredy Wijaya <fw...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Thu, 28 Feb 2019 04:27:05 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-8100 (WIP): Add initial support for Ranger

Posted by "Fredy Wijaya (Code Review)" <ge...@cloudera.org>.
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/12632 )

Change subject: IMPALA-8100 (WIP): Add initial support for Ranger
......................................................................


Patch Set 2:

This is a sneak-peak of initial support for Ranger that depends on this CR: https://gerrit.cloudera.org/c/12542 to illustrate the use of new authorization interfaces to implement Ranger.


-- 
To view, visit http://gerrit.cloudera.org:8080/12632
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I8cad9e609d20aae1ff645c84fd58a02afee70276
Gerrit-Change-Number: 12632
Gerrit-PatchSet: 2
Gerrit-Owner: Fredy Wijaya <fw...@cloudera.com>
Gerrit-Reviewer: Fredy Wijaya <fw...@cloudera.com>
Gerrit-Comment-Date: Thu, 28 Feb 2019 03:46:18 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-8100: Add initial support for Ranger

Posted by "Philip Zeyliger (Code Review)" <ge...@cloudera.org>.
Philip Zeyliger has posted comments on this change. ( http://gerrit.cloudera.org:8080/12632 )

Change subject: IMPALA-8100: Add initial support for Ranger
......................................................................


Patch Set 6:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/12632/6/be/src/service/frontend.cc
File be/src/service/frontend.cc:

http://gerrit.cloudera.org:8080/#/c/12632/6/be/src/service/frontend.cc@41
PS6, Line 41:     "org.apache.impala.authorization.sentry.SentryAuthorizationFactory",
> I'm actually having a second thought whether it's a good idea to make Autho
If we expose it as a pluggable thing, we have to maintain the pluggable API forever. Unless someone is clamoring for it, I'd rather we lock it down by default. Not a blocking concern.



-- 
To view, visit http://gerrit.cloudera.org:8080/12632
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I8cad9e609d20aae1ff645c84fd58a02afee70276
Gerrit-Change-Number: 12632
Gerrit-PatchSet: 6
Gerrit-Owner: Fredy Wijaya <fw...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Fredy Wijaya <fw...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Paul Rogers <pr...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Comment-Date: Tue, 05 Mar 2019 05:07:08 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-8100: Add initial support for Ranger

Posted by "Todd Lipcon (Code Review)" <ge...@cloudera.org>.
Todd Lipcon has posted comments on this change. ( http://gerrit.cloudera.org:8080/12632 )

Change subject: IMPALA-8100: Add initial support for Ranger
......................................................................


Patch Set 17: Code-Review+2

(1 comment)

http://gerrit.cloudera.org:8080/#/c/12632/16/fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationChecker.java
File fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationChecker.java:

http://gerrit.cloudera.org:8080/#/c/12632/16/fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationChecker.java@72
PS16, Line 72:        // Hive service definition does not have a concept of server. So we define
             :         // server to mean all access to all resource sets.
> It would be good if Hive could support it, but that might actually be a big
k. Let's come back to this at some point... I can also see the argument that those things may be better off living in a separate Impala-specific service, if it's possible to have a single plugin authorize against two service models. Or, maybe those things would be better off actually passed as a config flag to the impala cluster itself at startup time rather than being centrally administrated, because they are more about service administration than about the underlying data. (eg in a multi-cluster environment, if I "own" an impalad cluster, I should be able to shut it down, but that doesn't imply any permission for me to access any data)



-- 
To view, visit http://gerrit.cloudera.org:8080/12632
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I8cad9e609d20aae1ff645c84fd58a02afee70276
Gerrit-Change-Number: 12632
Gerrit-PatchSet: 17
Gerrit-Owner: Fredy Wijaya <fw...@cloudera.com>
Gerrit-Reviewer: Austin Nobis <an...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Fredy Wijaya <fw...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Paul Rogers <pr...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Comment-Date: Thu, 21 Mar 2019 04:18:23 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-8100: Add initial support for Ranger

Posted by "Fredy Wijaya (Code Review)" <ge...@cloudera.org>.
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/12632 )

Change subject: IMPALA-8100: Add initial support for Ranger
......................................................................


Patch Set 16:

Rebased to fix merge conflicts.


-- 
To view, visit http://gerrit.cloudera.org:8080/12632
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I8cad9e609d20aae1ff645c84fd58a02afee70276
Gerrit-Change-Number: 12632
Gerrit-PatchSet: 16
Gerrit-Owner: Fredy Wijaya <fw...@cloudera.com>
Gerrit-Reviewer: Austin Nobis <an...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Fredy Wijaya <fw...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Paul Rogers <pr...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Comment-Date: Wed, 20 Mar 2019 06:44:52 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-8100: Add initial support for Ranger

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12632 )

Change subject: IMPALA-8100: Add initial support for Ranger
......................................................................


Patch Set 11:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/2438/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/12632
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I8cad9e609d20aae1ff645c84fd58a02afee70276
Gerrit-Change-Number: 12632
Gerrit-PatchSet: 11
Gerrit-Owner: Fredy Wijaya <fw...@cloudera.com>
Gerrit-Reviewer: Austin Nobis <an...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Fredy Wijaya <fw...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Paul Rogers <pr...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Comment-Date: Fri, 15 Mar 2019 00:14:41 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-8100: Add initial support for Ranger

Posted by "Fredy Wijaya (Code Review)" <ge...@cloudera.org>.
Fredy Wijaya has uploaded a new patch set (#7). ( http://gerrit.cloudera.org:8080/12632 )

Change subject: IMPALA-8100: Add initial support for Ranger
......................................................................

IMPALA-8100: Add initial support for Ranger

This patch adds an initial support for Ranger that can be enabled via
the following flags in both impalad and catalogd to do enforcement.
- ranger_service_type=impala
- ranger_app_id=some_app_id
- authorization_factory_class=\
    org.apache.impala.authorization.ranger.RangerAuthorizationFactory

There's a change in DESCRIBE <table> privilege requirement to use ANY
privilege instead of VIEW_METADATA privilege as the first-level check
to play nicely with Ranger. This is not a security risk since the
column-level filtering logic after the first-level check will use
VIEW_METADATA privilege to filter out unauthorized column access. In
other words, DESCRIBE <table> may return an empty result instead of
an authorization error as long as there exists any privilege in the
given table.

This patch updates AuthorizationStmtTest with a parameterized test that
runs the tests against Sentry and Ranger.

Testing:
- Updated AuthorizationStmtTest with Ranger
- Ran all FE tests
- Ran all E2E authorization tests

Change-Id: I8cad9e609d20aae1ff645c84fd58a02afee70276
---
M be/src/service/frontend.cc
M be/src/util/backend-gflag-util.cc
M bin/rat_exclude_files.txt
M common/thrift/BackendGflags.thrift
M fe/pom.xml
M fe/src/main/java/org/apache/impala/analysis/DescribeTableStmt.java
M fe/src/main/java/org/apache/impala/authorization/AuthorizationProvider.java
M fe/src/main/java/org/apache/impala/authorization/DefaultAuthorizableFactory.java
M fe/src/main/java/org/apache/impala/authorization/Privilege.java
A fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationChecker.java
A fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationConfig.java
A fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationFactory.java
A fe/src/main/java/org/apache/impala/authorization/ranger/RangerImpalaPlugin.java
A fe/src/main/java/org/apache/impala/authorization/ranger/RangerImpalaResourceBuilder.java
M fe/src/main/java/org/apache/impala/authorization/sentry/SentryAuthorizationChecker.java
M fe/src/main/java/org/apache/impala/authorization/sentry/SentryAuthorizationFactory.java
M fe/src/main/java/org/apache/impala/service/BackendConfig.java
M fe/src/test/java/org/apache/impala/analysis/AnalyzerTest.java
M fe/src/test/java/org/apache/impala/analysis/AuditingTest.java
M fe/src/test/java/org/apache/impala/analysis/AuthorizationStmtTest.java
M fe/src/test/java/org/apache/impala/analysis/AuthorizationTest.java
M fe/src/test/java/org/apache/impala/analysis/ToSqlTest.java
M fe/src/test/java/org/apache/impala/authorization/sentry/SentryProxyTest.java
M fe/src/test/java/org/apache/impala/common/FrontendFixture.java
M fe/src/test/java/org/apache/impala/common/FrontendTestBase.java
M fe/src/test/java/org/apache/impala/testutil/CatalogServiceTestCatalog.java
M fe/src/test/java/org/apache/impala/testutil/ImpaladTestCatalog.java
A fe/src/test/resources/ranger-impala-audit.xml
A fe/src/test/resources/ranger-impala-security.xml
M impala-parent/pom.xml
M testdata/bin/create-load-data.sh
M testdata/cluster/.gitignore
A testdata/cluster/ranger/setup/impala_policy1.json
A testdata/cluster/ranger/setup/impala_policy2.json
A testdata/cluster/ranger/setup/impala_policy3.json
A testdata/cluster/ranger/setup/impala_service.json
A testdata/cluster/ranger/setup/impala_servicedef.json
A testdata/cluster/ranger/setup/impala_user.json.template
38 files changed, 1,377 insertions(+), 223 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/32/12632/7
-- 
To view, visit http://gerrit.cloudera.org:8080/12632
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I8cad9e609d20aae1ff645c84fd58a02afee70276
Gerrit-Change-Number: 12632
Gerrit-PatchSet: 7
Gerrit-Owner: Fredy Wijaya <fw...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Fredy Wijaya <fw...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Paul Rogers <pr...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>

[Impala-ASF-CR] IMPALA-8100: Add initial support for Ranger

Posted by "Todd Lipcon (Code Review)" <ge...@cloudera.org>.
Todd Lipcon has posted comments on this change. ( http://gerrit.cloudera.org:8080/12632 )

Change subject: IMPALA-8100: Add initial support for Ranger
......................................................................


Patch Set 16:

(2 comments)

http://gerrit.cloudera.org:8080/#/c/12632/16/be/src/service/frontend.cc
File be/src/service/frontend.cc:

http://gerrit.cloudera.org:8080/#/c/12632/16/be/src/service/frontend.cc@47
PS16, Line 47: (Required) 
this is only required if authorization is enabled, right?


http://gerrit.cloudera.org:8080/#/c/12632/16/fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationChecker.java
File fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationChecker.java:

http://gerrit.cloudera.org:8080/#/c/12632/16/fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationChecker.java@72
PS16, Line 72:        // Hive service definition does not have a concept of server. So we define
             :         // server to mean all access to all resource sets.
Is this something we need to address? I can't recall all the cases in which we need to do server-level authorization



-- 
To view, visit http://gerrit.cloudera.org:8080/12632
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I8cad9e609d20aae1ff645c84fd58a02afee70276
Gerrit-Change-Number: 12632
Gerrit-PatchSet: 16
Gerrit-Owner: Fredy Wijaya <fw...@cloudera.com>
Gerrit-Reviewer: Austin Nobis <an...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Fredy Wijaya <fw...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Paul Rogers <pr...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Comment-Date: Wed, 20 Mar 2019 23:52:38 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-8100: Add initial support for Ranger

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12632 )

Change subject: IMPALA-8100: Add initial support for Ranger
......................................................................


Patch Set 13:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/2439/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/12632
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I8cad9e609d20aae1ff645c84fd58a02afee70276
Gerrit-Change-Number: 12632
Gerrit-PatchSet: 13
Gerrit-Owner: Fredy Wijaya <fw...@cloudera.com>
Gerrit-Reviewer: Austin Nobis <an...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Fredy Wijaya <fw...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Paul Rogers <pr...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Comment-Date: Fri, 15 Mar 2019 16:16:52 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-8100 (WIP): Add initial support for Ranger

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12632 )

Change subject: IMPALA-8100 (WIP): Add initial support for Ranger
......................................................................


Patch Set 3:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/2295/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/12632
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I8cad9e609d20aae1ff645c84fd58a02afee70276
Gerrit-Change-Number: 12632
Gerrit-PatchSet: 3
Gerrit-Owner: Fredy Wijaya <fw...@cloudera.com>
Gerrit-Reviewer: Fredy Wijaya <fw...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Thu, 28 Feb 2019 05:06:14 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-8100: Add initial support for Ranger

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/12632 )

Change subject: IMPALA-8100: Add initial support for Ranger
......................................................................

IMPALA-8100: Add initial support for Ranger

This patch adds an initial support for Ranger that can be enabled via
the following flags in both impalad and catalogd to do enforcement.
- ranger_service_type=hive
- ranger_app_id=some_app_id
- authorization_factory_class=\
    org.apache.impala.authorization.ranger.RangerAuthorizationFactory

The Ranger plugin for Impala uses Hive service definition to allow
sharing Ranger policies between Hive and Impala. Temporarily the REFRESH
privilege uses "read" access type and it will be updated in the later
patch once Ranger supports "refresh" access type.

There's a change in DESCRIBE <table> privilege requirement to use ANY
privilege instead of VIEW_METADATA privilege as the first-level check
to play nicely with Ranger. This is not a security risk since the
column-level filtering logic after the first-level check will use
VIEW_METADATA privilege to filter out unauthorized column access. In
other words, DESCRIBE <table> may return an empty result instead of
an authorization error as long as there exists any privilege in the
given table.

This patch updates AuthorizationStmtTest with a parameterized test that
runs the tests against Sentry and Ranger.

Testing:
- Updated AuthorizationStmtTest with Ranger
- Ran all FE tests
- Ran all E2E authorization tests

Change-Id: I8cad9e609d20aae1ff645c84fd58a02afee70276
Reviewed-on: http://gerrit.cloudera.org:8080/12632
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
M be/src/service/frontend.cc
M be/src/util/backend-gflag-util.cc
M bin/rat_exclude_files.txt
M common/thrift/BackendGflags.thrift
M fe/pom.xml
M fe/src/main/java/org/apache/impala/analysis/DescribeTableStmt.java
M fe/src/main/java/org/apache/impala/authorization/AuthorizationChecker.java
M fe/src/main/java/org/apache/impala/authorization/AuthorizationPolicy.java
M fe/src/main/java/org/apache/impala/authorization/AuthorizationProvider.java
M fe/src/main/java/org/apache/impala/authorization/DefaultAuthorizableFactory.java
M fe/src/main/java/org/apache/impala/authorization/NoneAuthorizationFactory.java
M fe/src/main/java/org/apache/impala/authorization/Privilege.java
A fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationChecker.java
A fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationConfig.java
A fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationFactory.java
A fe/src/main/java/org/apache/impala/authorization/ranger/RangerImpalaPlugin.java
A fe/src/main/java/org/apache/impala/authorization/ranger/RangerImpalaResourceBuilder.java
M fe/src/main/java/org/apache/impala/authorization/sentry/SentryAuthorizationChecker.java
M fe/src/main/java/org/apache/impala/authorization/sentry/SentryAuthorizationFactory.java
M fe/src/main/java/org/apache/impala/service/BackendConfig.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M fe/src/test/java/org/apache/impala/analysis/AnalyzerTest.java
M fe/src/test/java/org/apache/impala/analysis/AuditingTest.java
M fe/src/test/java/org/apache/impala/analysis/AuthorizationStmtTest.java
M fe/src/test/java/org/apache/impala/analysis/AuthorizationTest.java
M fe/src/test/java/org/apache/impala/analysis/ToSqlTest.java
M fe/src/test/java/org/apache/impala/authorization/sentry/SentryProxyTest.java
M fe/src/test/java/org/apache/impala/common/FrontendFixture.java
M fe/src/test/java/org/apache/impala/common/FrontendTestBase.java
M fe/src/test/java/org/apache/impala/testutil/CatalogServiceTestCatalog.java
M fe/src/test/java/org/apache/impala/testutil/ImpaladTestCatalog.java
A fe/src/test/resources/ranger-hive-audit.xml
A fe/src/test/resources/ranger-hive-security.xml
M impala-parent/pom.xml
M testdata/bin/create-load-data.sh
M testdata/cluster/.gitignore
A testdata/cluster/ranger/setup/impala_service.json
A testdata/cluster/ranger/setup/impala_user.json.template
38 files changed, 1,132 insertions(+), 238 deletions(-)

Approvals:
  Impala Public Jenkins: Looks good to me, approved; Verified

-- 
To view, visit http://gerrit.cloudera.org:8080/12632
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I8cad9e609d20aae1ff645c84fd58a02afee70276
Gerrit-Change-Number: 12632
Gerrit-PatchSet: 21
Gerrit-Owner: Fredy Wijaya <fw...@cloudera.com>
Gerrit-Reviewer: Austin Nobis <an...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Fredy Wijaya <fw...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Paul Rogers <pr...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>

[Impala-ASF-CR] IMPALA-8100 (WIP): Add initial support for Ranger

Posted by "Fredy Wijaya (Code Review)" <ge...@cloudera.org>.
Fredy Wijaya has uploaded a new patch set (#3). ( http://gerrit.cloudera.org:8080/12632 )

Change subject: IMPALA-8100 (WIP): Add initial support for Ranger
......................................................................

IMPALA-8100 (WIP): Add initial support for Ranger

This patch adds an initial support for Ranger that can be enabled via
the following flags in both impalad and catalogd to do enforcement.
- ranger_service_type=impala
- ranger_app_id=some_app_id
- authorization_factory_class=\
    org.apache.impala.authorization.ranger.RangerAuthorizationFactory

Change-Id: I8cad9e609d20aae1ff645c84fd58a02afee70276
---
M be/src/service/frontend.cc
M be/src/util/backend-gflag-util.cc
M bin/rat_exclude_files.txt
M common/thrift/BackendGflags.thrift
M fe/pom.xml
M fe/src/main/java/org/apache/impala/authorization/AuthorizationProvider.java
A fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationChecker.java
A fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationConfig.java
A fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationFactory.java
A fe/src/main/java/org/apache/impala/authorization/ranger/RangerImpalaPlugin.java
A fe/src/main/java/org/apache/impala/authorization/ranger/RangerImpalaResourceBuilder.java
M fe/src/main/java/org/apache/impala/authorization/sentry/SentryAuthorizationFactory.java
M fe/src/main/java/org/apache/impala/service/BackendConfig.java
A fe/src/test/resources/ranger-impala-audit.xml
A fe/src/test/resources/ranger-impala-security.xml
M impala-parent/pom.xml
M testdata/bin/create-load-data.sh
M testdata/cluster/.gitignore
A testdata/cluster/ranger/setup/impala_service.json
A testdata/cluster/ranger/setup/impala_servicedef.json
A testdata/cluster/ranger/setup/impala_user.json.template
21 files changed, 651 insertions(+), 1 deletion(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/32/12632/3
-- 
To view, visit http://gerrit.cloudera.org:8080/12632
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I8cad9e609d20aae1ff645c84fd58a02afee70276
Gerrit-Change-Number: 12632
Gerrit-PatchSet: 3
Gerrit-Owner: Fredy Wijaya <fw...@cloudera.com>
Gerrit-Reviewer: Fredy Wijaya <fw...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>

[Impala-ASF-CR] IMPALA-8100: Add initial support for Ranger

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12632 )

Change subject: IMPALA-8100: Add initial support for Ranger
......................................................................


Patch Set 9:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/2384/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/12632
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I8cad9e609d20aae1ff645c84fd58a02afee70276
Gerrit-Change-Number: 12632
Gerrit-PatchSet: 9
Gerrit-Owner: Fredy Wijaya <fw...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Fredy Wijaya <fw...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Paul Rogers <pr...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Comment-Date: Thu, 07 Mar 2019 17:08:27 +0000
Gerrit-HasComments: No