You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Anonymous Coward (Code Review)" <ge...@cloudera.org> on 2023/07/19 05:14:22 UTC
[Impala-ASF-CR] IMPALA-12291 impala checks hdfs ranger policy
rlagk741@gmail.com has uploaded this change for review. ( http://gerrit.cloudera.org:8080/20221
Change subject: IMPALA-12291 impala checks hdfs ranger policy
......................................................................
IMPALA-12291 impala checks hdfs ranger policy
Change-Id: Id33c400fbe0c918b6b65d713b09009512835a4c9
---
M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
2 files changed, 28 insertions(+), 4 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/21/20221/1
--
To view, visit http://gerrit.cloudera.org:8080/20221
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: Id33c400fbe0c918b6b65d713b09009512835a4c9
Gerrit-Change-Number: 20221
Gerrit-PatchSet: 1
Gerrit-Owner: Anonymous Coward <rl...@gmail.com>
[Impala-ASF-CR] IMPALA-12291 impala checks hdfs ranger policy
Posted by "Anonymous Coward (Code Review)" <ge...@cloudera.org>.
Hello Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/20221
to look at the new patch set (#3).
Change subject: IMPALA-12291 impala checks hdfs ranger policy
......................................................................
IMPALA-12291 impala checks hdfs ranger policy
Change-Id: Id33c400fbe0c918b6b65d713b09009512835a4c9
---
M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
2 files changed, 41 insertions(+), 7 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/21/20221/3
--
To view, visit http://gerrit.cloudera.org:8080/20221
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Id33c400fbe0c918b6b65d713b09009512835a4c9
Gerrit-Change-Number: 20221
Gerrit-PatchSet: 3
Gerrit-Owner: Anonymous Coward <rl...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
[Impala-ASF-CR] IMPALA-12291 impala checks hdfs ranger policy
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20221 )
Change subject: IMPALA-12291 impala checks hdfs ranger policy
......................................................................
Patch Set 5:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/13579/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/20221
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id33c400fbe0c918b6b65d713b09009512835a4c9
Gerrit-Change-Number: 20221
Gerrit-PatchSet: 5
Gerrit-Owner: Anonymous Coward <rl...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Wed, 19 Jul 2023 08:45:52 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12291 impala checks hdfs ranger policy
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20221 )
Change subject: IMPALA-12291 impala checks hdfs ranger policy
......................................................................
Patch Set 5:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/9520/ DRY_RUN=true
--
To view, visit http://gerrit.cloudera.org:8080/20221
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id33c400fbe0c918b6b65d713b09009512835a4c9
Gerrit-Change-Number: 20221
Gerrit-PatchSet: 5
Gerrit-Owner: Anonymous Coward <rl...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Fri, 21 Jul 2023 13:06:06 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11871: Skip permissions loading and check on HDFS if Ranger is enabled
Posted by "Halim Kim (Code Review)" <ge...@cloudera.org>.
Halim Kim has posted comments on this change. ( http://gerrit.cloudera.org:8080/20221 )
Change subject: IMPALA-11871: Skip permissions loading and check on HDFS if Ranger is enabled
......................................................................
Patch Set 14: Code-Review+1
both negative tests test_insert_with_catalog_v1 and test_load_data_with_catalog_v1 checked thank you.
--
To view, visit http://gerrit.cloudera.org:8080/20221
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id33c400fbe0c918b6b65d713b09009512835a4c9
Gerrit-Change-Number: 20221
Gerrit-PatchSet: 14
Gerrit-Owner: Halim Kim <rl...@gmail.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Halim Kim <rl...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Comment-Date: Sat, 08 Jun 2024 09:44:14 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12291 impala checks hdfs ranger policy
Posted by "Anonymous Coward (Code Review)" <ge...@cloudera.org>.
Hello Quanlong Huang, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/20221
to look at the new patch set (#9).
Change subject: IMPALA-12291 impala checks hdfs ranger policy
......................................................................
IMPALA-12291 impala checks hdfs ranger policy
Change-Id: Id33c400fbe0c918b6b65d713b09009512835a4c9
---
M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
2 files changed, 23 insertions(+), 1 deletion(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/21/20221/9
--
To view, visit http://gerrit.cloudera.org:8080/20221
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Id33c400fbe0c918b6b65d713b09009512835a4c9
Gerrit-Change-Number: 20221
Gerrit-PatchSet: 9
Gerrit-Owner: Anonymous Coward <rl...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
[Impala-ASF-CR] IMPALA-12291 impala checks hdfs ranger policy
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20221 )
Change subject: IMPALA-12291 impala checks hdfs ranger policy
......................................................................
Patch Set 9:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/13616/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/20221
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id33c400fbe0c918b6b65d713b09009512835a4c9
Gerrit-Change-Number: 20221
Gerrit-PatchSet: 9
Gerrit-Owner: Anonymous Coward <rl...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Sat, 22 Jul 2023 05:42:01 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12291 impala checks hdfs ranger policy
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20221 )
Change subject: IMPALA-12291 impala checks hdfs ranger policy
......................................................................
Patch Set 2:
Build Failed
https://jenkins.impala.io/job/gerrit-code-review-checks/13575/ : Initial code review checks failed. See linked job for details on the failure.
--
To view, visit http://gerrit.cloudera.org:8080/20221
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id33c400fbe0c918b6b65d713b09009512835a4c9
Gerrit-Change-Number: 20221
Gerrit-PatchSet: 2
Gerrit-Owner: Anonymous Coward <rl...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Wed, 19 Jul 2023 07:02:52 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12291 impala checks hdfs ranger policy
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20221 )
Change subject: IMPALA-12291 impala checks hdfs ranger policy
......................................................................
Patch Set 1:
(5 comments)
http://gerrit.cloudera.org:8080/#/c/20221/1/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
File fe/src/main/java/org/apache/impala/catalog/HdfsTable.java:
http://gerrit.cloudera.org:8080/#/c/20221/1/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java@841
PS1, Line 841: if (hdfsAuthClass != null && hdfsAuthClass.equals("org.apache.ranger.authorization.hadoop.RangerHdfsAuthorizer")) {
line too long (119 > 90)
http://gerrit.cloudera.org:8080/#/c/20221/1/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java@850
PS1, Line 850: if (hdfsRangerEnabled ? checkAccess(location, fs, FsAction.getFsAction("rw-")) : perms.canReadAndWrite()) {
line too long (115 > 90)
http://gerrit.cloudera.org:8080/#/c/20221/1/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java@852
PS1, Line 852: } else if (hdfsRangerEnabled ? checkAccess(location, fs, FsAction.getFsAction("r--")) : perms.canRead()) {
line too long (114 > 90)
http://gerrit.cloudera.org:8080/#/c/20221/1/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java@854
PS1, Line 854: } else if (hdfsRangerEnabled ? checkAccess(location, fs, FsAction.getFsAction("-w-")) : perms.canWrite()) {
line too long (115 > 90)
http://gerrit.cloudera.org:8080/#/c/20221/1/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java@3171
PS1, Line 3171: public static boolean checkAccess(Path path, FileSystem fs, FsAction action) throws IOException {
line too long (99 > 90)
--
To view, visit http://gerrit.cloudera.org:8080/20221
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id33c400fbe0c918b6b65d713b09009512835a4c9
Gerrit-Change-Number: 20221
Gerrit-PatchSet: 1
Gerrit-Owner: Anonymous Coward <rl...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Wed, 19 Jul 2023 05:15:19 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11871: Skip permissions loading and check on HDFS if Ranger is enabled
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20221 )
Change subject: IMPALA-11871: Skip permissions loading and check on HDFS if Ranger is enabled
......................................................................
Patch Set 13: Verified-1
Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/10672/
--
To view, visit http://gerrit.cloudera.org:8080/20221
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id33c400fbe0c918b6b65d713b09009512835a4c9
Gerrit-Change-Number: 20221
Gerrit-PatchSet: 13
Gerrit-Owner: Halim Kim <rl...@gmail.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Halim Kim <rl...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Comment-Date: Tue, 28 May 2024 14:48:47 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11871: Skip permissions loading and check on HDFS if Ranger is enabled
Posted by "Fang-Yu Rao (Code Review)" <ge...@cloudera.org>.
Fang-Yu Rao has posted comments on this change. ( http://gerrit.cloudera.org:8080/20221 )
Change subject: IMPALA-11871: Skip permissions loading and check on HDFS if Ranger is enabled
......................................................................
Patch Set 13:
> Patch Set 13:
>
> Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/10673/
According to https://jenkins.impala.io/job/ubuntu-20.04-dockerised-tests/1831/, TestIcebergTable.test_convert_table failed, which seems to occur before according to https://issues.apache.org/jira/browse/IMPALA-12266 (Sporadic failure after migrating a table to Iceberg). Will trigger the dryrun job again.
--
To view, visit http://gerrit.cloudera.org:8080/20221
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id33c400fbe0c918b6b65d713b09009512835a4c9
Gerrit-Change-Number: 20221
Gerrit-PatchSet: 13
Gerrit-Owner: Halim Kim <rl...@gmail.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Halim Kim <rl...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Comment-Date: Wed, 29 May 2024 08:52:38 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12291 impala checks hdfs ranger policy
Posted by "Fang-Yu Rao (Code Review)" <ge...@cloudera.org>.
Fang-Yu Rao has posted comments on this change. ( http://gerrit.cloudera.org:8080/20221 )
Change subject: IMPALA-12291 impala checks hdfs ranger policy
......................................................................
Patch Set 9:
(2 comments)
Thanks for the help Halim! I left some comments in HdfsTable.java. Let us know what you think about them.
http://gerrit.cloudera.org:8080/#/c/20221/5/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
File fe/src/main/java/org/apache/impala/catalog/HdfsTable.java:
http://gerrit.cloudera.org:8080/#/c/20221/5/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java@852
PS5, Line 852: location = location.getParent();
> Thank you for your detailed review.
Thanks for the suggestion Quanlong!
I think at the moment skipping the permissions checking is not a bad idea. Specifically, we could change the if-statement at https://gerrit.cloudera.org/c/20221/5/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java#846 to something like the following as Halim suggested above.
Note that instead of introducing a new variable denoting whether HDFS' Ranger plug-in is enabled, I just use the BackendConfig instance to verify whether the authorization provider is Ranger. Thus when Impala's Ranger plug-in is not enabled and the table is not stored on the listed file systems in the cloud, permissions checking would still occur.
So if there is any permission-related issue during the execution by Impala's backend, we will have a runtime error after this patch as Quanlong mentioned but by skipping the permissions checking we won't have a regression in performance especially when there is a large number of partitions within a table.
if (assumeReadWriteAccess(fs) ||
BackendConfig.INSTANCE.getAuthorizationProvider().equalsIgnoreCase("ranger")) {
return TAccessLevel.READ_WRITE;
}
I would also suggest we add the following end-to-end test in https://github.com/apache/impala/blob/master/tests/authorization/test_ranger.py to verify Impala frontend's behavior after this patch. I verified that it would work with my change suggested above.
@pytest.mark.execute_serially
@SkipIfFS.hdfs_acls
@CustomClusterTestSuite.with_args(
impalad_args=IMPALAD_ARGS, catalogd_args=CATALOGD_ARGS)
def test_insert_with_catalog_v1(self, unique_name):
"""
Test that when Ranger is the authorization provider in the legacy catalog mode,
Impala does not throw an AnalysisException when an authorized user tries to execute
an INSERT query against a table which is not writable according to Impala's
FsPermissionChecker.
"""
user = getuser()
admin_client = self.create_impala_client()
unique_database = unique_name + "_db"
unique_table = unique_name + "_tbl"
table_path = "test-warehouse/{0}.db/{1}".format(unique_database, unique_table)
try:
admin_client.execute("drop database if exists {0} cascade"
.format(unique_database), user=ADMIN)
admin_client.execute("create database {0}".format(unique_database), user=ADMIN)
admin_client.execute("create table {0}.{1} (x int)"
.format(unique_database, unique_table), user=ADMIN)
admin_client.execute("grant insert on table {0}.{1} to user {2}"
.format(unique_database, unique_table, user))
# Change the owner user and group of the HDFS path corresponding to the table
# so that according to Impala's FsPermissionChecker, the table could not be
# writable to the user that loads the table. This user usually is the one
# representing the Impala service.
self.hdfs_client.chown(table_path, "another_user", "another_group")
# Invalidate the table metadata to force the catalog server to reload the HDFS
# table.
admin_client.execute("invalidate metadata {0}.{1}"
.format(unique_database, unique_table), user=ADMIN)
# Verify that the user granted the INSERT privilege on the table does not encounter
# the AnalysisException that would have been thrown if we had not skipped the
# permissions checking in HdfsTable#getAvailableAccessLevel().
self._run_query_as_user("insert into {0}.{1} values (1)"
.format(unique_database, unique_table), user, True)
finally:
admin_client.execute("revoke insert on table {0}.{1} from user {2}"
.format(unique_database, unique_table, user))
admin_client.execute("drop database if exists {0} cascade"
.format(unique_database), user=ADMIN)
http://gerrit.cloudera.org:8080/#/c/20221/5/fe/src/main/java/org/apache/impala/service/Frontend.java
File fe/src/main/java/org/apache/impala/service/Frontend.java:
http://gerrit.cloudera.org:8080/#/c/20221/5/fe/src/main/java/org/apache/impala/service/Frontend.java@2109
PS5, Line 2109: LOG.warn("Analysis Exception query {}: {}",
: queryCtx.client_request.stmt, errorMsg);
Not very sure if I missed something. Since we will throw the exception that is caught, is it necessary to log or even catch the exception here?
--
To view, visit http://gerrit.cloudera.org:8080/20221
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id33c400fbe0c918b6b65d713b09009512835a4c9
Gerrit-Change-Number: 20221
Gerrit-PatchSet: 9
Gerrit-Owner: Halim Kim <rl...@gmail.com>
Gerrit-Reviewer: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Wed, 03 Apr 2024 00:46:18 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12291 impala checks hdfs ranger policy
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20221 )
Change subject: IMPALA-12291 impala checks hdfs ranger policy
......................................................................
Patch Set 3:
Build Failed
https://jenkins.impala.io/job/gerrit-code-review-checks/13576/ : Initial code review checks failed. See linked job for details on the failure.
--
To view, visit http://gerrit.cloudera.org:8080/20221
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id33c400fbe0c918b6b65d713b09009512835a4c9
Gerrit-Change-Number: 20221
Gerrit-PatchSet: 3
Gerrit-Owner: Anonymous Coward <rl...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Wed, 19 Jul 2023 07:47:32 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12291 impala checks hdfs ranger policy
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20221 )
Change subject: IMPALA-12291 impala checks hdfs ranger policy
......................................................................
Patch Set 7:
(3 comments)
http://gerrit.cloudera.org:8080/#/c/20221/7/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
File fe/src/main/java/org/apache/impala/catalog/HdfsTable.java:
http://gerrit.cloudera.org:8080/#/c/20221/7/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java@900
PS7, Line 900:
line has trailing whitespace
http://gerrit.cloudera.org:8080/#/c/20221/7/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java@902
PS7, Line 902: // It is possible to check existence of proper permissions with hadoop namenode rpc
line has trailing whitespace
http://gerrit.cloudera.org:8080/#/c/20221/7/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java@904
PS7, Line 904: // of table partitions get bigger. Therefore, Assume all hdfs files have READ_WRITE
line has trailing whitespace
--
To view, visit http://gerrit.cloudera.org:8080/20221
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id33c400fbe0c918b6b65d713b09009512835a4c9
Gerrit-Change-Number: 20221
Gerrit-PatchSet: 7
Gerrit-Owner: Anonymous Coward <rl...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Sat, 22 Jul 2023 05:17:09 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11871: Skip permissions loading and check on HDFS if Ranger is enabled
Posted by "Fang-Yu Rao (Code Review)" <ge...@cloudera.org>.
Fang-Yu Rao has posted comments on this change. ( http://gerrit.cloudera.org:8080/20221 )
Change subject: IMPALA-11871: Skip permissions loading and check on HDFS if Ranger is enabled
......................................................................
Patch Set 11:
I revised the commit message and added an end-to-end test for the LOAD DATA statement in the patch set 11.
--
To view, visit http://gerrit.cloudera.org:8080/20221
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id33c400fbe0c918b6b65d713b09009512835a4c9
Gerrit-Change-Number: 20221
Gerrit-PatchSet: 11
Gerrit-Owner: Halim Kim <rl...@gmail.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Halim Kim <rl...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Sat, 13 Apr 2024 00:59:39 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12291 impala checks hdfs ranger policy
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20221 )
Change subject: IMPALA-12291 impala checks hdfs ranger policy
......................................................................
Patch Set 8:
(3 comments)
http://gerrit.cloudera.org:8080/#/c/20221/8/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
File fe/src/main/java/org/apache/impala/catalog/HdfsTable.java:
http://gerrit.cloudera.org:8080/#/c/20221/8/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java@900
PS8, Line 900:
line has trailing whitespace
http://gerrit.cloudera.org:8080/#/c/20221/8/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java@902
PS8, Line 902: // It is possible to check existence of proper permissions with hadoop namenode rpc
line has trailing whitespace
http://gerrit.cloudera.org:8080/#/c/20221/8/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java@904
PS8, Line 904: // of table partitions get bigger. Therefore, Assume all hdfs files have READ_WRITE
line has trailing whitespace
--
To view, visit http://gerrit.cloudera.org:8080/20221
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id33c400fbe0c918b6b65d713b09009512835a4c9
Gerrit-Change-Number: 20221
Gerrit-PatchSet: 8
Gerrit-Owner: Anonymous Coward <rl...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Sat, 22 Jul 2023 05:18:08 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12291 impala checks hdfs ranger policy
Posted by "Anonymous Coward (Code Review)" <ge...@cloudera.org>.
Hello Quanlong Huang, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/20221
to look at the new patch set (#6).
Change subject: IMPALA-12291 impala checks hdfs ranger policy
......................................................................
IMPALA-12291 impala checks hdfs ranger policy
Change-Id: Id33c400fbe0c918b6b65d713b09009512835a4c9
---
M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
2 files changed, 26 insertions(+), 4 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/21/20221/6
--
To view, visit http://gerrit.cloudera.org:8080/20221
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Id33c400fbe0c918b6b65d713b09009512835a4c9
Gerrit-Change-Number: 20221
Gerrit-PatchSet: 6
Gerrit-Owner: Anonymous Coward <rl...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
[Impala-ASF-CR] IMPALA-12291 impala checks hdfs ranger policy
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20221 )
Change subject: IMPALA-12291 impala checks hdfs ranger policy
......................................................................
Patch Set 9: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/20221
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id33c400fbe0c918b6b65d713b09009512835a4c9
Gerrit-Change-Number: 20221
Gerrit-PatchSet: 9
Gerrit-Owner: Anonymous Coward <rl...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Sat, 22 Jul 2023 16:12:03 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11871: Skip permissions loading and check on HDFS if Ranger is enabled
Posted by "Yida Wu (Code Review)" <ge...@cloudera.org>.
Yida Wu has posted comments on this change. ( http://gerrit.cloudera.org:8080/20221 )
Change subject: IMPALA-11871: Skip permissions loading and check on HDFS if Ranger is enabled
......................................................................
Patch Set 11:
(3 comments)
http://gerrit.cloudera.org:8080/#/c/20221/11/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
File fe/src/main/java/org/apache/impala/catalog/HdfsTable.java:
http://gerrit.cloudera.org:8080/#/c/20221/11/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java@931
PS11, Line 931:
nit. unnecessary newline
http://gerrit.cloudera.org:8080/#/c/20221/11/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java@3241
PS11, Line 3241:
nit. unnecessary newline
http://gerrit.cloudera.org:8080/#/c/20221/11/tests/authorization/test_ranger.py
File tests/authorization/test_ranger.py:
http://gerrit.cloudera.org:8080/#/c/20221/11/tests/authorization/test_ranger.py@87
PS11, Line 87: Impala does not throw an AnalysisException
Am I right in saying that if a ranger user doesn't have write permissions, this patch will allow the analysis check to pass but eventually result in another exception in this insertion case because the user lacks the necessary rights?
--
To view, visit http://gerrit.cloudera.org:8080/20221
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id33c400fbe0c918b6b65d713b09009512835a4c9
Gerrit-Change-Number: 20221
Gerrit-PatchSet: 11
Gerrit-Owner: Halim Kim <rl...@gmail.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Halim Kim <rl...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Comment-Date: Sat, 25 May 2024 16:05:30 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11871: Skip permissions loading and check on HDFS if Ranger is enabled
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20221 )
Change subject: IMPALA-11871: Skip permissions loading and check on HDFS if Ranger is enabled
......................................................................
Patch Set 13:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/16230/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/20221
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id33c400fbe0c918b6b65d713b09009512835a4c9
Gerrit-Change-Number: 20221
Gerrit-PatchSet: 13
Gerrit-Owner: Halim Kim <rl...@gmail.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Halim Kim <rl...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Comment-Date: Tue, 28 May 2024 10:02:55 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12291 impala checks hdfs ranger policy
Posted by "Anonymous Coward (Code Review)" <ge...@cloudera.org>.
Hello Quanlong Huang, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/20221
to look at the new patch set (#8).
Change subject: IMPALA-12291 impala checks hdfs ranger policy
......................................................................
IMPALA-12291 impala checks hdfs ranger policy
Change-Id: Id33c400fbe0c918b6b65d713b09009512835a4c9
---
M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
2 files changed, 23 insertions(+), 1 deletion(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/21/20221/8
--
To view, visit http://gerrit.cloudera.org:8080/20221
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Id33c400fbe0c918b6b65d713b09009512835a4c9
Gerrit-Change-Number: 20221
Gerrit-PatchSet: 8
Gerrit-Owner: Anonymous Coward <rl...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
[Impala-ASF-CR] IMPALA-12291 impala checks hdfs ranger policy
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20221 )
Change subject: IMPALA-12291 impala checks hdfs ranger policy
......................................................................
Patch Set 7:
Build Failed
https://jenkins.impala.io/job/gerrit-code-review-checks/13614/ : Initial code review checks failed. See linked job for details on the failure.
--
To view, visit http://gerrit.cloudera.org:8080/20221
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id33c400fbe0c918b6b65d713b09009512835a4c9
Gerrit-Change-Number: 20221
Gerrit-PatchSet: 7
Gerrit-Owner: Anonymous Coward <rl...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Sat, 22 Jul 2023 05:31:37 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12291 impala checks hdfs ranger policy
Posted by "Anonymous Coward (Code Review)" <ge...@cloudera.org>.
rlagk741@gmail.com has removed Impala Public Jenkins from this change. ( http://gerrit.cloudera.org:8080/20221 )
Change subject: IMPALA-12291 impala checks hdfs ranger policy
......................................................................
Removed reviewer Impala Public Jenkins.
--
To view, visit http://gerrit.cloudera.org:8080/20221
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: deleteReviewer
Gerrit-Change-Id: Id33c400fbe0c918b6b65d713b09009512835a4c9
Gerrit-Change-Number: 20221
Gerrit-PatchSet: 4
Gerrit-Owner: Anonymous Coward <rl...@gmail.com>
[Impala-ASF-CR] IMPALA-11871: Skip permissions loading and check on HDFS if Ranger is enabled
Posted by "Fang-Yu Rao (Code Review)" <ge...@cloudera.org>.
Fang-Yu Rao has posted comments on this change. ( http://gerrit.cloudera.org:8080/20221 )
Change subject: IMPALA-11871: Skip permissions loading and check on HDFS if Ranger is enabled
......................................................................
Patch Set 11:
Hi all, please let me know if there are additional comments on this patch. Thank you very much for the help!
--
To view, visit http://gerrit.cloudera.org:8080/20221
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id33c400fbe0c918b6b65d713b09009512835a4c9
Gerrit-Change-Number: 20221
Gerrit-PatchSet: 11
Gerrit-Owner: Halim Kim <rl...@gmail.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Halim Kim <rl...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Tue, 07 May 2024 08:53:20 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11871: Skip permissions check on HDFS files if Ranger is enabled
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20221 )
Change subject: IMPALA-11871: Skip permissions check on HDFS files if Ranger is enabled
......................................................................
Patch Set 10:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/10528/ DRY_RUN=true
--
To view, visit http://gerrit.cloudera.org:8080/20221
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id33c400fbe0c918b6b65d713b09009512835a4c9
Gerrit-Change-Number: 20221
Gerrit-PatchSet: 10
Gerrit-Owner: Halim Kim <rl...@gmail.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Halim Kim <rl...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Thu, 11 Apr 2024 01:16:53 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11871: Skip permissions loading and check on HDFS if Ranger is enabled
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20221 )
Change subject: IMPALA-11871: Skip permissions loading and check on HDFS if Ranger is enabled
......................................................................
Patch Set 13:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/10673/ DRY_RUN=true
--
To view, visit http://gerrit.cloudera.org:8080/20221
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id33c400fbe0c918b6b65d713b09009512835a4c9
Gerrit-Change-Number: 20221
Gerrit-PatchSet: 13
Gerrit-Owner: Halim Kim <rl...@gmail.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Halim Kim <rl...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Comment-Date: Wed, 29 May 2024 03:44:00 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11871: Skip permissions loading and check on HDFS if Ranger is enabled
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20221 )
Change subject: IMPALA-11871: Skip permissions loading and check on HDFS if Ranger is enabled
......................................................................
Patch Set 13:
Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/10674/
--
To view, visit http://gerrit.cloudera.org:8080/20221
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id33c400fbe0c918b6b65d713b09009512835a4c9
Gerrit-Change-Number: 20221
Gerrit-PatchSet: 13
Gerrit-Owner: Halim Kim <rl...@gmail.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Halim Kim <rl...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Comment-Date: Wed, 29 May 2024 13:55:06 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12291 impala checks hdfs ranger policy
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20221 )
Change subject: IMPALA-12291 impala checks hdfs ranger policy
......................................................................
Patch Set 9:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/9523/ DRY_RUN=true
--
To view, visit http://gerrit.cloudera.org:8080/20221
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id33c400fbe0c918b6b65d713b09009512835a4c9
Gerrit-Change-Number: 20221
Gerrit-PatchSet: 9
Gerrit-Owner: Anonymous Coward <rl...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Sat, 22 Jul 2023 11:51:18 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12291 impala checks hdfs ranger policy
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20221 )
Change subject: IMPALA-12291 impala checks hdfs ranger policy
......................................................................
Patch Set 6:
(3 comments)
http://gerrit.cloudera.org:8080/#/c/20221/6/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
File fe/src/main/java/org/apache/impala/catalog/HdfsTable.java:
http://gerrit.cloudera.org:8080/#/c/20221/6/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java@900
PS6, Line 900:
line has trailing whitespace
http://gerrit.cloudera.org:8080/#/c/20221/6/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java@902
PS6, Line 902: // It is possible to check existence of proper permissions with hadoop namenode rpc
line has trailing whitespace
http://gerrit.cloudera.org:8080/#/c/20221/6/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java@904
PS6, Line 904: // of table partitions get bigger. Therefore, Assume all hdfs files have READ_WRITE
line has trailing whitespace
--
To view, visit http://gerrit.cloudera.org:8080/20221
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id33c400fbe0c918b6b65d713b09009512835a4c9
Gerrit-Change-Number: 20221
Gerrit-PatchSet: 6
Gerrit-Owner: Anonymous Coward <rl...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Sat, 22 Jul 2023 05:15:33 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11871: Skip permissions loading and check on HDFS if Ranger is enabled
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20221 )
Change subject: IMPALA-11871: Skip permissions loading and check on HDFS if Ranger is enabled
......................................................................
Patch Set 13:
Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/10673/
--
To view, visit http://gerrit.cloudera.org:8080/20221
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id33c400fbe0c918b6b65d713b09009512835a4c9
Gerrit-Change-Number: 20221
Gerrit-PatchSet: 13
Gerrit-Owner: Halim Kim <rl...@gmail.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Halim Kim <rl...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Comment-Date: Wed, 29 May 2024 08:45:25 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11871: Skip permissions loading and check on HDFS if Ranger is enabled
Posted by "Fang-Yu Rao (Code Review)" <ge...@cloudera.org>.
Fang-Yu Rao has uploaded a new patch set (#13) to the change originally created by Halim Kim. ( http://gerrit.cloudera.org:8080/20221 )
Change subject: IMPALA-11871: Skip permissions loading and check on HDFS if Ranger is enabled
......................................................................
IMPALA-11871: Skip permissions loading and check on HDFS if Ranger is enabled
Before this patch, Impala checked whether the Impala service user had
the WRITE access to the target HDFS table/partition(s) during the
analysis of the INSERT and LOAD DATA statements in the legacy catalog
mode. The access levels of the corresponding HDFS table and partitions
were computed by the catalog server solely based on the HDFS permissions
and ACLs when the table and partitions were instantiated.
After this patch, we skip loading HDFS permissions and assume the
Impala service user has the READ_WRITE permission on all the HDFS paths
associated with the target table during query analysis when Ranger is
enabled. The assumption could be removed after Impala's implementation
of FsPermissionChecker could additionally take Ranger's policies of HDFS
into consideration when performing the check.
Testing:
- Added end-to-end tests to verify Impala's behavior with respect to
the INSERT and LOAD DATA statements when Ranger is enabled in the
legacy catalog mode.
Change-Id: Id33c400fbe0c918b6b65d713b09009512835a4c9
---
M bin/rat_exclude_files.txt
M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
A testdata/data/load_data_with_catalog_v1.txt
M tests/authorization/test_ranger.py
4 files changed, 163 insertions(+), 1 deletion(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/21/20221/13
--
To view, visit http://gerrit.cloudera.org:8080/20221
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Id33c400fbe0c918b6b65d713b09009512835a4c9
Gerrit-Change-Number: 20221
Gerrit-PatchSet: 13
Gerrit-Owner: Halim Kim <rl...@gmail.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Halim Kim <rl...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
[Impala-ASF-CR] IMPALA-11871: Skip permissions loading and check on HDFS if Ranger is enabled
Posted by "Fang-Yu Rao (Code Review)" <ge...@cloudera.org>.
Fang-Yu Rao has posted comments on this change. ( http://gerrit.cloudera.org:8080/20221 )
Change subject: IMPALA-11871: Skip permissions loading and check on HDFS if Ranger is enabled
......................................................................
Patch Set 13:
> Patch Set 13: Verified-1
>
> Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/10672/
According to https://jenkins.impala.io/job/gerrit-verify-dryrun/10672/console and https://jenkins.impala.io/job/ubuntu-20.04-dockerised-tests/1830/, TestResultSpooling.test_slow_query was the only failed test. But I was not able to reproduce the failed test locally. I will trigger the dryrun job again to see if we could get a successful run this time.
--
To view, visit http://gerrit.cloudera.org:8080/20221
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id33c400fbe0c918b6b65d713b09009512835a4c9
Gerrit-Change-Number: 20221
Gerrit-PatchSet: 13
Gerrit-Owner: Halim Kim <rl...@gmail.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Halim Kim <rl...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Comment-Date: Wed, 29 May 2024 03:43:28 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11871: Skip permissions loading and check on HDFS if Ranger is enabled
Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/20221 )
Change subject: IMPALA-11871: Skip permissions loading and check on HDFS if Ranger is enabled
......................................................................
Patch Set 14: Code-Review+2
Carry +1 from other reviewers. Thank Fang-Yu for making this merged! Thank Halim for the contribution!
--
To view, visit http://gerrit.cloudera.org:8080/20221
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id33c400fbe0c918b6b65d713b09009512835a4c9
Gerrit-Change-Number: 20221
Gerrit-PatchSet: 14
Gerrit-Owner: Halim Kim <rl...@gmail.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Halim Kim <rl...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Comment-Date: Tue, 11 Jun 2024 01:21:31 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12291 impala checks hdfs ranger policy
Posted by "Aman Sinha (Code Review)" <ge...@cloudera.org>.
Aman Sinha has posted comments on this change. ( http://gerrit.cloudera.org:8080/20221 )
Change subject: IMPALA-12291 impala checks hdfs ranger policy
......................................................................
Patch Set 9:
> Patch Set 9:
>
> Hi all, based on our discussion so far, there are 2 approaches discussed. Any any better alternative is also appreciated!
> - Set the access level of a loaded table to both READ and WRITE as long as Ranger is used as the authorization provider.
> - Introduce a startup flag to allow the administrator to decide whether to skip the file system permissions check during table loading.
>
> There are some implementation details to consider for each approach.
> - How to determine whether Ranger is enabled (corresponding to the 1st approach)? It seems checking the value of the key 'dfs.namenode.inode.attributes.provider.class' in core-site.xml via the instance of Configuration as done in the patch set 9 could not be verified easily via a new test due to HDFS Ranger plug-in not being configured correctly. To be more specific, if we try to add the following configuration via https://github.com/apache/impala/blob/master/testdata/cluster/node_templates/common/etc/hadoop/conf/core-site.xml.py, the name node of HDFS could not be started. More plumbing has to be done to set up the plug-in. On the other hand, using the instance of BackendConfig allows us to add an end-to-end test to briefly verify Impala's behavior after the patch more easily.
>
> <property>
> <name>dfs.namenode.inode.attributes.provider.class</name>
> <value>org.apache.ranger.authorization.hadoop.RangerHdfsAuthorizer</value>
> </property>
>
> - What should be the default behavior with respect to setting the access level of a loaded table after this patch (corresponding to the 2nd approach)? It looks like making Impala assume the READ and WRITE access by default may be better especially in the legacy catalog mode. This relieves the Impala administrator of having to manually tweak the HDFS access control entries of the target HDFS paths that are not writable to the Impala service every time when an end user encounters such a problem.
>
> I have also collected the related tests that need to be revised if we decide to adopt the 2nd approach and make Impala assume the READ and WRITE access by default.
> - https://github.com/apache/impala/blob/master/tests/metadata/test_hdfs_permissions.py#L56 (TestHdfsPermissions.test_insert_into_read_only_table()).
> - https://github.com/apache/impala/blob/master/tests/query_test/test_insert_behaviour.py#L563 (TestInsertBehaviour.test_multiple_group_acls).
> - https://github.com/apache/impala/blob/master/tests/query_test/test_insert_behaviour.py#L331 (TestInsertBehaviour.test_readonly_table_dir).
> - https://github.com/apache/impala/blob/master/tests/query_test/test_insert_behaviour.py#L362 (TestInsertBehaviour.test_insert_acl_permissions).
> - https://github.com/apache/impala/blob/master/tests/query_test/test_insert_behaviour.py#L439 (TestInsertBehaviour.test_load_permissions).
> - https://github.com/apache/impala/blob/master/tests/query_test/test_insert_behaviour.py#L252 (TestInsertBehaviour.test_mixed_partition_permissions).
> - https://github.com/apache/impala/blob/master/tests/query_test/test_insert_behaviour.py#L202 (TestInsertBehaviour.test_insert_file_permissions).
> - https://github.com/apache/impala/blob/master/fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java#L4008 (AnalyzeStmtsTest.TestLoadData).
> - https://github.com/apache/impala/blob/master/fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java#L4042-L4047 (AnalyzeStmtsTest.TestInsert).
> - https://github.com/apache/impala/blob/master/fe/src/test/java/org/apache/impala/catalog/CatalogObjectToFromThriftTest.java#L110-L119 (CatalogObjectToFromThriftTest.TestPartitionedTable).
The first option sounds good to me:
- Set the access level of a loaded table to both READ and WRITE as long as Ranger is used as the authorization provider.
The second option of introducing a startup flag would create confusion because it would only apply to the legacy catalog mode, not the local catalog mode. Further, for the cloud object stores, we are in any case doing the first approach.
The downside of the first approach is that it will defer the full access level checks to execution time and suppose some partitions allow the write operation while other partitions don't, we can end up with partial writes into the table. However, this issue already exists with the local catalog mode (which is the default mode).
--
To view, visit http://gerrit.cloudera.org:8080/20221
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id33c400fbe0c918b6b65d713b09009512835a4c9
Gerrit-Change-Number: 20221
Gerrit-PatchSet: 9
Gerrit-Owner: Halim Kim <rl...@gmail.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Halim Kim <rl...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Fri, 05 Apr 2024 23:37:30 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11871: Skip permissions check on HDFS files if Ranger is enabled
Posted by "Fang-Yu Rao (Code Review)" <ge...@cloudera.org>.
Fang-Yu Rao has uploaded a new patch set (#10) to the change originally created by Halim Kim. ( http://gerrit.cloudera.org:8080/20221 )
Change subject: IMPALA-11871: Skip permissions check on HDFS files if Ranger is enabled
......................................................................
IMPALA-11871: Skip permissions check on HDFS files if Ranger is enabled
This patch assumes the Impala service user has the READ_WRITE
permission on all the HDFS paths associated with a table during query
analysis when Ranger is enabled. The assumption could be removed after
Impala's implementation of FsPermissionChecker could additionally take
Ranger's policies of HDFS into consideration when performing the check.
Testing:
- Added an end-to-end test to verify Impala's behavior with respect to
the INSERT statement when Ranger is enabled in the legacy catalog
mode.
Change-Id: Id33c400fbe0c918b6b65d713b09009512835a4c9
---
M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
M tests/authorization/test_ranger.py
2 files changed, 59 insertions(+), 1 deletion(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/21/20221/10
--
To view, visit http://gerrit.cloudera.org:8080/20221
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Id33c400fbe0c918b6b65d713b09009512835a4c9
Gerrit-Change-Number: 20221
Gerrit-PatchSet: 10
Gerrit-Owner: Halim Kim <rl...@gmail.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Halim Kim <rl...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
[Impala-ASF-CR] IMPALA-11871: Skip permissions check on HDFS files if Ranger is enabled
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20221 )
Change subject: IMPALA-11871: Skip permissions check on HDFS files if Ranger is enabled
......................................................................
Patch Set 10: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/20221
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id33c400fbe0c918b6b65d713b09009512835a4c9
Gerrit-Change-Number: 20221
Gerrit-PatchSet: 10
Gerrit-Owner: Halim Kim <rl...@gmail.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Halim Kim <rl...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Thu, 11 Apr 2024 06:27:29 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11871: Skip permissions loading and check on HDFS if Ranger is enabled
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20221 )
Change subject: IMPALA-11871: Skip permissions loading and check on HDFS if Ranger is enabled
......................................................................
Patch Set 14: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/20221
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id33c400fbe0c918b6b65d713b09009512835a4c9
Gerrit-Change-Number: 20221
Gerrit-PatchSet: 14
Gerrit-Owner: Halim Kim <rl...@gmail.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Halim Kim <rl...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Comment-Date: Thu, 30 May 2024 07:58:04 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11871: Skip permissions loading and check on HDFS if Ranger is enabled
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20221 )
Change subject: IMPALA-11871: Skip permissions loading and check on HDFS if Ranger is enabled
......................................................................
Patch Set 12:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/16229/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/20221
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id33c400fbe0c918b6b65d713b09009512835a4c9
Gerrit-Change-Number: 20221
Gerrit-PatchSet: 12
Gerrit-Owner: Halim Kim <rl...@gmail.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Halim Kim <rl...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Comment-Date: Tue, 28 May 2024 10:01:31 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12291 impala checks hdfs ranger policy
Posted by "Halim Kim (Code Review)" <ge...@cloudera.org>.
Halim Kim has posted comments on this change. ( http://gerrit.cloudera.org:8080/20221 )
Change subject: IMPALA-12291 impala checks hdfs ranger policy
......................................................................
Patch Set 9:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/20221/5/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
File fe/src/main/java/org/apache/impala/catalog/HdfsTable.java:
http://gerrit.cloudera.org:8080/#/c/20221/5/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java@852
PS5, Line 852: location = location.getParent();
> Thanks for the suggestion Quanlong!
Your test code will be very helpful. Thank you Fang-Yu.
I agree with the idea to hdfs permission check in query analysis when ranger is enabled. however, It seems like query planning gets less thorough just because ranger is enabled.
how about adding a flag to decide whether hdfs permission should be checked or not like --hdfs_permission_check=true (or false)?
I think it is more reasonable to give user a choice. In other words, Asking user if you want a AnalysisException or Runtime Exception.
Of course, we should add guidance to use --hdfs_permission_check flag. Recommend to disable it when ranger is enabled.
http://gerrit.cloudera.org:8080/#/c/20221/5/fe/src/main/java/org/apache/impala/service/Frontend.java
File fe/src/main/java/org/apache/impala/service/Frontend.java:
http://gerrit.cloudera.org:8080/#/c/20221/5/fe/src/main/java/org/apache/impala/service/Frontend.java@2109
PS5, Line 2109: LOG.warn("Analysis Exception query {}: {}",
: queryCtx.client_request.stmt, errorMsg);
> Not very sure if I missed something. Since we will throw the exception that
In retrospect, I think I found it difficult to know which query is in trouble so that I put one more catch statement to catch the query.
But now, I realize that there is already AnalysisException message like below.
```
AnalysisException: Unable to INSERT into target table (testdb.testtable) because Impala does not have WRITE access to HDFS location: hdfs://testcluster/warehouse/testdb.db/testtable
```
To conclude, It is not necessary to catch like that because we can find which query has permission issue by an AnalysisException thrown.
Thank you for commenting.
--
To view, visit http://gerrit.cloudera.org:8080/20221
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id33c400fbe0c918b6b65d713b09009512835a4c9
Gerrit-Change-Number: 20221
Gerrit-PatchSet: 9
Gerrit-Owner: Halim Kim <rl...@gmail.com>
Gerrit-Reviewer: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Halim Kim <rl...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Wed, 03 Apr 2024 02:53:02 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12291 impala checks hdfs ranger policy
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20221 )
Change subject: IMPALA-12291 impala checks hdfs ranger policy
......................................................................
Patch Set 1:
Build Failed
https://jenkins.impala.io/job/gerrit-code-review-checks/13574/ : Initial code review checks failed. See linked job for details on the failure.
--
To view, visit http://gerrit.cloudera.org:8080/20221
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id33c400fbe0c918b6b65d713b09009512835a4c9
Gerrit-Change-Number: 20221
Gerrit-PatchSet: 1
Gerrit-Owner: Anonymous Coward <rl...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Wed, 19 Jul 2023 05:29:31 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12291 impala checks hdfs ranger policy
Posted by "Anonymous Coward (Code Review)" <ge...@cloudera.org>.
Hello Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/20221
to look at the new patch set (#2).
Change subject: IMPALA-12291 impala checks hdfs ranger policy
......................................................................
IMPALA-12291 impala checks hdfs ranger policy
Change-Id: Id33c400fbe0c918b6b65d713b09009512835a4c9
---
M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
2 files changed, 42 insertions(+), 7 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/21/20221/2
--
To view, visit http://gerrit.cloudera.org:8080/20221
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Id33c400fbe0c918b6b65d713b09009512835a4c9
Gerrit-Change-Number: 20221
Gerrit-PatchSet: 2
Gerrit-Owner: Anonymous Coward <rl...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
[Impala-ASF-CR] IMPALA-12291 impala checks hdfs ranger policy
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20221 )
Change subject: IMPALA-12291 impala checks hdfs ranger policy
......................................................................
Patch Set 8:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/13615/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/20221
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id33c400fbe0c918b6b65d713b09009512835a4c9
Gerrit-Change-Number: 20221
Gerrit-PatchSet: 8
Gerrit-Owner: Anonymous Coward <rl...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Sat, 22 Jul 2023 05:39:41 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12291 impala checks hdfs ranger policy
Posted by "Fang-Yu Rao (Code Review)" <ge...@cloudera.org>.
Fang-Yu Rao has posted comments on this change. ( http://gerrit.cloudera.org:8080/20221 )
Change subject: IMPALA-12291 impala checks hdfs ranger policy
......................................................................
Patch Set 9:
Hi all, based on our discussion so far, there are 2 approaches discussed. Any any better alternative is also appreciated!
- Set the access level of a loaded table to both READ and WRITE as long as Ranger is used as the authorization provider.
- Introduce a startup flag to allow the administrator to decide whether to skip the file system permissions check during table loading.
There are some implementation details to consider for each approach.
- How to determine whether Ranger is enabled (corresponding to the 1st approach)? It seems checking the value of the key 'dfs.namenode.inode.attributes.provider.class' in core-site.xml via the instance of Configuration as done in the patch set 9 could not be verified easily via a new test due to HDFS Ranger plug-in not being configured correctly. To be more specific, if we try to add the following configuration via https://github.com/apache/impala/blob/master/testdata/cluster/node_templates/common/etc/hadoop/conf/core-site.xml.py, the name node of HDFS could not be started. More plumbing has to be done to set up the plug-in. On the other hand, using the instance of BackendConfig allows us to add an end-to-end test to briefly verify Impala's behavior after the patch more easily.
<property>
<name>dfs.namenode.inode.attributes.provider.class</name>
<value>org.apache.ranger.authorization.hadoop.RangerHdfsAuthorizer</value>
</property>
- What should be the default behavior with respect to setting the access level of a loaded table after this patch (corresponding to the 2nd approach)? It looks like making Impala assume the READ and WRITE access by default may be better especially in the legacy catalog mode. This relieves the Impala administrator of having to manually tweak the HDFS access control entries of the target HDFS paths that are not writable to the Impala service every time when an end user encounters such a problem.
I have also collected the related tests that need to be revised if we decide to adopt the 2nd approach and make Impala assume the READ and WRITE access by default.
- https://github.com/apache/impala/blob/master/tests/metadata/test_hdfs_permissions.py#L56 (TestHdfsPermissions.test_insert_into_read_only_table()).
- https://github.com/apache/impala/blob/master/tests/query_test/test_insert_behaviour.py#L563 (TestInsertBehaviour.test_multiple_group_acls).
- https://github.com/apache/impala/blob/master/tests/query_test/test_insert_behaviour.py#L331 (TestInsertBehaviour.test_readonly_table_dir).
- https://github.com/apache/impala/blob/master/tests/query_test/test_insert_behaviour.py#L362 (TestInsertBehaviour.test_insert_acl_permissions).
- https://github.com/apache/impala/blob/master/tests/query_test/test_insert_behaviour.py#L439 (TestInsertBehaviour.test_load_permissions).
- https://github.com/apache/impala/blob/master/tests/query_test/test_insert_behaviour.py#L252 (TestInsertBehaviour.test_mixed_partition_permissions).
- https://github.com/apache/impala/blob/master/tests/query_test/test_insert_behaviour.py#L202 (TestInsertBehaviour.test_insert_file_permissions).
- https://github.com/apache/impala/blob/master/fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java#L4008 (AnalyzeStmtsTest.TestLoadData).
- https://github.com/apache/impala/blob/master/fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java#L4042-L4047 (AnalyzeStmtsTest.TestInsert).
- https://github.com/apache/impala/blob/master/fe/src/test/java/org/apache/impala/catalog/CatalogObjectToFromThriftTest.java#L110-L119 (CatalogObjectToFromThriftTest.TestPartitionedTable).
--
To view, visit http://gerrit.cloudera.org:8080/20221
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id33c400fbe0c918b6b65d713b09009512835a4c9
Gerrit-Change-Number: 20221
Gerrit-PatchSet: 9
Gerrit-Owner: Halim Kim <rl...@gmail.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Halim Kim <rl...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Fri, 05 Apr 2024 18:30:54 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11871: Skip permissions loading and check on HDFS if Ranger is enabled
Posted by "Fang-Yu Rao (Code Review)" <ge...@cloudera.org>.
Fang-Yu Rao has posted comments on this change. ( http://gerrit.cloudera.org:8080/20221 )
Change subject: IMPALA-11871: Skip permissions loading and check on HDFS if Ranger is enabled
......................................................................
Patch Set 14:
> Patch Set 13:
>
> Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/10674/
According to https://jenkins.impala.io/job/ubuntu-20.04-from-scratch/2689/, the following 2 frontend tests failed.
- PlannerTest.testFkPkJoinDetectionWithHDFSNumRowsEstDisabled
- PlannerTest.testSimplifyOuterJoins
The corresponding golden files are the following.
- https://github.com/apache/impala/blob/master/testdata/workloads/functional-planner/queries/PlannerTest/fk-pk-join-detection.test
- https://github.com/apache/impala/blob/master/testdata/workloads/functional-planner/queries/PlannerTest/outer-to-inner-joins.test
It's unlikely to be caused by this patch. I will thus re-trigger a dryrun job and see if there are still failed tests.
--
To view, visit http://gerrit.cloudera.org:8080/20221
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id33c400fbe0c918b6b65d713b09009512835a4c9
Gerrit-Change-Number: 20221
Gerrit-PatchSet: 14
Gerrit-Owner: Halim Kim <rl...@gmail.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Halim Kim <rl...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Comment-Date: Thu, 30 May 2024 02:48:48 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11871: Skip permissions loading and check on HDFS if Ranger is enabled
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20221 )
Change subject: IMPALA-11871: Skip permissions loading and check on HDFS if Ranger is enabled
......................................................................
Patch Set 15: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/20221
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id33c400fbe0c918b6b65d713b09009512835a4c9
Gerrit-Change-Number: 20221
Gerrit-PatchSet: 15
Gerrit-Owner: Halim Kim <rl...@gmail.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Halim Kim <rl...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Comment-Date: Tue, 11 Jun 2024 01:22:10 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11871: Skip permissions loading and check on HDFS if Ranger is enabled
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20221 )
Change subject: IMPALA-11871: Skip permissions loading and check on HDFS if Ranger is enabled
......................................................................
Patch Set 15: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/20221
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id33c400fbe0c918b6b65d713b09009512835a4c9
Gerrit-Change-Number: 20221
Gerrit-PatchSet: 15
Gerrit-Owner: Halim Kim <rl...@gmail.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Halim Kim <rl...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Comment-Date: Tue, 11 Jun 2024 06:28:51 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11871: Skip permissions loading and check on HDFS if Ranger is enabled
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20221 )
Change subject: IMPALA-11871: Skip permissions loading and check on HDFS if Ranger is enabled
......................................................................
Patch Set 11:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/15887/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/20221
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id33c400fbe0c918b6b65d713b09009512835a4c9
Gerrit-Change-Number: 20221
Gerrit-PatchSet: 11
Gerrit-Owner: Halim Kim <rl...@gmail.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Halim Kim <rl...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Sat, 13 Apr 2024 01:20:29 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12291 impala checks hdfs ranger policy
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20221 )
Change subject: IMPALA-12291 impala checks hdfs ranger policy
......................................................................
Patch Set 5: Verified-1
Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/9519/
--
To view, visit http://gerrit.cloudera.org:8080/20221
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id33c400fbe0c918b6b65d713b09009512835a4c9
Gerrit-Change-Number: 20221
Gerrit-PatchSet: 5
Gerrit-Owner: Anonymous Coward <rl...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Fri, 21 Jul 2023 12:43:32 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12291 impala checks hdfs ranger policy
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20221 )
Change subject: IMPALA-12291 impala checks hdfs ranger policy
......................................................................
Patch Set 2:
(5 comments)
http://gerrit.cloudera.org:8080/#/c/20221/2/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
File fe/src/main/java/org/apache/impala/catalog/HdfsTable.java:
http://gerrit.cloudera.org:8080/#/c/20221/2/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java@841
PS2, Line 841: if (hdfsAuthClass != null
line has trailing whitespace
http://gerrit.cloudera.org:8080/#/c/20221/2/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java@842
PS2, Line 842: && hdfsAuthClass.equals("org.apache.ranger.authorization.hadoop.RangerHdfsAuthorizer")) {
line too long (97 > 90)
http://gerrit.cloudera.org:8080/#/c/20221/2/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java@864
PS2, Line 864: return TAccessLevel.WRITE_ONLY;
line has trailing whitespace
http://gerrit.cloudera.org:8080/#/c/20221/2/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java@866
PS2, Line 866:
line has trailing whitespace
http://gerrit.cloudera.org:8080/#/c/20221/2/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java@3181
PS2, Line 3181: public static boolean checkAccess(Path path, FileSystem fs, FsAction action)
line has trailing whitespace
--
To view, visit http://gerrit.cloudera.org:8080/20221
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id33c400fbe0c918b6b65d713b09009512835a4c9
Gerrit-Change-Number: 20221
Gerrit-PatchSet: 2
Gerrit-Owner: Anonymous Coward <rl...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Wed, 19 Jul 2023 06:48:23 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12291 impala checks hdfs ranger policy
Posted by "Anonymous Coward (Code Review)" <ge...@cloudera.org>.
Hello Quanlong Huang, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/20221
to look at the new patch set (#7).
Change subject: IMPALA-12291 impala checks hdfs ranger policy
......................................................................
IMPALA-12291 impala checks hdfs ranger policy
Change-Id: Id33c400fbe0c918b6b65d713b09009512835a4c9
---
M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
2 files changed, 25 insertions(+), 3 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/21/20221/7
--
To view, visit http://gerrit.cloudera.org:8080/20221
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Id33c400fbe0c918b6b65d713b09009512835a4c9
Gerrit-Change-Number: 20221
Gerrit-PatchSet: 7
Gerrit-Owner: Anonymous Coward <rl...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
[Impala-ASF-CR] IMPALA-12291 impala checks hdfs ranger policy
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20221 )
Change subject: IMPALA-12291 impala checks hdfs ranger policy
......................................................................
Patch Set 5:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/9519/ DRY_RUN=true
--
To view, visit http://gerrit.cloudera.org:8080/20221
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id33c400fbe0c918b6b65d713b09009512835a4c9
Gerrit-Change-Number: 20221
Gerrit-PatchSet: 5
Gerrit-Owner: Anonymous Coward <rl...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Fri, 21 Jul 2023 08:32:25 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12291 impala checks hdfs ranger policy
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20221 )
Change subject: IMPALA-12291 impala checks hdfs ranger policy
......................................................................
Patch Set 5: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/20221
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id33c400fbe0c918b6b65d713b09009512835a4c9
Gerrit-Change-Number: 20221
Gerrit-PatchSet: 5
Gerrit-Owner: Anonymous Coward <rl...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Fri, 21 Jul 2023 17:18:48 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11871: Skip permissions loading and check on HDFS if Ranger is enabled
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/20221 )
Change subject: IMPALA-11871: Skip permissions loading and check on HDFS if Ranger is enabled
......................................................................
IMPALA-11871: Skip permissions loading and check on HDFS if Ranger is enabled
Before this patch, Impala checked whether the Impala service user had
the WRITE access to the target HDFS table/partition(s) during the
analysis of the INSERT and LOAD DATA statements in the legacy catalog
mode. The access levels of the corresponding HDFS table and partitions
were computed by the catalog server solely based on the HDFS permissions
and ACLs when the table and partitions were instantiated.
After this patch, we skip loading HDFS permissions and assume the
Impala service user has the READ_WRITE permission on all the HDFS paths
associated with the target table during query analysis when Ranger is
enabled. The assumption could be removed after Impala's implementation
of FsPermissionChecker could additionally take Ranger's policies of HDFS
into consideration when performing the check.
Testing:
- Added end-to-end tests to verify Impala's behavior with respect to
the INSERT and LOAD DATA statements when Ranger is enabled in the
legacy catalog mode.
Change-Id: Id33c400fbe0c918b6b65d713b09009512835a4c9
Reviewed-on: http://gerrit.cloudera.org:8080/20221
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
M bin/rat_exclude_files.txt
M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
A testdata/data/load_data_with_catalog_v1.txt
M tests/authorization/test_ranger.py
4 files changed, 163 insertions(+), 1 deletion(-)
Approvals:
Impala Public Jenkins: Looks good to me, approved; Verified
--
To view, visit http://gerrit.cloudera.org:8080/20221
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: Id33c400fbe0c918b6b65d713b09009512835a4c9
Gerrit-Change-Number: 20221
Gerrit-PatchSet: 16
Gerrit-Owner: Halim Kim <rl...@gmail.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Halim Kim <rl...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
[Impala-ASF-CR] IMPALA-11871: Skip permissions check on HDFS files if Ranger is enabled
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20221 )
Change subject: IMPALA-11871: Skip permissions check on HDFS files if Ranger is enabled
......................................................................
Patch Set 10:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/15864/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/20221
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id33c400fbe0c918b6b65d713b09009512835a4c9
Gerrit-Change-Number: 20221
Gerrit-PatchSet: 10
Gerrit-Owner: Halim Kim <rl...@gmail.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Halim Kim <rl...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Thu, 11 Apr 2024 01:38:41 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11871: Skip permissions loading and check on HDFS if Ranger is enabled
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20221 )
Change subject: IMPALA-11871: Skip permissions loading and check on HDFS if Ranger is enabled
......................................................................
Patch Set 11:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/10539/ DRY_RUN=true
--
To view, visit http://gerrit.cloudera.org:8080/20221
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id33c400fbe0c918b6b65d713b09009512835a4c9
Gerrit-Change-Number: 20221
Gerrit-PatchSet: 11
Gerrit-Owner: Halim Kim <rl...@gmail.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Halim Kim <rl...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Sat, 13 Apr 2024 00:57:58 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11871: Skip permissions loading and check on HDFS if Ranger is enabled
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20221 )
Change subject: IMPALA-11871: Skip permissions loading and check on HDFS if Ranger is enabled
......................................................................
Patch Set 11: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/20221
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id33c400fbe0c918b6b65d713b09009512835a4c9
Gerrit-Change-Number: 20221
Gerrit-PatchSet: 11
Gerrit-Owner: Halim Kim <rl...@gmail.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Halim Kim <rl...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Sat, 13 Apr 2024 06:05:53 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11871: Skip permissions loading and check on HDFS if Ranger is enabled
Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/20221 )
Change subject: IMPALA-11871: Skip permissions loading and check on HDFS if Ranger is enabled
......................................................................
Patch Set 11:
(4 comments)
http://gerrit.cloudera.org:8080/#/c/20221/11/tests/authorization/test_ranger.py
File tests/authorization/test_ranger.py:
http://gerrit.cloudera.org:8080/#/c/20221/11/tests/authorization/test_ranger.py@89
PS11, Line 89: Impala's FsPermissionChecker
nit: "HDFS permissions" might be more clear.
http://gerrit.cloudera.org:8080/#/c/20221/11/tests/authorization/test_ranger.py@168
PS11, Line 168: we will encounter an AnalysisException
: # thrown from partitionSpec_.analyze() in LoadDataStmt#analyze() due to
Details of the codes might be stale in the future. We can just comment that LOAD DATA can't create new partitions.
http://gerrit.cloudera.org:8080/#/c/20221/11/tests/authorization/test_ranger.py@173
PS11, Line 173: user=ADMIN)
Could you add a test here showing the same LOAD DATA statement fails?
http://gerrit.cloudera.org:8080/#/c/20221/11/tests/authorization/test_ranger.py@197
PS11, Line 197: unique_table, partition_column, partition_value), user, True)
Or could you add a test after revoking the privilege? It'd be better to have some negative tests (i.e. auth failures).
--
To view, visit http://gerrit.cloudera.org:8080/20221
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id33c400fbe0c918b6b65d713b09009512835a4c9
Gerrit-Change-Number: 20221
Gerrit-PatchSet: 11
Gerrit-Owner: Halim Kim <rl...@gmail.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Halim Kim <rl...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Comment-Date: Tue, 28 May 2024 02:46:40 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11871: Skip permissions loading and check on HDFS if Ranger is enabled
Posted by "Fang-Yu Rao (Code Review)" <ge...@cloudera.org>.
Fang-Yu Rao has posted comments on this change. ( http://gerrit.cloudera.org:8080/20221 )
Change subject: IMPALA-11871: Skip permissions loading and check on HDFS if Ranger is enabled
......................................................................
Patch Set 13:
(9 comments)
Hi all, I have addressed Quanlong's and Yida's comments on patch set 11. Let me know if there are other suggestions. Thanks!
http://gerrit.cloudera.org:8080/#/c/20221/11/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
File fe/src/main/java/org/apache/impala/catalog/HdfsTable.java:
http://gerrit.cloudera.org:8080/#/c/20221/11/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java@931
PS11, Line 931: return false;
> nit. unnecessary newline
Done
http://gerrit.cloudera.org:8080/#/c/20221/11/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java@3241
PS11, Line 3241: public boolean isParquetTable() {
> nit. unnecessary newline
Done
http://gerrit.cloudera.org:8080/#/c/20221/11/tests/authorization/test_ranger.py
File tests/authorization/test_ranger.py:
http://gerrit.cloudera.org:8080/#/c/20221/11/tests/authorization/test_ranger.py@87
PS11, Line 87: Impala does not throw an AnalysisException
> Am I right in saying that if a ranger user doesn't have write permissions,
Thanks Yida!
You are correct. More precisely, if the Impala service user does not have the write permission on the destination HDFS path corresponding to the destination table or partition, with this patch the query analysis will succeed but we will encounter a runtime exception later due to the Impala service user not having the write permission. Note that usually the authorized user here refers to the requesting user that submits the INSERT query to the Impala service for execution and the requesting user is usually different from the Impala service user.
http://gerrit.cloudera.org:8080/#/c/20221/11/tests/authorization/test_ranger.py@89
PS11, Line 89: HDFS permission.
> nit: "HDFS permissions" might be more clear.
Thanks Quanlong! I will revise the code comment accordingly in the next patch.
http://gerrit.cloudera.org:8080/#/c/20221/11/tests/authorization/test_ranger.py@168
PS11, Line 168: _value)
: authz_err = "AuthorizationException: User '{0}' does not have privileges
> Details of the codes might be stale in the future. We can just comment that
Thanks Quanlong! I will revise the code comment accordingly in the next patch.
http://gerrit.cloudera.org:8080/#/c/20221/11/tests/authorization/test_ranger.py@173
PS11, Line 173: .format(unique_database), user=ADMIN)
> Could you add a test here showing the same LOAD DATA statement fails?
Sure. I will add 2 negative tests in the next patch to verify we need the following 2 privileges granted to the requesting user to execute the LOAD DATA statement. I could also add 1 negative test in the previous test test_insert_with_catalog_v1 too.
http://gerrit.cloudera.org:8080/#/c/20221/11/tests/authorization/test_ranger.py@197
PS11, Line 197:
> Or could you add a test after revoking the privilege? It'd be better to hav
Thanks Quanlong!
I will add 2 negative tests in the next patch as mentioned in my response above.
http://gerrit.cloudera.org:8080/#/c/20221/12/tests/authorization/test_ranger.py
File tests/authorization/test_ranger.py:
http://gerrit.cloudera.org:8080/#/c/20221/12/tests/authorization/test_ranger.py@203
PS12, Line 203:
> flake8: E271 multiple spaces after keyword
Done
http://gerrit.cloudera.org:8080/#/c/20221/12/tests/authorization/test_ranger.py@210
PS12, Line 210:
> flake8: E271 multiple spaces after keyword
Done
--
To view, visit http://gerrit.cloudera.org:8080/20221
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id33c400fbe0c918b6b65d713b09009512835a4c9
Gerrit-Change-Number: 20221
Gerrit-PatchSet: 13
Gerrit-Owner: Halim Kim <rl...@gmail.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Halim Kim <rl...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Comment-Date: Tue, 28 May 2024 09:45:04 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11871: Skip permissions loading and check on HDFS if Ranger is enabled
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20221 )
Change subject: IMPALA-11871: Skip permissions loading and check on HDFS if Ranger is enabled
......................................................................
Patch Set 13:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/10672/ DRY_RUN=true
--
To view, visit http://gerrit.cloudera.org:8080/20221
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id33c400fbe0c918b6b65d713b09009512835a4c9
Gerrit-Change-Number: 20221
Gerrit-PatchSet: 13
Gerrit-Owner: Halim Kim <rl...@gmail.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Halim Kim <rl...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Comment-Date: Tue, 28 May 2024 09:43:18 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11871: Skip permissions loading and check on HDFS if Ranger is enabled
Posted by "Fang-Yu Rao (Code Review)" <ge...@cloudera.org>.
Fang-Yu Rao has uploaded a new patch set (#12) to the change originally created by Halim Kim. ( http://gerrit.cloudera.org:8080/20221 )
Change subject: IMPALA-11871: Skip permissions loading and check on HDFS if Ranger is enabled
......................................................................
IMPALA-11871: Skip permissions loading and check on HDFS if Ranger is enabled
Before this patch, Impala checked whether the Impala service user had
the WRITE access to the target HDFS table/partition(s) during the
analysis of the INSERT and LOAD DATA statements in the legacy catalog
mode. The access levels of the corresponding HDFS table and partitions
were computed by the catalog server solely based on the HDFS permissions
and ACLs when the table and partitions were instantiated.
After this patch, we skip loading HDFS permissions and assume the
Impala service user has the READ_WRITE permission on all the HDFS paths
associated with the target table during query analysis when Ranger is
enabled. The assumption could be removed after Impala's implementation
of FsPermissionChecker could additionally take Ranger's policies of HDFS
into consideration when performing the check.
Testing:
- Added end-to-end tests to verify Impala's behavior with respect to
the INSERT and LOAD DATA statements when Ranger is enabled in the
legacy catalog mode.
Change-Id: Id33c400fbe0c918b6b65d713b09009512835a4c9
---
M bin/rat_exclude_files.txt
M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
A testdata/data/load_data_with_catalog_v1.txt
M tests/authorization/test_ranger.py
4 files changed, 163 insertions(+), 1 deletion(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/21/20221/12
--
To view, visit http://gerrit.cloudera.org:8080/20221
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Id33c400fbe0c918b6b65d713b09009512835a4c9
Gerrit-Change-Number: 20221
Gerrit-PatchSet: 12
Gerrit-Owner: Halim Kim <rl...@gmail.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Halim Kim <rl...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
[Impala-ASF-CR] IMPALA-12291 impala checks hdfs ranger policy
Posted by "Fang-Yu Rao (Code Review)" <ge...@cloudera.org>.
Fang-Yu Rao has posted comments on this change. ( http://gerrit.cloudera.org:8080/20221 )
Change subject: IMPALA-12291 impala checks hdfs ranger policy
......................................................................
Patch Set 9:
(2 comments)
Thanks for the reply Halim!
Other than changing how the catalog server initializes the access level of an HdfsTable during table loading, it may also make sense to disable analyzeWriteAccess() at
https://github.com/apache/impala/blob/master/fe/src/main/java/org/apache/impala/analysis/InsertStmt.java#L406-L408 depending on the value of the newly introduced startup flag.
I do not have a strong objection against adding a new startup flag in this patch. But maybe Quanlong and Aman could chime in to see if there is a better alternative since we have already got a lot of flags.
http://gerrit.cloudera.org:8080/#/c/20221/5/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
File fe/src/main/java/org/apache/impala/catalog/HdfsTable.java:
http://gerrit.cloudera.org:8080/#/c/20221/5/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java@852
PS5, Line 852: location = location.getParent();
> Your test code will be very helpful. Thank you Fang-Yu.
Thanks Halim! Not a problem.
Adding a startup flag like 'hdfs_permission_check' or 'skip_fs_permissions_check_in_analysis' to give the Impala administrator a choice is reasonable and it does not have to depend on whether Ranger is enabled.
It would be good that we mention the purpose of not performing the file system permissions check somewhere (maybe in the commit message).
This prevents users from encountering an AnalysisException during query
analysis when the target table or any partition inside is not writable
to the Impala service according to Impala's FsPermissionChecker() even
though the Impala service user is granted the READ and WRITE privileges
on the respective file system paths via the Ranger policy repository of the
corresponding storage.
If we add such a startup flag, e.g., 'skip_fs_permissions_check_in_analysis', then we could still add end-to-end tests to https://github.com/apache/impala/blob/master/tests/authorization/test_ranger.py like the following to make sure file system permissions check is still performed if the Impala administrator does not want to skip it.
@pytest.mark.execute_serially
@SkipIfFS.hdfs_acls
@CustomClusterTestSuite.with_args(
impalad_args="{0} {1}".format(CATALOGD_ARGS,
"--skip_fs_permissions_check_in_analysis=false"),
catalogd_args="{0} {1}".format(CATALOGD_ARGS,
"--skip_fs_permissions_check_in_analysis=false"))
def test_insert_with_catalog_v1_not_skip_fs_permissions_check(self, unique_name):
self._test_insert_with_catalog_v1(unique_name, False)
def _test_insert_with_catalog_v1(self, unique_name, skip_fs_permissions_check=True):
"""
Test that when Ranger is the authorization provider in the legacy catalog mode,
Impala skips or performs the file system permissions checking in query analysis
depending on the startup flag 'skip_fs_permissions_check_in_analysis'.
"""
user = getuser()
admin_client = self.create_impala_client()
unique_database = unique_name + "_db"
unique_table = unique_name + "_tbl"
table_path = "test-warehouse/{0}.db/{1}".format(unique_database, unique_table)
try:
admin_client.execute("drop database if exists {0} cascade"
.format(unique_database), user=ADMIN)
admin_client.execute("create database {0}".format(unique_database), user=ADMIN)
admin_client.execute("create table {0}.{1} (x int)"
.format(unique_database, unique_table), user=ADMIN)
admin_client.execute("grant insert on table {0}.{1} to user {2}"
.format(unique_database, unique_table, user))
# Change the owner user and group of the HDFS path corresponding to the table
# so that according to Impala's FsPermissionChecker, the table could not be
# writable to the user that loads the table. This user usually is the one
# representing the Impala service.
self.hdfs_client.chown(table_path, "another_user", "another_group")
# Invalidate the table metadata to force the catalog server to reload the HDFS
# table.
admin_client.execute("invalidate metadata {0}.{1}"
.format(unique_database, unique_table), user=ADMIN)
# Verify that Impala skips or performs the permissions checking in
# HdfsTable#getAvailableAccessLevel() depending on the startup flag
# 'skip_fs_permissions_check_in_analysis'.
query = "insert into {0}.{1} values (1)".format(unique_database, unique_table)
if skip_fs_permissions_check:
self._run_query_as_user(query, user, True)
else:
result = self._run_query_as_user(query, user, False)
err = "Unable to INSERT into target table"
assert err in str(result)
finally:
admin_client.execute("revoke insert on table {0}.{1} from user {2}"
.format(unique_database, unique_table, user))
admin_client.execute("drop database if exists {0} cascade"
.format(unique_database), user=ADMIN)
http://gerrit.cloudera.org:8080/#/c/20221/5/fe/src/main/java/org/apache/impala/service/Frontend.java
File fe/src/main/java/org/apache/impala/service/Frontend.java:
http://gerrit.cloudera.org:8080/#/c/20221/5/fe/src/main/java/org/apache/impala/service/Frontend.java@2109
PS5, Line 2109: LOG.warn("Analysis Exception query {}: {}",
: queryCtx.client_request.stmt, errorMsg);
> In retrospect, I think I found it difficult to know which query is in troub
Ack
--
To view, visit http://gerrit.cloudera.org:8080/20221
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id33c400fbe0c918b6b65d713b09009512835a4c9
Gerrit-Change-Number: 20221
Gerrit-PatchSet: 9
Gerrit-Owner: Halim Kim <rl...@gmail.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Halim Kim <rl...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Wed, 03 Apr 2024 23:31:53 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11871: Skip permissions loading and check on HDFS if Ranger is enabled
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20221 )
Change subject: IMPALA-11871: Skip permissions loading and check on HDFS if Ranger is enabled
......................................................................
Patch Set 14:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/10677/ DRY_RUN=true
--
To view, visit http://gerrit.cloudera.org:8080/20221
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id33c400fbe0c918b6b65d713b09009512835a4c9
Gerrit-Change-Number: 20221
Gerrit-PatchSet: 14
Gerrit-Owner: Halim Kim <rl...@gmail.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Halim Kim <rl...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Comment-Date: Thu, 30 May 2024 02:51:09 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11871: Skip permissions loading and check on HDFS if Ranger is enabled
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20221 )
Change subject: IMPALA-11871: Skip permissions loading and check on HDFS if Ranger is enabled
......................................................................
Patch Set 15:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/10705/ DRY_RUN=false
--
To view, visit http://gerrit.cloudera.org:8080/20221
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id33c400fbe0c918b6b65d713b09009512835a4c9
Gerrit-Change-Number: 20221
Gerrit-PatchSet: 15
Gerrit-Owner: Halim Kim <rl...@gmail.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Halim Kim <rl...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Comment-Date: Tue, 11 Jun 2024 01:22:11 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11871: Skip permissions loading and check on HDFS if Ranger is enabled
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20221 )
Change subject: IMPALA-11871: Skip permissions loading and check on HDFS if Ranger is enabled
......................................................................
Patch Set 12:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/20221/12/tests/authorization/test_ranger.py
File tests/authorization/test_ranger.py:
http://gerrit.cloudera.org:8080/#/c/20221/12/tests/authorization/test_ranger.py@203
PS12, Line 203:
flake8: E271 multiple spaces after keyword
http://gerrit.cloudera.org:8080/#/c/20221/12/tests/authorization/test_ranger.py@210
PS12, Line 210:
flake8: E271 multiple spaces after keyword
--
To view, visit http://gerrit.cloudera.org:8080/20221
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id33c400fbe0c918b6b65d713b09009512835a4c9
Gerrit-Change-Number: 20221
Gerrit-PatchSet: 12
Gerrit-Owner: Halim Kim <rl...@gmail.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Halim Kim <rl...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Comment-Date: Tue, 28 May 2024 09:37:33 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11871: Skip permissions loading and check on HDFS if Ranger is enabled
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20221 )
Change subject: IMPALA-11871: Skip permissions loading and check on HDFS if Ranger is enabled
......................................................................
Patch Set 13:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/10674/ DRY_RUN=true
--
To view, visit http://gerrit.cloudera.org:8080/20221
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id33c400fbe0c918b6b65d713b09009512835a4c9
Gerrit-Change-Number: 20221
Gerrit-PatchSet: 13
Gerrit-Owner: Halim Kim <rl...@gmail.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Halim Kim <rl...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Comment-Date: Wed, 29 May 2024 08:53:12 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11871: Skip permissions loading and check on HDFS if Ranger is enabled
Posted by "Yida Wu (Code Review)" <ge...@cloudera.org>.
Yida Wu has posted comments on this change. ( http://gerrit.cloudera.org:8080/20221 )
Change subject: IMPALA-11871: Skip permissions loading and check on HDFS if Ranger is enabled
......................................................................
Patch Set 14: Code-Review+1
--
To view, visit http://gerrit.cloudera.org:8080/20221
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id33c400fbe0c918b6b65d713b09009512835a4c9
Gerrit-Change-Number: 20221
Gerrit-PatchSet: 14
Gerrit-Owner: Halim Kim <rl...@gmail.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Halim Kim <rl...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Comment-Date: Wed, 05 Jun 2024 17:48:20 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11871: Skip permissions loading and check on HDFS if Ranger is enabled
Posted by "Fang-Yu Rao (Code Review)" <ge...@cloudera.org>.
Fang-Yu Rao has uploaded a new patch set (#11) to the change originally created by Halim Kim. ( http://gerrit.cloudera.org:8080/20221 )
Change subject: IMPALA-11871: Skip permissions loading and check on HDFS if Ranger is enabled
......................................................................
IMPALA-11871: Skip permissions loading and check on HDFS if Ranger is enabled
Before this patch, Impala checked whether the Impala service user had
the WRITE access to the target HDFS table/partition(s) during the
analysis of the INSERT and LOAD DATA statements in the legacy catalog
mode. The access levels of the corresponding HDFS table and partitions
were computed by the catalog server solely based on the HDFS permissions
and ACLs when the table and partitions were instantiated.
After this patch, we skip loading HDFS permissions and assume the
Impala service user has the READ_WRITE permission on all the HDFS paths
associated with the target table during query analysis when Ranger is
enabled. The assumption could be removed after Impala's implementation
of FsPermissionChecker could additionally take Ranger's policies of HDFS
into consideration when performing the check.
Testing:
- Added end-to-end tests to verify Impala's behavior with respect to
the INSERT and LOAD DATA statements when Ranger is enabled in the
legacy catalog mode.
Change-Id: Id33c400fbe0c918b6b65d713b09009512835a4c9
---
M bin/rat_exclude_files.txt
M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
A testdata/data/load_data_with_catalog_v1.txt
M tests/authorization/test_ranger.py
4 files changed, 147 insertions(+), 1 deletion(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/21/20221/11
--
To view, visit http://gerrit.cloudera.org:8080/20221
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Id33c400fbe0c918b6b65d713b09009512835a4c9
Gerrit-Change-Number: 20221
Gerrit-PatchSet: 11
Gerrit-Owner: Halim Kim <rl...@gmail.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Halim Kim <rl...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
[Impala-ASF-CR] IMPALA-11871: Skip permissions check on HDFS files if Ranger is enabled
Posted by "Fang-Yu Rao (Code Review)" <ge...@cloudera.org>.
Fang-Yu Rao has posted comments on this change. ( http://gerrit.cloudera.org:8080/20221 )
Change subject: IMPALA-11871: Skip permissions check on HDFS files if Ranger is enabled
......................................................................
Patch Set 10:
I slightly revised the patch set 9 and added an end-to-end test. Will try to see if we could add another test for the LOAD DATA statement.
--
To view, visit http://gerrit.cloudera.org:8080/20221
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id33c400fbe0c918b6b65d713b09009512835a4c9
Gerrit-Change-Number: 20221
Gerrit-PatchSet: 10
Gerrit-Owner: Halim Kim <rl...@gmail.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Halim Kim <rl...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Thu, 11 Apr 2024 01:16:23 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12291 impala checks hdfs ranger policy
Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/20221 )
Change subject: IMPALA-12291 impala checks hdfs ranger policy
......................................................................
Patch Set 5:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/20221/5/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
File fe/src/main/java/org/apache/impala/catalog/HdfsTable.java:
http://gerrit.cloudera.org:8080/#/c/20221/5/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java@852
PS5, Line 852: if (checkAccess(location, fs, FsAction.getFsAction("rw-")))
These introduce additional NameNode RPCs which brings significant overhead for tables with tens of thousands of partitions. We might lose the improvement we got in IMPALA-7320 where we get the permissions in batch by the listStatus APIs (more details in HdfsTable#preloadPermissionsCache()).
Another solution is always assuming read-write access on the partition dirs and skip these checks. We already do so for cloud storage tables (in assumeReadWriteAccess()). We also do so in LocalCatalog mode:
https://github.com/apache/impala/blob/ae95c43eda016e3380f395a235b373cec1337522/fe/src/main/java/org/apache/impala/catalog/local/LocalFsTable.java#L288-L298
We haven't decided whether to fix this or not in IMPALA-7539. I tend to skip checking fs level permissions and let the fs fail the write. What do you think?
http://gerrit.cloudera.org:8080/#/c/20221/5/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java@854
PS5, Line 854: FsAction.getFsAction("r--")
We can use the constants, e.g. FsAction.READ.
--
To view, visit http://gerrit.cloudera.org:8080/20221
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id33c400fbe0c918b6b65d713b09009512835a4c9
Gerrit-Change-Number: 20221
Gerrit-PatchSet: 5
Gerrit-Owner: Anonymous Coward <rl...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Fri, 21 Jul 2023 13:35:45 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12291 impala checks hdfs ranger policy
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20221 )
Change subject: IMPALA-12291 impala checks hdfs ranger policy
......................................................................
Patch Set 6:
Build Failed
https://jenkins.impala.io/job/gerrit-code-review-checks/13613/ : Initial code review checks failed. See linked job for details on the failure.
--
To view, visit http://gerrit.cloudera.org:8080/20221
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id33c400fbe0c918b6b65d713b09009512835a4c9
Gerrit-Change-Number: 20221
Gerrit-PatchSet: 6
Gerrit-Owner: Anonymous Coward <rl...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Sat, 22 Jul 2023 05:30:07 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12291 impala checks hdfs ranger policy
Posted by "Anonymous Coward (Code Review)" <ge...@cloudera.org>.
rlagk741@gmail.com has posted comments on this change. ( http://gerrit.cloudera.org:8080/20221 )
Change subject: IMPALA-12291 impala checks hdfs ranger policy
......................................................................
Patch Set 9:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/20221/5/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
File fe/src/main/java/org/apache/impala/catalog/HdfsTable.java:
http://gerrit.cloudera.org:8080/#/c/20221/5/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java@852
PS5, Line 852: location = location.getParent();
> These introduce additional NameNode RPCs which brings significant overhead
Thank you for your detailed review.
I was reluctant to add fs action code too, because it directly access HDFS and it produces more load on the Namenode as you have said.
My first idea was the reaseon Impala thoroughly checks HDFS permission in query plan stage is that if Impala query fails without permission in query execution stage, there could be a severe problem. (I don't know much about impala be)
If not so, I agree with your another solution that assumes read-write access. like below
```
if (assumeReadWriteAccess(fs) || hdfsRangerEnabled) return TAccessLevel.READ_WRITE;
```
--
To view, visit http://gerrit.cloudera.org:8080/20221
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id33c400fbe0c918b6b65d713b09009512835a4c9
Gerrit-Change-Number: 20221
Gerrit-PatchSet: 9
Gerrit-Owner: Anonymous Coward <rl...@gmail.com>
Gerrit-Reviewer: Anonymous Coward <rl...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Tue, 25 Jul 2023 00:40:42 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12291 impala checks hdfs ranger policy
Posted by "Anonymous Coward (Code Review)" <ge...@cloudera.org>.
Hello Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/20221
to look at the new patch set (#5).
Change subject: IMPALA-12291 impala checks hdfs ranger policy
......................................................................
IMPALA-12291 impala checks hdfs ranger policy
Change-Id: Id33c400fbe0c918b6b65d713b09009512835a4c9
---
M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
2 files changed, 41 insertions(+), 6 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/21/20221/5
--
To view, visit http://gerrit.cloudera.org:8080/20221
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Id33c400fbe0c918b6b65d713b09009512835a4c9
Gerrit-Change-Number: 20221
Gerrit-PatchSet: 5
Gerrit-Owner: Anonymous Coward <rl...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>