You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues-all@impala.apache.org by "ASF subversion and git services (Jira)" <ji...@apache.org> on 2024/06/11 06:53:00 UTC
[jira] [Commented] (IMPALA-11871) INSERT statement does not respect Ranger policies for HDFS
[ https://issues.apache.org/jira/browse/IMPALA-11871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17853923#comment-17853923 ]
ASF subversion and git services commented on IMPALA-11871:
----------------------------------------------------------
Commit f7e629935b77f412bf74aeebd704af88f03de351 in impala's branch refs/heads/master from halim.kim
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=f7e629935 ]
IMPALA-11871: Skip permissions loading and check on HDFS if Ranger is enabled
Before this patch, Impala checked whether the Impala service user had
the WRITE access to the target HDFS table/partition(s) during the
analysis of the INSERT and LOAD DATA statements in the legacy catalog
mode. The access levels of the corresponding HDFS table and partitions
were computed by the catalog server solely based on the HDFS permissions
and ACLs when the table and partitions were instantiated.
After this patch, we skip loading HDFS permissions and assume the
Impala service user has the READ_WRITE permission on all the HDFS paths
associated with the target table during query analysis when Ranger is
enabled. The assumption could be removed after Impala's implementation
of FsPermissionChecker could additionally take Ranger's policies of HDFS
into consideration when performing the check.
Testing:
- Added end-to-end tests to verify Impala's behavior with respect to
the INSERT and LOAD DATA statements when Ranger is enabled in the
legacy catalog mode.
Change-Id: Id33c400fbe0c918b6b65d713b09009512835a4c9
Reviewed-on: http://gerrit.cloudera.org:8080/20221
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
> INSERT statement does not respect Ranger policies for HDFS
> ----------------------------------------------------------
>
> Key: IMPALA-11871
> URL: https://issues.apache.org/jira/browse/IMPALA-11871
> Project: IMPALA
> Issue Type: Bug
> Components: Frontend
> Reporter: Fang-Yu Rao
> Assignee: Fang-Yu Rao
> Priority: Major
>
> In a cluster with Ranger auth (and with legacy catalog mode), even if you provide RWX to cm_hdfs -> all-path for the user impala, inserting into a table whose HDFS POSIX permissions happen to exclude impala access will result in an
> {noformat}
> "AnalysisException: Unable to INSERT into target table (default.t1) because Impala does not have WRITE access to HDFS location: hdfs://nightly-71x-vx-2.nightly-71x-vx.root.hwx.site:8020/warehouse/tablespace/external/hive/t1"{noformat}
>
> {noformat}
> [root@nightly-71x-vx-3 ~]# hdfs dfs -getfacl /warehouse/tablespace/external/hive/t1
> file: /warehouse/tablespace/external/hive/t1
> owner: hive
> group: supergroup
> user::rwx
> user:impala:rwx #effective:r-x
> group::rwx #effective:r-x
> mask::r-x
> other::---
> default:user::rwx
> default:user:impala:rwx
> default:group::rwx
> default:mask::rwx
> default:other::--- {noformat}
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> ANALYSIS
> Stack trace from a version of Cloudera's distribution of Impala (impalad version 3.4.0-SNAPSHOT RELEASE (build {*}db20b59a093c17ea4699117155d58fe874f7d68f{*})):
> {noformat}
> at org.apache.impala.catalog.FeFsTable$Utils.checkWriteAccess(FeFsTable.java:585)
> at org.apache.impala.analysis.InsertStmt.analyzeWriteAccess(InsertStmt.java:545)
> at org.apache.impala.analysis.InsertStmt.analyze(InsertStmt.java:391)
> at org.apache.impala.analysis.AnalysisContext.analyze(AnalysisContext.java:463)
> at org.apache.impala.analysis.AnalysisContext.analyzeAndAuthorize(AnalysisContext.java:426)
> at org.apache.impala.service.Frontend.doCreateExecRequest(Frontend.java:1570)
> at org.apache.impala.service.Frontend.getTExecRequest(Frontend.java:1536)
> at org.apache.impala.service.Frontend.createExecRequest(Frontend.java:1506)
> at org.apache.impala.service.JniFrontend.createExecRequest(JniFrontend.java:155){noformat}
> The exception occurs at analysis time, so I tested and succeeded in writing directly into the said directory.
> {noformat}
> [root@nightly-71x-vx-3 ~]# hdfs dfs -touchz /warehouse/tablespace/external/hive/t1/test
> [root@nightly-71x-vx-3 ~]# hdfs dfs -ls /warehouse/tablespace/external/hive/t1/
> Found 8 items
> rw-rw---+ 3 hive supergroup 417 2023-01-27 17:37 /warehouse/tablespace/external/hive/t1/000000_0
> rw-rw---+ 3 hive supergroup 417 2023-01-27 17:44 /warehouse/tablespace/external/hive/t1/000000_0_copy_1
> rw-rw---+ 3 hive supergroup 417 2023-01-27 17:49 /warehouse/tablespace/external/hive/t1/000000_0_copy_2
> rw-rw---+ 3 hive supergroup 417 2023-01-27 17:53 /warehouse/tablespace/external/hive/t1/000000_0_copy_3
> rw-rw---+ 3 impala hive 355 2023-01-27 17:17 /warehouse/tablespace/external/hive/t1/4c4477c12c51ad96-3126b52d00000000_2029811630_data.0.parq
> rw-rw---+ 3 impala hive 355 2023-01-27 17:39 /warehouse/tablespace/external/hive/t1/9945b25bb37d1ff2-473c147800000000_574471191_data.0.parq
> drwxrwx---+ - impala hive 0 2023-01-27 17:39 /warehouse/tablespace/external/hive/t1/_impala_insert_staging
> rw-rw---+ 3 impala supergroup 0 2023-01-27 18:01 /warehouse/tablespace/external/hive/t1/test{noformat}
> Reviewing the code[1], I traced the {{TAccessLevel}} to the catalogd. And if I add user impala to group supergroup on the catalogd host, this query will succeed past the authorization.
> Additionally, this query does not trip up during analysis when catalog v2 is enabled because the method {{getFirstLocationWithoutWriteAccess()}} is not implemented there yet and always returns null[2].
> [1] [https://github.com/apache/impala/blob/master/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java#L494-L504]
> [2] [https://github.com/apache/impala/blob/master/fe/src/main/java/org/apache/impala/catalog/local/LocalFsTable.java#L295-L298]
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> Ideally, when Ranger authorization is in place, we should:
> 1) Not check access level during analysis
> 2) Incorporate Ranger ACLs during analysis
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org