You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hive.apache.org by "ASF GitHub Bot (Jira)" <ji...@apache.org> on 2021/05/28 07:40:00 UTC
[jira] [Work logged] (HIVE-25174) HiveMetastoreAuthorizer didn't
check URI permission for AlterTableEvent
[ https://issues.apache.org/jira/browse/HIVE-25174?focusedWorklogId=603418&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-603418 ]
ASF GitHub Bot logged work on HIVE-25174:
-----------------------------------------
Author: ASF GitHub Bot
Created on: 28/May/21 07:39
Start Date: 28/May/21 07:39
Worklog Time Spent: 10m
Work Description: symious opened a new pull request #2327:
URL: https://github.com/apache/hive/pull/2327
### What changes were proposed in this pull request?
When Using Ranger on Hive MetaStore, we met an issue that users without permission to table's HDFS path succeeded in running "msck repair table TABLENAME".
This command is not authorized when we use `StorageBasedAuthorizer`, after checking the code, we found `StorageBasedAuthorizer` would check the permission of table's HDFS path, while `HiveMetastoreAuthorizer` used by Ranger won't when dealing with the event of `AlterTableEvent`.
This ticket is to add the URI permission check on AlterTableEvent for `HiveMetastoreAuthorizer`.
### Why are the changes needed?
When using `StorageBasedAuthorizer`, the command of `msck repair table` would fail if the user don't have write permission to the table's path. But when using `HiveMetastoreAuthorizer` with Ranger, the command would succeed even the user don't have write permission to the table's path.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Can be manually test with `alter table` command. Need to set Ranger as Authorizer for Hive MetaStore. Before the test, we need to ensure test user doesn't have write permission on the table's path.
* before applying patch
```
spark-sql>
> alter table yiyang_people add columns(id int);
Time taken: 2.379 seconds
21/05/28 15:33:17 INFO SparkSQLCLIDriver: Time taken: 2.379 seconds
spark-sql>
```
* after applying patch
```
spark-sql>
>
> alter table yiyang_people add columns(id int);
21/05/28 15:30:59 WARN HiveExternalCatalog: Could not alter schema of table `default`.`yiyang_people` in a Hive compatible way. Updating Hive metastore in Spark SQL specific format.
java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.sql.hive.client.Shim_v0_12.alterTable(HiveShim.scala:400)
at org.apache.spark.sql.hive.client.HiveClientImpl$$anonfun$alterTableDataSchema$1.apply$mcV$sp(HiveClientImpl.scala:536)
at org.apache.spark.sql.hive.client.HiveClientImpl$$anonfun$alterTableDataSchema$1.apply(HiveClientImpl.scala:515)
at org.apache.spark.sql.hive.client.HiveClientImpl$$anonfun$alterTableDataSchema$1.apply(HiveClientImpl.scala:515)
at org.apache.spark.sql.hive.client.HiveClientImpl$$anonfun$withHiveState$1.apply(HiveClientImpl.scala:277)
at org.apache.spark.sql.hive.client.HiveClientImpl.liftedTree1$1(HiveClientImpl.scala:215)
at org.apache.spark.sql.hive.client.HiveClientImpl.retryLocked(HiveClientImpl.scala:214)
at org.apache.spark.sql.hive.client.HiveClientImpl.withHiveState(HiveClientImpl.scala:260)
at org.apache.spark.sql.hive.client.HiveClientImpl.alterTableDataSchema(HiveClientImpl.scala:515)
at org.apache.spark.sql.hive.HiveExternalCatalog$$anonfun$alterTableDataSchema$1.apply$mcV$sp(HiveExternalCatalog.scala:664)
at org.apache.spark.sql.hive.HiveExternalCatalog$$anonfun$alterTableDataSchema$1.apply(HiveExternalCatalog.scala:650)
at org.apache.spark.sql.hive.HiveExternalCatalog$$anonfun$alterTableDataSchema$1.apply(HiveExternalCatalog.scala:650)
at org.apache.spark.sql.hive.HiveExternalCatalog.withClient(HiveExternalCatalog.scala:97)
at org.apache.spark.sql.hive.HiveExternalCatalog.alterTableDataSchema(HiveExternalCatalog.scala:650)
at org.apache.spark.sql.catalyst.catalog.ExternalCatalogWithListener.alterTableDataSchema(ExternalCatalogWithListener.scala:124)
at org.apache.spark.sql.catalyst.catalog.SessionCatalog.alterTableDataSchema(SessionCatalog.scala:391)
at org.apache.spark.sql.execution.command.AlterTableAddColumnsCommand.run(tables.scala:203)
at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:70)
at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:68)
at org.apache.spark.sql.execution.command.ExecutedCommandExec.executeCollect(commands.scala:79)
at org.apache.spark.sql.Dataset$$anonfun$6.apply(Dataset.scala:194)
at org.apache.spark.sql.Dataset$$anonfun$6.apply(Dataset.scala:194)
at org.apache.spark.sql.Dataset$$anonfun$52.apply(Dataset.scala:3370)
at org.apache.spark.sql.execution.SQLExecution$$anonfun$withNewExecutionId$1.apply(SQLExecution.scala:80)
at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:127)
at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:75)
at org.apache.spark.sql.Dataset.org$apache$spark$sql$Dataset$$withAction(Dataset.scala:3369)
at org.apache.spark.sql.Dataset.<init>(Dataset.scala:194)
at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:79)
at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:643)
at org.apache.spark.sql.SQLContext.sql(SQLContext.scala:694)
at org.apache.spark.sql.hive.thriftserver.SparkSQLDriver.run(SparkSQLDriver.scala:62)
at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.processCmd(SparkSQLCLIDriver.scala:371)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:376)
at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver$.main(SparkSQLCLIDriver.scala:274)
at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.main(SparkSQLCLIDriver.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:845)
at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:161)
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:184)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:920)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:929)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Unable to alter table. Permission denied: user [yiyang.zhou] does not have [READ_WRITE] privilege on [hdfs://NN_IP:8020/user/yiyang.zhou/people]
at org.apache.hadoop.hive.ql.metadata.Hive.alterTable(Hive.java:499)
at org.apache.hadoop.hive.ql.metadata.Hive.alterTable(Hive.java:485)
... 52 more
Caused by: MetaException(message:Permission denied: user [yiyang.zhou] does not have [READ_WRITE] privilege on [hdfs://NN_IP:8020/user/yiyang.zhou/people])
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$alter_table_with_cascade_result$alter_table_with_cascade_resultStandardScheme.read(ThriftHiveMetastore.java:41136)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$alter_table_with_cascade_result$alter_table_with_cascade_resultStandardScheme.read(ThriftHiveMetastore.java:41113)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$alter_table_with_cascade_result.read(ThriftHiveMetastore.java:41055)
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:86)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_alter_table_with_cascade(ThriftHiveMetastore.java:1382)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.alter_table_with_cascade(ThriftHiveMetastore.java:1366)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.alter_table(HiveMetaStoreClient.java:341)
at org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.alter_table(SessionHiveMetaStoreClient.java:251)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:156)
at com.sun.proxy.$Proxy5.alter_table(Unknown Source)
at org.apache.hadoop.hive.ql.metadata.Hive.alterTable(Hive.java:497)
... 53 more
Error in query: org.apache.hadoop.hive.ql.metadata.HiveException: Unable to alter table. Permission denied: user [yiyang.zhou] does not have [READ_WRITE] privilege on [hdfs://NN_IP:8020/user/yiyang.zhou/people];
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Issue Time Tracking
-------------------
Worklog Id: (was: 603418)
Remaining Estimate: 0h
Time Spent: 10m
> HiveMetastoreAuthorizer didn't check URI permission for AlterTableEvent
> -----------------------------------------------------------------------
>
> Key: HIVE-25174
> URL: https://issues.apache.org/jira/browse/HIVE-25174
> Project: Hive
> Issue Type: Improvement
> Reporter: Janus Chow
> Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> When Using Ranger on Hive MetaStore, we met an issue that users without permission to table's HDFS path succeeded in running "msck repair table TABLENAME".
> This command is not authorized when we use `StorageBasedAuthorizer`, after checking the code, we found `StorageBasedAuthorizer` would check the permission of table's HDFS path, while `HiveMetastoreAuthorizer` used by Ranger won't when dealing with the event of `AlterTableEvent`.
> This ticket is to add the URI permission check on AlterTableEvent for `HiveMetastoreAuthorizer`.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)