You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@drill.apache.org by "Arina Ielchiieva (Jira)" <ji...@apache.org> on 2019/11/04 12:49:00 UTC

[jira] [Updated] (DRILL-6923) Show schemas uses default(user defined) schema first for resolving table from information_schema

     [ https://issues.apache.org/jira/browse/DRILL-6923?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Arina Ielchiieva updated DRILL-6923:
------------------------------------
    Fix Version/s:     (was: 1.17.0)

> Show schemas uses default(user defined) schema first for resolving table from information_schema
> ------------------------------------------------------------------------------------------------
>
>                 Key: DRILL-6923
>                 URL: https://issues.apache.org/jira/browse/DRILL-6923
>             Project: Apache Drill
>          Issue Type: Bug
>          Components: Storage - Hive
>    Affects Versions: 1.14.0
>            Reporter: Igor Guzenko
>            Assignee: Igor Guzenko
>            Priority: Minor
>
> Show tables tries to find table `information_schema`.`schemata` in default (user defined) schema, and after failed attempt it resolves table successfully against root schema. Please check description below for details explained using example with hive plugin. 
> *Abstract* 
> When Drill used with enabled Hive SQL Standard authorization, execution of queries like,
> {code:sql}
> USE hive.db_general;
> SHOW SCHEMAS LIKE 'hive.%'; {code}
> results in error DrillRuntimeException: Failed to use the Hive authorization components: Error getting object from metastore for Object [type=TABLE_OR_VIEW, name=db_general.information_schema] . 
> *Details* 
> Consider showSchemas() test similar to one defined in TestSqlStdBasedAuthorization : 
> {code:java}
> @Test
> public void showSchemas() throws Exception {
>   test("USE " + hivePluginName + "." + db_general);
>   testBuilder()
>       .sqlQuery("SHOW SCHEMAS LIKE 'hive.%'")
>       .unOrdered()
>       .baselineColumns("SCHEMA_NAME")
>       .baselineValues("hive.db_general")
>       .baselineValues("hive.default")
>       .go();
> }
> {code}
> Currently execution of such test will produce following stacktrace: 
> {code:none}
> Caused by: org.apache.drill.common.exceptions.DrillRuntimeException: Failed to use the Hive authorization components: Error getting object from metastore for Object [type=TABLE_OR_VIEW, name=db_general.information_schema]
> at org.apache.drill.exec.store.hive.HiveAuthorizationHelper.authorize(HiveAuthorizationHelper.java:149)
> at org.apache.drill.exec.store.hive.HiveAuthorizationHelper.authorizeReadTable(HiveAuthorizationHelper.java:134)
> at org.apache.drill.exec.store.hive.DrillHiveMetaStoreClient$HiveClientWithAuthzWithCaching.getHiveReadEntry(DrillHiveMetaStoreClient.java:450)
> at org.apache.drill.exec.store.hive.schema.HiveSchemaFactory$HiveSchema.getSelectionBaseOnName(HiveSchemaFactory.java:233)
> at org.apache.drill.exec.store.hive.schema.HiveSchemaFactory$HiveSchema.getDrillTable(HiveSchemaFactory.java:214)
> at org.apache.drill.exec.store.hive.schema.HiveDatabaseSchema.getTable(HiveDatabaseSchema.java:63)
> at org.apache.calcite.jdbc.SimpleCalciteSchema.getImplicitTable(SimpleCalciteSchema.java:83)
> at org.apache.calcite.jdbc.CalciteSchema.getTable(CalciteSchema.java:288)
> at org.apache.calcite.sql.validate.EmptyScope.resolve_(EmptyScope.java:143)
> at org.apache.calcite.sql.validate.EmptyScope.resolveTable(EmptyScope.java:99)
> at org.apache.calcite.sql.validate.DelegatingScope.resolveTable(DelegatingScope.java:203)
> at org.apache.calcite.sql.validate.IdentifierNamespace.resolveImpl(IdentifierNamespace.java:105)
> at org.apache.calcite.sql.validate.IdentifierNamespace.validateImpl(IdentifierNamespace.java:177)
> at org.apache.calcite.sql.validate.AbstractNamespace.validate(AbstractNamespace.java:84)
> at org.apache.calcite.sql.validate.SqlValidatorImpl.validateNamespace(SqlValidatorImpl.java:967)
> at org.apache.calcite.sql.validate.SqlValidatorImpl.validateQuery(SqlValidatorImpl.java:943)
> at org.apache.calcite.sql.validate.SqlValidatorImpl.validateFrom(SqlValidatorImpl.java:3032)
> at org.apache.drill.exec.planner.sql.SqlConverter$DrillValidator.validateFrom(SqlConverter.java:274)
> at org.apache.calcite.sql.validate.SqlValidatorImpl.validateFrom(SqlValidatorImpl.java:3014)
> at org.apache.drill.exec.planner.sql.SqlConverter$DrillValidator.validateFrom(SqlConverter.java:274)
> at org.apache.calcite.sql.validate.SqlValidatorImpl.validateSelect(SqlValidatorImpl.java:3284)
> at org.apache.calcite.sql.validate.SelectNamespace.validateImpl(SelectNamespace.java:60)
> at org.apache.calcite.sql.validate.AbstractNamespace.validate(AbstractNamespace.java:84)
> at org.apache.calcite.sql.validate.SqlValidatorImpl.validateNamespace(SqlValidatorImpl.java:967)
> at org.apache.calcite.sql.validate.SqlValidatorImpl.validateQuery(SqlValidatorImpl.java:943)
> at org.apache.calcite.sql.SqlSelect.validate(SqlSelect.java:225)
> at org.apache.calcite.sql.validate.SqlValidatorImpl.validateScopedExpression(SqlValidatorImpl.java:918)
> at org.apache.calcite.sql.validate.SqlValidatorImpl.validate(SqlValidatorImpl.java:628)
> at org.apache.drill.exec.planner.sql.SqlConverter.validate(SqlConverter.java:192)
> at org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.validateNode(DefaultSqlHandler.java:664)
> at org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.validateAndConvert(DefaultSqlHandler.java:200)
> at org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.getPlan(DefaultSqlHandler.java:173)
> at org.apache.drill.exec.planner.sql.DrillSqlWorker.getQueryPlan(DrillSqlWorker.java:155)
> at org.apache.drill.exec.planner.sql.DrillSqlWorker.getPlan(DrillSqlWorker.java:90)
> at org.apache.drill.exec.work.foreman.Foreman.runSQL(Foreman.java:584)
> at org.apache.drill.exec.work.foreman.Foreman.run(Foreman.java:272)
> at .......(:0)
> Caused by: org.apache.hadoop.hive.ql.security.authorization.plugin.HiveAuthzPluginException: Error getting object from metastore for Object [type=TABLE_OR_VIEW, name=db_general.information_schema]
> at org.apache.hadoop.hive.ql.security.authorization.plugin.sqlstd.SQLAuthorizationUtils.throwGetObjErr(SQLAuthorizationUtils.java:310)
> at org.apache.hadoop.hive.ql.security.authorization.plugin.sqlstd.SQLAuthorizationUtils.isOwner(SQLAuthorizationUtils.java:272)
> at org.apache.hadoop.hive.ql.security.authorization.plugin.sqlstd.SQLAuthorizationUtils.getPrivilegesFromMetaStore(SQLAuthorizationUtils.java:212)
> at org.apache.hadoop.hive.ql.security.authorization.plugin.sqlstd.SQLStdHiveAuthorizationValidator.checkPrivileges(SQLStdHiveAuthorizationValidator.java:131)
> at org.apache.hadoop.hive.ql.security.authorization.plugin.sqlstd.SQLStdHiveAuthorizationValidator.checkPrivileges(SQLStdHiveAuthorizationValidator.java:83)
> at org.apache.hadoop.hive.ql.security.authorization.plugin.HiveAuthorizerImpl.checkPrivileges(HiveAuthorizerImpl.java:87)
> at org.apache.drill.exec.store.hive.HiveAuthorizationHelper.authorize(HiveAuthorizationHelper.java:145)
> ... 36 more
> Caused by: NoSuchObjectException(message:db_general.information_schema table not found)
> at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_table_req_result$get_table_req_resultStandardScheme.read(ThriftHiveMetastore.java:55064)
> at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_table_req_result$get_table_req_resultStandardScheme.read(ThriftHiveMetastore.java:55032)
> at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_table_req_result.read(ThriftHiveMetastore.java:54963)
> at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:86)
> at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_table_req(ThriftHiveMetastore.java:1563)
> at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_table_req(ThriftHiveMetastore.java:1550)
> at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getTable(HiveMetaStoreClient.java:1344)
> at org.apache.hadoop.hive.ql.security.authorization.plugin.sqlstd.SQLAuthorizationUtils.isOwner(SQLAuthorizationUtils.java:269)
> ... 41 more
> {code}
> After analyzing of the stack trace you may notice that Drill tried to find table with name `_information_schema_` in _*hive.db_general*_ schema. This `information_schema` appeared after call to _*ShowSchemasHandler.rewrite(sqlNode)*_ method , where our original query
> {code:sql}
> SHOW SCHEMAS LIKE 'hive.%' {code}
>  was rewritten to 
> {code:sql}
> SELECT `SCHEMA_NAME`
> FROM `information_schema`.`SCHEMATA`
> WHERE LOWER(`SCHEMA_NAME`) LIKE 'hive.%'
> {code}
>  After rewriting Drill initiates validation process mostly performed by Calcite. After some chain of validation method calls, performed validation of from section which results into call to Calcite's EmptyScope.resolveTable(...) method. This method tries to resolve table using following order: Look in the default schema, then default catalog, then root schema. Since, before execution of SHOW SCHEMAS default schema was set to _*hive.db_general*_ , Calcite assumes that `information_schema` is table name and tries to find it using  _*hive.db_general*_ schema instance. Usually this search mechanism works fine because this first attempt to find table in default schema results into *null*, after that Calcite tries to resolve path ["information_schema", "SCHEMATA"]  against root schema, which is like an empty string, schema name becomes *information_schema* and table name is *schemata*. Then as expected schema and table resolved successfully. 
> Please notice, that this issue isn't specific to only hive storage plugin, there is high probability that under similar conditions it may be reproduced with other storage plugins. 
> As a possible solution, we need to change table resolution to always search against root schema and ignore default schema selected by user when search performed for information_schema table. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)