You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by "Xiao Yu (Jira)" <ji...@apache.org> on 2021/08/25 17:53:00 UTC
[jira] [Created] (HIVE-25481) Schema Validation with MySQL Throws
Exceptions
Xiao Yu created HIVE-25481:
------------------------------
Summary: Schema Validation with MySQL Throws Exceptions
Key: HIVE-25481
URL: https://issues.apache.org/jira/browse/HIVE-25481
Project: Hive
Issue Type: Bug
Components: Database/Schema
Affects Versions: 2.3.6
Environment: * Bigtop 1.5.0 distributed Hadoop (Hive 2.3.6)
* MariaDB (but also likely will happen with MySQL as well)
* MySQL JDBC connector (8.0.26)
Reporter: Xiao Yu
When running with a MySQL/MariaDB backend setting {{datanucleus.schema.validateTables}} to true causes warnings like the following to be thrown:
{code}
2021-08-24 17:56:54,765 [WARN ] DataNucleus.Query: Query for candidates of org.apache.hadoop.hive.metastore.model.MPartition and subclasses resulted in no possible candidates
org.datanucleus.store.rdbms.exceptions.NotATableException: Table "`PARTITIONS`" is of wrong type SYSTEM VIEW. Should be a table
at org.datanucleus.store.rdbms.table.TableImpl.validate(TableImpl.java:161) ~[datanucleus-rdbms-4.1.19.jar:?]
at org.datanucleus.store.rdbms.RDBMSStoreManager$ClassAdder.performTablesValidation(RDBMSStoreManager.java:3407) ~[datanucleus-rdbms-4.1.19.jar:?]
at org.datanucleus.store.rdbms.RDBMSStoreManager$ClassAdder.run(RDBMSStoreManager.java:2896) ~[datanucleus-rdbms-4.1.19.jar:?]
at org.datanucleus.store.rdbms.AbstractSchemaTransaction.execute(AbstractSchemaTransaction.java:119) ~[datanucleus-rdbms-4.1.19.jar:?]
at org.datanucleus.store.rdbms.RDBMSStoreManager.manageClasses(RDBMSStoreManager.java:1627) ~[datanucleus-rdbms-4.1.19.jar:?]
at org.datanucleus.store.rdbms.RDBMSStoreManager.getDatastoreClass(RDBMSStoreManager.java:672) ~[datanucleus-rdbms-4.1.19.jar:?]
at org.datanucleus.store.rdbms.query.RDBMSQueryUtils.getStatementForCandidates(RDBMSQueryUtils.java:425) ~[datanucleus-rdbms-4.1.19.jar:?]
at org.datanucleus.store.rdbms.query.JDOQLQuery.compileQueryFull(JDOQLQuery.java:865) [datanucleus-rdbms-4.1.19.jar:?]
at org.datanucleus.store.rdbms.query.JDOQLQuery.compileInternal(JDOQLQuery.java:347) [datanucleus-rdbms-4.1.19.jar:?]
at org.datanucleus.store.query.Query.executeQuery(Query.java:1816) [datanucleus-core-4.1.17.jar:?]
at org.datanucleus.store.query.Query.executeWithArray(Query.java:1744) [datanucleus-core-4.1.17.jar:?]
at org.datanucleus.api.jdo.JDOQuery.executeInternal(JDOQuery.java:368) [datanucleus-api-jdo-4.2.4.jar:?]
at org.datanucleus.api.jdo.JDOQuery.execute(JDOQuery.java:241) [datanucleus-api-jdo-4.2.4.jar:?]
at org.apache.hadoop.hive.metastore.ObjectStore.getPartitionNamesNoTxn(ObjectStore.java:2498) [hive-exec-2.3.6.jar:2.3.6]
at org.apache.hadoop.hive.metastore.ObjectStore.listPartitionNames(ObjectStore.java:2231) [hive-exec-2.3.6.jar:2.3.6]
at jdk.internal.reflect.GeneratedMethodAccessor122.invoke(Unknown Source) ~[?:?]
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
at java.base/java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
at org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:101) [hive-exec-2.3.6.jar:2.3.6]
at com.sun.proxy.$Proxy41.listPartitionNames(Unknown Source) [?:?]
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_partition_names(HiveMetaStore.java:3708) [hive-exec-2.3.6.jar:2.3.6]
at jdk.internal.reflect.GeneratedMethodAccessor121.invoke(Unknown Source) ~[?:?]
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
at java.base/java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:148) [hive-exec-2.3.6.jar:2.3.6]
at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:107) [hive-exec-2.3.6.jar:2.3.6]
at com.sun.proxy.$Proxy42.get_partition_names(Unknown Source) [?:?]
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$get_partition_names.getResult(ThriftHiveMetastore.java:12287) [hive-exec-2.3.6.jar:2.3.6]
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$get_partition_names.getResult(ThriftHiveMetastore.java:12271) [hive-exec-2.3.6.jar:2.3.6]
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) [hive-exec-2.3.6.jar:2.3.6]
at org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:110) [hive-exec-2.3.6.jar:2.3.6]
at org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:106) [hive-exec-2.3.6.jar:2.3.6]
at java.base/java.security.AccessController.doPrivileged(Native Method) ~[?:?]
at java.base/javax.security.auth.Subject.doAs(Subject.java:423) [?:?]
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1926) [hadoop-common-2.10.1.jar:?]
at org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:118) [hive-exec-2.3.6.jar:2.3.6]
at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286) [hive-exec-2.3.6.jar:2.3.6]
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.base/java.lang.Thread.run(Thread.java:829) [?:?]
{code}
It appears this is because the schema validation is mistaking the internal {{INFORMATION_SCHEMA.PARTITIONS}} table with the Hive created {{[$HIVE_DB].PARTITIONS}} table. From the MySQL side it appears this is happening because the validation routine is iterating through all databases that it has access to not just using the specific DB configured via {{javax.jdo.option.ConnectionURL}}. We see the following set of queries issued when this error occurs:
{code}
SET autocommit=0
SELECT @@session.tx_isolation
SHOW DATABASES
SHOW FULL TABLES FROM `heartbeat` LIKE PARTITIONS
SHOW FULL TABLES FROM `information_schema` LIKE PARTITIONS
SHOW FULL TABLES FROM `wyeast-hive` LIKE PARTITIONS
SHOW DATABASES
SHOW FULL TABLES FROM `heartbeat` LIKE PARTITIONS
SHOW FULL TABLES FROM `information_schema` LIKE PARTITIONS
SHOW FULL TABLES FROM `wyeast-hive` LIKE PARTITIONS
rollback
commit
rollback
SET autocommit=1
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)