You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by "Eli Acherkan (JIRA)" <ji...@apache.org> on 2014/04/08 16:55:15 UTC
[jira] [Commented] (HIVE-6113) Unable to instantiate
org.apache.hadoop.hive.metastore.HiveMetaStoreClient
[ https://issues.apache.org/jira/browse/HIVE-6113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13963055#comment-13963055 ]
Eli Acherkan commented on HIVE-6113:
------------------------------------
The exact same issue reproduces here. Hive 0.12 on MapR 3.1.0 with MySQL metastore. The exception appears when there are several processes working with Hive concurrently.
>From our analysis the problem seems related to the one described here: http://mail-archives.apache.org/mod_mbox/hive-user/201107.mbox/%3C4F6B25AFFFCAFE44B6259A412D5F9B1033183876@ExchMBX104.netflix.com%3E
h5. Analysis:
At certain times, Hive's DataNucleus decides to create and then drop tables called "DELETEME"+timestamp in the metastore schema on MySQL (see [ProbleTable|http://sourceforge.net/p/datanucleus/code/HEAD/tree/platform/store.rdbms/tags/datanucleus-rdbms-3.2.2/src/java/org/datanucleus/store/rdbms/table/ProbeTable.java]).
During other flows, DataNucleus queries MySQL for the list of all the columns of all the tables (see [RDBMSSchemaHandler.refreshTableData|http://sourceforge.net/p/datanucleus/code/HEAD/tree/platform/store.rdbms/tags/datanucleus-rdbms-3.2.2/src/java/org/datanucleus/store/rdbms/schema/RDBMSSchemaHandler.java#l872]). MySQL's JDBC driver implements the DatabaseMetaData.getColumns method by querying the DB for a list of all the tables, and then iterating over that list and querying for each table's columns (see [com.mysql.jdbc.DatabaseMetaData|http://bazaar.launchpad.net/~mysql/connectorj/5.1/view/head:/src/com/mysql/jdbc/DatabaseMetaData.java#L2581]). If a table is deleted from the DB during this operation, DatabaseMetaData.getColumns will throw an exception.
This exception is interpreted by Hive to mean that the "default" Hive database doesn't exist. Hive tries to create it, inserting a row into the metastore.DBS table in MySQL, which triggers the "Duplicate entry 'default' for key 'UNIQUE_DATABASE'" exception.
I'm not completely clear about the conditions for a) DataNucleus creating and dropping a "DELETEME" table, and b) DataNucleus calling DatabaseMetaData.getColumns, so unfortunately I can't yet provide a clear test case. But in our lab environment under load we were able to reproduce the exception once every few minutes.
h5. Workaround:
As suggested by the link above, setting the *datanucleus.fixedDatastore* property to *true* (e.g. in hive-site.xml or elsewhere) seems to solve the problem. However, it means that the metastore schema is no longer automatically created on-demand, and requires using Hive's schematool command to manually create the metastore schema.
> Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient
> --------------------------------------------------------------------------
>
> Key: HIVE-6113
> URL: https://issues.apache.org/jira/browse/HIVE-6113
> Project: Hive
> Issue Type: Bug
> Components: Database/Schema
> Affects Versions: 0.12.0
> Environment: hadoop-0.20.2-cdh3u3,hive-0.12.0
> Reporter: William Stone
> Priority: Critical
> Labels: HiveMetaStoreClient, metastore, unable_instantiate
>
> When I exccute SQL "use fdm; desc formatted fdm.tableName;" in python, throw Error as followed.
> but when I tryit again , It will success.
> 2013-12-25 03:01:32,290 ERROR exec.DDLTask (DDLTask.java:execute(435)) - org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient
> at org.apache.hadoop.hive.ql.metadata.Hive.getDatabase(Hive.java:1143)
> at org.apache.hadoop.hive.ql.metadata.Hive.databaseExists(Hive.java:1128)
> at org.apache.hadoop.hive.ql.exec.DDLTask.switchDatabase(DDLTask.java:3479)
> at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:237)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:151)
> at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:65)
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1414)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1192)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1020)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:888)
> at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:260)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:217)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:507)
> at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:875)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:769)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:708)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:197)
> Caused by: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient
> at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1217)
> at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:62)
> at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:72)
> at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:2372)
> at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:2383)
> at org.apache.hadoop.hive.ql.metadata.Hive.getDatabase(Hive.java:1139)
> ... 20 more
> Caused by: java.lang.reflect.InvocationTargetException
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
> at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
> at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1210)
> ... 25 more
> Caused by: javax.jdo.JDODataStoreException: Exception thrown flushing changes to datastore
> NestedThrowables:
> java.sql.BatchUpdateException: Duplicate entry 'default' for key 'UNIQUE_DATABASE'
> at org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:451)
> at org.datanucleus.api.jdo.JDOTransaction.commit(JDOTransaction.java:165)
> at org.apache.hadoop.hive.metastore.ObjectStore.commitTransaction(ObjectStore.java:358)
> at org.apache.hadoop.hive.metastore.ObjectStore.createDatabase(ObjectStore.java:404)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.hive.metastore.RetryingRawStore.invoke(RetryingRawStore.java:124)
> at $Proxy9.createDatabase(Unknown Source)
> at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB_core(HiveMetaStore.java:422)
> at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB(HiveMetaStore.java:441)
> at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:326)
> at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.<init>(HiveMetaStore.java:286)
> at org.apache.hadoop.hive.metastore.RetryingHMSHandler.<init>(RetryingHMSHandler.java:54)
> at org.apache.hadoop.hive.metastore.RetryingHMSHandler.getProxy(RetryingHMSHandler.java:59)
> at org.apache.hadoop.hive.metastore.HiveMetaStore.newHMSHandler(HiveMetaStore.java:4060)
> at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:121)
> ... 30 more
> Caused by: java.sql.BatchUpdateException: Duplicate entry 'default' for key 'UNIQUE_DATABASE'
> at com.mysql.jdbc.PreparedStatement.executeBatchSerially(PreparedStatement.java:2028)
> at com.mysql.jdbc.PreparedStatement.executeBatch(PreparedStatement.java:1451)
> at com.jolbox.bonecp.StatementHandle.executeBatch(StatementHandle.java:469)
> at org.datanucleus.store.rdbms.ParamLoggingPreparedStatement.executeBatch(ParamLoggingPreparedStatement.java:372)
> at org.datanucleus.store.rdbms.SQLController.processConnectionStatement(SQLController.java:628)
> at org.datanucleus.store.rdbms.SQLController.processStatementsForConnection(SQLController.java:596)
> at org.datanucleus.store.rdbms.SQLController$1.transactionFlushed(SQLController.java:683)
> at org.datanucleus.store.connection.AbstractManagedConnection.transactionFlushed(AbstractManagedConnection.java:86)
> at org.datanucleus.store.connection.ConnectionManagerImpl$2.transactionFlushed(ConnectionManagerImpl.java:454)
> at org.datanucleus.TransactionImpl.flush(TransactionImpl.java:199)
> at org.datanucleus.TransactionImpl.commit(TransactionImpl.java:263)
> at org.datanucleus.api.jdo.JDOTransaction.commit(JDOTransaction.java:98)
> ... 46 more
> Caused by: com.mysql.jdbc.exceptions.jdbc4.MySQLIntegrityConstraintViolationException: Duplicate entry 'default' for key 'UNIQUE_DATABASE'
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
> at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
> at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
> at com.mysql.jdbc.Util.getInstance(Util.java:386)
> at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1039)
> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3609)
> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3541)
> at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2002)
> at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2163)
> at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2624)
> at com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:2127)
> at com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2427)
> at com.mysql.jdbc.PreparedStatement.executeBatchSerially(PreparedStatement.java:1980)
> ... 57 more
--
This message was sent by Atlassian JIRA
(v6.2#6252)