[jira] [Commented] (HIVE-8519) Hive metastore lock wait timeout

["Sungpeo Kook (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HIVE-8519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17006712#comment-17006712 ] 

Sungpeo Kook commented on HIVE-8519:
------------------------------------

This error is caused by the difference in values ​​for *hive.metastore.client.socket.timeout* which is set to hive-site.xml.

The value for the property is used by hiveserver2 and hivemetastore.

But as you know, you can set properties differently for each service. In this case, hiveserver2 and hivemetastore can have different value for the same proeprty.

 

Above situation occured when hiveserver2's timeout < hivemetastore's timeout.

 

Supposed, you are trying to drop table which includes many partitions. It might be taking long time.
 # hiveserver2 request to drop table
 # hivemetastore start to drop table including partitions in a transaction.
 # After hive.metastore.client.socket.timeout, hiveserver2 got timeout exception and retry to drop table again.
 # hivemetastore is still doing drop table which is requested before, and start to drop table again.
 # New thread for dropping table get lock wait timeout exception.

 

+If you set hive.metastore.client.socket.timeout to same value for hiveserver2 and hivemetastore, you wouldn't face this issue.+

> Hive metastore lock wait timeout
> --------------------------------
>
>                 Key: HIVE-8519
>                 URL: https://issues.apache.org/jira/browse/HIVE-8519
>             Project: Hive
>          Issue Type: Bug
>          Components: Metastore
>    Affects Versions: 0.10.0
>            Reporter: Jason Liao
>            Assignee: Sungpeo Kook
>            Priority: Major
>
> We got a lot of exception as below when doing a drop table partition, which made hive query every every slow. For example, it will cost 250s while executing use db_test;
> Log:
> 2014-10-17 04:04:46,873 ERROR Datastore.Persist (Log4JLogger.java:error(115)) - Update of object "org.apache.hadoop.hive.metastore.model.MStorageDescriptor@13c9c4b3" using statement "UPDATE `SDS` SET `CD_ID`=? WHERE `SD_ID`=?" failed : java.sql.SQLException: Lock wait timeout exceeded; try restarting transaction
>         at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1074)
>         at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4096)
>         at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4028)
>         at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2490)
>         at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2651)
>         at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2734)
>         at com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:2155)
>         at com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2458)
>         at com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2375)
>         at com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2359)
>         at org.apache.commons.dbcp.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:105)
>         at org.apache.commons.dbcp.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:105)
>         at org.datanucleus.store.rdbms.ParamLoggingPreparedStatement.executeUpdate(ParamLoggingPreparedStatement.java:399)
>         at org.datanucleus.store.rdbms.SQLController.executeStatementUpdate(SQLController.java:439)
>         at org.datanucleus.store.rdbms.request.UpdateRequest.execute(UpdateRequest.java:374)
>         at org.datanucleus.store.rdbms.RDBMSPersistenceHandler.updateTable(RDBMSPersistenceHandler.java:417)
>         at org.datanucleus.store.rdbms.RDBMSPersistenceHandler.updateObject(RDBMSPersistenceHandler.java:390)
>         at org.datanucleus.state.JDOStateManager.flush(JDOStateManager.java:5012)
>         at org.datanucleus.FlushOrdered.execute(FlushOrdered.java:106)
>         at org.datanucleus.ExecutionContextImpl.flushInternal(ExecutionContextImpl.java:4019)
>         at org.datanucleus.ExecutionContextThreadedImpl.flushInternal(ExecutionContextThreadedImpl.java:450)
>         at org.datanucleus.store.query.Query.prepareDatastore(Query.java:1575)
>         at org.datanucleus.store.query.Query.executeQuery(Query.java:1760)
>         at org.datanucleus.store.query.Query.executeWithArray(Query.java:1672)
>         at org.datanucleus.api.jdo.JDOQuery.execute(JDOQuery.java:243)
>         at org.apache.hadoop.hive.metastore.ObjectStore.listStorageDescriptorsWithCD(ObjectStore.java:2185)
>         at org.apache.hadoop.hive.metastore.ObjectStore.removeUnusedColumnDescriptor(ObjectStore.java:2131)
>         at org.apache.hadoop.hive.metastore.ObjectStore.preDropStorageDescriptor(ObjectStore.java:2162)
>         at org.apache.hadoop.hive.metastore.ObjectStore.dropPartitionCommon(ObjectStore.java:1361)
>         at org.apache.hadoop.hive.metastore.ObjectStore.dropPartition(ObjectStore.java:1301)
>         at sun.reflect.GeneratedMethodAccessor17.invoke(Unknown Source)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>         at java.lang.reflect.Method.invoke(Method.java:597)
>         at org.apache.hadoop.hive.metastore.RetryingRawStore.invoke(RetryingRawStore.java:111)
>         at $Proxy4.dropPartition(Unknown Source)
>         at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.drop_partition_common(HiveMetaStore.java:1865)
>         at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.drop_partition(HiveMetaStore.java:1911)
>         at sun.reflect.GeneratedMethodAccessor52.invoke(Unknown Source)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>         at java.lang.reflect.Method.invoke(Method.java:597)
>         at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:105)
>         at $Proxy5.drop_partition(Unknown Source)



--
This message was sent by Atlassian Jira
(v8.3.4#803005)