You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by "Andrew Wong (JIRA)" <ji...@apache.org> on 2019/05/20 23:05:02 UTC
[jira] [Created] (HIVE-21759) Confusing error message as a result
of concurrent drop and create operations for managed tables
Andrew Wong created HIVE-21759:
----------------------------------
Summary: Confusing error message as a result of concurrent drop and create operations for managed tables
Key: HIVE-21759
URL: https://issues.apache.org/jira/browse/HIVE-21759
Project: Hive
Issue Type: Bug
Components: Metastore
Affects Versions: 2.3.1
Environment: Kudu has been working to synchronize its internal catalog with the Hive Metastore. This issue was caught in the context of a Kudu unit test when we started using MANAGED tables for the tables created by Kudu. I note in this [WIP patch|https://gerrit.cloudera.org/c/13315/2] the test in which I am seeing the flakiness.
Reporter: Andrew Wong
It's possible for concurrent DROP TABLE and CREATE TABLE operations to yield the following sequence of events:
# The HMS receives a request to create managed table T
# The HMS will create a (HD)FS directory for T if one doesn't already exist
# The HMS will create a metadata entry for table T, making it visible to clients
# The client sends a DROP TABLE request for table T
# The table entry is dropped in the HMS
# The HMS will eventually drop the directory from (HD)FS
If in between #5 and #6, the client sends another create table request for T, the request will begin to be processed; the table will not exist in the HMS (even though its directory exists), so the create will begin to proceed as if no underlying data exists. Eventually, #6 will complete, and if this coincides with the part of the second create table operation that enumerates the files in the directory (e.g. when [generating the CreateTable notification event|https://github.com/apache/hive/blob/branch-2/hcatalog/server-extensions/src/main/java/org/apache/hive/hcatalog/listener/DbNotificationListener.java#L156]), the directory will not be there, potentially resulting in the following stack traces:
{{21:44:29.974 [HMS - ERROR - pool-8-thread-7] (RetryingHMSHandler.java:200) MetaException(message:java.lang.RuntimeException: java.io.FileNotFoundException: File [file:/tmp/dist-test-taskI7tPbq/test-tmp/master-stress-test.0.HmsSentryConfigurations_MasterStressTest.Test_2.1558388453623586-26465/minicluster-data/warehouse/table_c469fde3312d473fb7ac37f526e4dce0|file:///tmp/dist-test-taskI7tPbq/test-tmp/master-stress-test.0.HmsSentryConfigurations_MasterStressTest.Test_2.1558388453623586-26465/minicluster-data/warehouse/table_c469fde3312d473fb7ac37f526e4dce0] does not exist)}}
{{ at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.newMetaException(HiveMetaStore.java:6189)}}
{{ at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_with_environment_context(HiveMetaStore.java:1553)}}
{{ at sun.reflect.GeneratedMethodAccessor12.invoke(Unknown Source)}}
{{ at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)}}
{{ at java.lang.reflect.Method.invoke(Method.java:498)}}
{{ at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:148)}}
{{ at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:107)}}
{{ at com.sun.proxy.$Proxy24.create_table_with_environment_context(Unknown Source)}}
{{ at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$create_table_with_environment_context.getResult(ThriftHiveMetastore.java:11191)}}
{{ at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$create_table_with_environment_context.getResult(ThriftHiveMetastore.java:11175)}}
{{ at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)}}
{{ at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)}}
{{ at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor$1.run(HadoopThriftAuthBridge.java:626)}}
{{ at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor$1.run(HadoopThriftAuthBridge.java:621)}}
{{ at java.security.AccessController.doPrivileged(Native Method)}}
{{ at javax.security.auth.Subject.doAs(Subject.java:422)}}
{{ at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1844)}}
{{ at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor.process(HadoopThriftAuthBridge.java:621)}}
{{ at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)}}
{{ at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)}}
{{ at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)}}
{{ at java.lang.Thread.run(Thread.java:748)}}
{{ Caused by: java.lang.RuntimeException: java.io.FileNotFoundException: File [file:/tmp/dist-test-taskI7tPbq/test-tmp/master-stress-test.0.HmsSentryConfigurations_MasterStressTest.Test_2.1558388453623586-26465/minicluster-data/warehouse/table_c469fde3312d473fb7ac37f526e4dce0|file:///tmp/dist-test-taskI7tPbq/test-tmp/master-stress-test.0.HmsSentryConfigurations_MasterStressTest.Test_2.1558388453623586-26465/minicluster-data/warehouse/table_c469fde3312d473fb7ac37f526e4dce0] does not exist}}
{{ at org.apache.hive.hcatalog.listener.DbNotificationListener$FileIterator.<init>(DbNotificationListener.java:222)}}
{{ at org.apache.hive.hcatalog.listener.DbNotificationListener.onCreateTable(DbNotificationListener.java:156)}}
{{ at org.apache.hadoop.hive.metastore.MetaStoreListenerNotifier$13.notify(MetaStoreListenerNotifier.java:76)}}
{{ at org.apache.hadoop.hive.metastore.MetaStoreListenerNotifier.notifyEvent(MetaStoreListenerNotifier.java:175)}}
{{ at org.apache.hadoop.hive.metastore.MetaStoreListenerNotifier.notifyEvent(MetaStoreListenerNotifier.java:205)}}
{{ at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_core(HiveMetaStore.java:1500)}}
{{ at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_with_environment_context(HiveMetaStore.java:1539)}}
{{ ... 20 more}}
{{ Caused by: java.io.FileNotFoundException: File [file:/tmp/dist-test-taskI7tPbq/test-tmp/master-stress-test.0.HmsSentryConfigurations_MasterStressTest.Test_2.1558388453623586-26465/minicluster-data/warehouse/table_c469fde3312d473fb7ac37f526e4dce0|file:///tmp/dist-test-taskI7tPbq/test-tmp/master-stress-test.0.HmsSentryConfigurations_MasterStressTest.Test_2.1558388453623586-26465/minicluster-data/warehouse/table_c469fde3312d473fb7ac37f526e4dce0] does not exist}}
{{ at org.apache.hadoop.fs.RawLocalFileSystem.listStatus(RawLocalFileSystem.java:455)}}
{{ at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1537)}}
{{ at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1580)}}
{{ at org.apache.hadoop.fs.ChecksumFileSystem.listStatus(ChecksumFileSystem.java:675)}}
{{ at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1537)}}
{{ at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1580)}}
{{ at org.apache.hive.hcatalog.listener.DbNotificationListener$FileIterator.<init>(DbNotificationListener.java:219)}}
{{ ... 26 more}}
The client will see that the create table failed because of a MetaException, in which a file was not found. This is a somewhat confusing message, though the end behavior is cut and dry: there will be no HMS entry (should be rolled back as a part of the transaction) and no directory following this sequence of events.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)