You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues-all@impala.apache.org by "Tianyi Wang (JIRA)" <ji...@apache.org> on 2018/06/15 19:07:00 UTC
[jira] [Commented] (IMPALA-7154) Error making 'dropDatabase' RPC to
Hive Metastore
[ https://issues.apache.org/jira/browse/IMPALA-7154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16514246#comment-16514246 ]
Tianyi Wang commented on IMPALA-7154:
-------------------------------------
Saw it again. I took a brief look. The problem was that the connection to HMS was lost after the drop database commend was sent. HMS client however didn't know the command took effect and retried so it failed.
{noformat}
I0614 21:46:06.395942 14644 ClientCnxn.java:975] Opening socket connection to server localhost/0:0:0:
0:0:0:0:1:2181. Will not attempt to authenticate using SASL (unknown error)
W0614 21:46:06.396255 14644 ClientCnxn.java:1102] Session 0x0 for server null, unexpected error, clos
ing socket connection and attempting reconnect
Java exception follows:
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
W0614 21:46:06.548285 12859 RetryingMetaStoreClient.java:148] MetaStoreClient lost connection. Attemp
ting to reconnect.
Java exception follows:
org.apache.thrift.transport.TTransportException: java.net.SocketTimeoutException: Read timed out
at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:129)
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429)
at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318)
at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219)
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:77)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_drop_database(ThriftHiveMet
astore.java:733)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.drop_database(ThriftHiveMetastor
e.java:718)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.dropDatabase(HiveMetaStoreClient.java:810)
at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:101
)
at com.sun.proxy.$Proxy5.dropDatabase(Unknown Source)
at org.apache.impala.service.CatalogOpExecutor.dropDatabase(CatalogOpExecutor.java:1305)
at org.apache.impala.service.CatalogOpExecutor.execDdlRequest(CatalogOpExecutor.java:300)
at org.apache.impala.service.JniCatalog.execDdl(JniCatalog.java:146)
Caused by: java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:152)
at java.net.SocketInputStream.read(SocketInputStream.java:122)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:275)
at java.io.BufferedInputStream.read(BufferedInputStream.java:334)
at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127)
... 16 more
I0614 21:46:06.550910 12859 HiveMetaStoreClient.java:547] Closed a connection to metastore, current c
onnections: 9
I0614 21:46:06.550966 12859 HiveMetaStoreClient.java:391] Trying to connect to metastore with URI thr
ift://localhost:9083
I0614 21:46:06.551259 12859 HiveMetaStoreClient.java:465] Opened a connection to metastore, current c
onnections: 10
I0614 21:46:06.551961 12859 HiveMetaStoreClient.java:518] Connected to metastore.
I0614 21:46:06.555757 12859 jni-util.cc:230] org.apache.impala.common.ImpalaRuntimeException: Error m
aking 'dropDatabase' RPC to Hive Metastore:
at org.apache.impala.service.CatalogOpExecutor.dropDatabase(CatalogOpExecutor.java:1309)
at org.apache.impala.service.CatalogOpExecutor.execDdlRequest(CatalogOpExecutor.java:300)
at org.apache.impala.service.JniCatalog.execDdl(JniCatalog.java:146)
Caused by: NoSuchObjectException(message:test_fuzz_nested_types_67367717)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_database_result$get_database_result
StandardScheme.read(ThriftHiveMetastore.java:16387)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_database_result$get_database_result
StandardScheme.read(ThriftHiveMetastore.java:16364)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_database_result.read(ThriftHiveMeta
store.java:16295)
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:86)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_database(ThriftHiveMeta
store.java:702)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_database(ThriftHiveMetastore
.java:689)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getDatabase(HiveMetaStoreClient.java:1232)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.dropDatabase(HiveMetaStoreClient.java:791)
at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:101
)
at com.sun.proxy.$Proxy5.dropDatabase(Unknown Source)
at org.apache.impala.service.CatalogOpExecutor.dropDatabase(CatalogOpExecutor.java:1305)
... 2 more
{noformat}
> Error making 'dropDatabase' RPC to Hive Metastore
> -------------------------------------------------
>
> Key: IMPALA-7154
> URL: https://issues.apache.org/jira/browse/IMPALA-7154
> Project: IMPALA
> Issue Type: Bug
> Components: Catalog
> Affects Versions: Impala 2.13.0
> Reporter: Tim Armstrong
> Assignee: Vuk Ercegovac
> Priority: Critical
> Labels: broken-build, flaky
> Attachments: TEST-impala-parallel.log.gz, TEST-impala-parallel.xml.gz, catalogd.ec2-m2-4xlarge-centos-6-4-0f46.vpc.cloudera.com.jenkins.log.INFO.20180608-024815.32143.gz, hive.log.gz
>
>
> {noformat}
> conftest.py:293: in cleanup
> {'sync_ddl': sync_ddl})
> common/impala_test_suite.py:528: in wrapper
> return function(*args, **kwargs)
> common/impala_test_suite.py:535: in execute_query_expect_success
> result = cls.__execute_query(impalad_client, query, query_options)
> common/impala_test_suite.py:620: in __execute_query
> return impalad_client.execute(query, user=user)
> common/impala_connection.py:160: in execute
> return self.__beeswax_client.execute(sql_stmt, user=user)
> beeswax/impala_beeswax.py:173: in execute
> handle = self.__execute_query(query_string.strip(), user=user)
> beeswax/impala_beeswax.py:339: in __execute_query
> handle = self.execute_query_async(query_string, user=user)
> beeswax/impala_beeswax.py:335: in execute_query_async
> return self.__do_rpc(lambda: self.imp_service.query(query,))
> beeswax/impala_beeswax.py:460: in __do_rpc
> raise ImpalaBeeswaxException(self.__build_error_message(b), b)
> E ImpalaBeeswaxException: ImpalaBeeswaxException:
> E INNER EXCEPTION: <class 'beeswaxd.ttypes.BeeswaxException'>
> E MESSAGE: ImpalaRuntimeException: Error making 'dropDatabase' RPC to Hive Metastore:
> E CAUSED BY: NoSuchObjectException: test_resolution_by_name_56b45511
> {noformat}
> The backtrace in the catalogd log is:
> {noformat}
> I0608 05:49:26.111824 24195 jni-util.cc:230] org.apache.impala.common.ImpalaRuntimeException: Error making 'dropDatabase' RPC to Hive Metastore:
> at org.apache.impala.service.CatalogOpExecutor.dropDatabase(CatalogOpExecutor.java:1309)
> at org.apache.impala.service.CatalogOpExecutor.execDdlRequest(CatalogOpExecutor.java:300)
> at org.apache.impala.service.JniCatalog.execDdl(JniCatalog.java:146)
> Caused by: NoSuchObjectException(message:test_resolution_by_name_56b45511)
> at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_database_result$get_database_resultStandardScheme.read(ThriftHiveMetastore.java:16387)
> at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_database_result$get_database_resultStandardScheme.read(ThriftHiveMetastore.java:16364)
> at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_database_result.read(ThriftHiveMetastore.java:16295)
> at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:86)
> at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_database(ThriftHiveMetastore.java:702)
> at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_database(ThriftHiveMetastore.java:689)
> at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getDatabase(HiveMetaStoreClient.java:1232)
> at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.dropDatabase(HiveMetaStoreClient.java:791)
> at sun.reflect.GeneratedMethodAccessor26.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:101)
> at com.sun.proxy.$Proxy5.dropDatabase(Unknown Source)
> at org.apache.impala.service.CatalogOpExecutor.dropDatabase(CatalogOpExecutor.java:1305)
> ... 2 more
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org