You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues-all@impala.apache.org by "Vihang Karajgaonkar (Jira)" <ji...@apache.org> on 2020/06/24 17:05:00 UTC
[jira] [Commented] (IMPALA-9887) Custom cluster tests fail due to
HMS connectivity issues
[ https://issues.apache.org/jira/browse/IMPALA-9887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17144065#comment-17144065 ]
Vihang Karajgaonkar commented on IMPALA-9887:
---------------------------------------------
Looks like HMS service was indeed down since 17:04 on the server time. Hence all the custom cluster tests fail after that.
{noformat}
2020-06-23T17:04:53,577 INFO [pool-6-thread-7] HiveMetaStore.audit: ugi=jenkins ip=127.0.0.1 cmd=Cleaning up thread local RawStore...
2020-06-23T17:04:53,577 INFO [pool-6-thread-3] metastore.HiveMetaStore: 8: Done cleaning up thread local RawStore
2020-06-23T17:04:53,577 INFO [pool-6-thread-5] HiveMetaStore.audit: ugi=jenkins ip=127.0.0.1 cmd=Done cleaning up thread local RawStore
2020-06-23T17:04:53,577 INFO [pool-6-thread-3] HiveMetaStore.audit: ugi=jenkins ip=127.0.0.1 cmd=Done cleaning up thread local RawStore
2020-06-23T17:04:53,577 INFO [pool-6-thread-7] metastore.ObjectStore: RawStore: org.apache.hadoop.hive.metastore.ObjectStore@329ae9d3, with PersistenceManager: org.datanucleus.api.jdo.JDOPersistenceManager@698e1141 will be shutdown
2020-06-23T17:04:53,577 INFO [pool-6-thread-92] metastore.HiveMetaStore: 97: Done cleaning up thread local RawStore
2020-06-23T17:04:53,577 INFO [pool-6-thread-92] HiveMetaStore.audit: ugi=jenkins ip=127.0.0.1 cmd=Done cleaning up thread local RawStore
2020-06-23T17:04:53,577 INFO [pool-6-thread-7] metastore.HiveMetaStore: 12: Done cleaning up thread local RawStore
2020-06-23T17:04:53,577 INFO [pool-6-thread-7] HiveMetaStore.audit: ugi=jenkins ip=127.0.0.1 cmd=Done cleaning up thread local RawStore
2020-06-23T17:04:53,772 INFO [shutdown-hook-0] metastore.HiveMetaStore: Shutting down hive metastore.
2020-06-23T17:04:53,781 INFO [shutdown-hook-0] metastore.HiveMetaStore: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down HiveMetaStore at impala-ec2-centos74-r5-4xlarge-ondemand-05f2.vpc.cloudera.com/127.0.0.1
************************************************************/
{noformat}
> Custom cluster tests fail due to HMS connectivity issues
> --------------------------------------------------------
>
> Key: IMPALA-9887
> URL: https://issues.apache.org/jira/browse/IMPALA-9887
> Project: IMPALA
> Issue Type: Bug
> Affects Versions: Impala 4.0
> Reporter: Vihang Karajgaonkar
> Priority: Blocker
> Labels: broken-build
>
> It has happened atleast couple of times in this week on the ASAN builds. The custom cluster tests fails with the test setup error and logs suggest that coordinator nodes don't start up due to the following exception trace:
> {noformat}
> F0623 17:22:42.725920 25786 frontend.cc:136] IllegalStateException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient
> CAUSED BY: RuntimeException: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient
> CAUSED BY: InvocationTargetException: null
> CAUSED BY: MetaException: Could not connect to meta store using any of the URIs provided. Most recent failure: org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused (Connection refused)
> at org.apache.thrift.transport.TSocket.open(TSocket.java:226)
> at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:631)
> at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:241)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at org.apache.hadoop.hive.metastore.utils.JavaUtils.newInstance(JavaUtils.java:84)
> at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:95)
> at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:148)
> at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:119)
> at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:112)
> at org.apache.impala.catalog.MetaStoreClientPool$MetaStoreClient.<init>(MetaStoreClientPool.java:99)
> at org.apache.impala.catalog.MetaStoreClientPool$MetaStoreClient.<init>(MetaStoreClientPool.java:78)
> at org.apache.impala.catalog.MetaStoreClientPool.initClients(MetaStoreClientPool.java:174)
> at org.apache.impala.catalog.MetaStoreClientPool.<init>(MetaStoreClientPool.java:163)
> at org.apache.impala.catalog.MetaStoreClientPool.<init>(MetaStoreClientPool.java:155)
> at org.apache.impala.service.Frontend.<init>(Frontend.java:331)
> at org.apache.impala.service.Frontend.<init>(Frontend.java:288)
> at org.apache.impala.service.JniFrontend.<init>(JniFrontend.java:144)
> Caused by: java.net.ConnectException: Connection refused (Connection refused)
> at java.net.PlainSocketImpl.socketConnect(Native Method)
> at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
> at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
> at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
> at java.net.Socket.connect(Socket.java:589)
> at org.apache.thrift.transport.TSocket.open(TSocket.java:221)
> ... 19 more
> . Impalad exiting.
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org