You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues-all@impala.apache.org by "Laszlo Gaal (JIRA)" <ji...@apache.org> on 2019/01/15 16:43:00 UTC
[jira] [Commented] (IMPALA-8008) InvocationTargetException when
making HMS RPCs
[ https://issues.apache.org/jira/browse/IMPALA-8008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16743208#comment-16743208 ]
Laszlo Gaal commented on IMPALA-8008:
-------------------------------------
[~fredyw], would you mind adding some context as to how & why this item got resolved? Thanks!
> InvocationTargetException when making HMS RPCs
> ----------------------------------------------
>
> Key: IMPALA-8008
> URL: https://issues.apache.org/jira/browse/IMPALA-8008
> Project: IMPALA
> Issue Type: Bug
> Components: Frontend
> Affects Versions: Impala 3.2.0
> Reporter: bharath v
> Assignee: Fredy Wijaya
> Priority: Major
> Labels: broken-build, flaky
> Fix For: Impala 3.2.0
>
>
> A bunch of tests failed with the following HMS exception in logs. Looks like a flaky HMS issue but raising this jira to rule out any obvious bugs.
> *Failed tests*
> {noformat}
> authorization.test_grant_revoke.TestGrantRevoke.test_same_name_for_role_and_user_invalidate_metadata
> authorization.test_owner_privileges.TestOwnerPrivileges.test_owner_privileges_with_grant[protocol: beeswax | exec_option: {'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 0, 'disable_codegen': False, 'abort_on_error': 1, 'debug_action': None, 'exec_single_node_rows_threshold': 0} | table_format: text/none]
> authorization.test_owner_privileges.TestOwnerPrivileges.test_owner_privileges_disabled[protocol: beeswax | exec_option: {'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 0, 'disable_codegen': False, 'abort_on_error': 1, 'debug_action': None, 'exec_single_node_rows_threshold': 0} | table_format: text/none]
> authorization.test_owner_privileges.TestOwnerPrivileges.test_owner_privileges_without_grant[protocol: beeswax | exec_option: {'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 0, 'disable_codegen': False, 'abort_on_error': 1, 'debug_action': None, 'exec_single_node_rows_threshold': 0} | table_format: text/none]
> authorization.test_owner_privileges.TestOwnerPrivileges.test_owner_privileges_different_cases[protocol: beeswax | exec_option: {'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 0, 'disable_codegen': False, 'abort_on_error': 1, 'debug_action': None, 'exec_single_node_rows_threshold': 0} | table_format: text/none]
> authorization.test_show_grant.TestShowGrant.test_show_grant_user[protocol: beeswax | exec_option: {'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 0, 'disable_codegen': False, 'abort_on_error': 1, 'debug_action': None, 'exec_single_node_rows_threshold': 0} | table_format: text/none]
> authorization.test_show_grant.TestShowGrant.test_show_grant_in_hs2[protocol: beeswax | exec_option: {'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 0, 'disable_codegen': False, 'abort_on_error': 1, 'debug_action': None, 'exec_single_node_rows_threshold': 0} | table_format: text/none]
> query_test.test_insert.TestInsertQueries.test_insert_large_string[protocol: beeswax | exec_option: {'sync_ddl': 0, 'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 0, 'disable_codegen': False, 'abort_on_error': 1, 'debug_action': None, 'exec_single_node_rows_threshold': 0} | table_format: text/none]
> {noformat}
> *Full stack trace*
> {noformat}
> 2018-12-19T14:29:05,730 WARN [pool-8-thread-3] metastore.ObjectStore: Failed to get database test_owner_privileges_without_grant_fd071aba, returning NoSuchObjectException
> 2018-12-19T14:29:05,779 INFO [pool-8-thread-3] common.FileUtils: Creating directory if it doesn't exist: s3a://impala-test-uswest2-2/test-warehouse/test_owner_privileges_without_grant_fd071aba.db
> 2018-12-19T14:29:05,950 ERROR [pool-8-thread-3] transport.RetryClientInvocationHandler: failed to execute notifyHmsEvent
> java.lang.reflect.InvocationTargetException: null
> at sun.reflect.GeneratedMethodAccessor44.invoke(Unknown Source) ~[?:?]
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_144]
> at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_144]
> at org.apache.sentry.core.common.transport.RetryClientInvocationHandler.invokeImpl(RetryClientInvocationHandler.java:95) [sentry-core-common-2.1.0-cdh6.x-SNAPSHOT.jar:?]
> at org.apache.sentry.core.common.transport.SentryClientInvocationHandler.invoke(SentryClientInvocationHandler.java:41) [sentry-core-common-2.1.0-cdh6.x-SNAPSHOT.jar:?]
> at com.sun.proxy.$Proxy38.notifyHmsEvent(Unknown Source) [?:?]
> at org.apache.sentry.binding.metastore.SentrySyncHMSNotificationsPostEventListener.notifyHmsEvent(SentrySyncHMSNotificationsPostEventListener.java:252) [sentry-binding-hive-2.1.0-cdh6.x-SNAPSHOT.jar:?]
> at org.apache.sentry.binding.metastore.SentrySyncHMSNotificationsPostEventListener.onCreateDatabase(SentrySyncHMSNotificationsPostEventListener.java:195) [sentry-binding-hive-2.1.0-cdh6.x-SNAPSHOT.jar:?]
> at org.apache.hadoop.hive.metastore.MetaStoreListenerNotifier$15.notify(MetaStoreListenerNotifier.java:64) [hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
> at org.apache.hadoop.hive.metastore.MetaStoreListenerNotifier.notifyEvent(MetaStoreListenerNotifier.java:175) [hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
> at org.apache.hadoop.hive.metastore.MetaStoreListenerNotifier.notifyEvent(MetaStoreListenerNotifier.java:205) [hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
> at org.apache.hadoop.hive.metastore.MetaStoreListenerNotifier.notifyEvent(MetaStoreListenerNotifier.java:243) [hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
> at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_database_core(HiveMetaStore.java:896) [hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
> at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_database(HiveMetaStore.java:928) [hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
> at sun.reflect.GeneratedMethodAccessor72.invoke(Unknown Source) ~[?:?]
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_144]
> at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_144]
> at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:140) [hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
> at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:99) [hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
> at com.sun.proxy.$Proxy26.create_database(Unknown Source) [?:?]
> at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$create_database.getResult(ThriftHiveMetastore.java:10589) [hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
> at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$create_database.getResult(ThriftHiveMetastore.java:10573) [hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) [hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
> at org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:110) [hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
> at org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:106) [hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
> at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_144]
> at javax.security.auth.Subject.doAs(Subject.java:422) [?:1.8.0_144]
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1875) [hadoop-common-3.0.0-cdh6.x-20181219.140835-1.jar:?]
> at org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:118) [hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
> at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286) [hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_144]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_144]
> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_144]
> Caused by: org.apache.sentry.core.common.exception.SentryUserException: Thrift exception occurred
> at org.apache.sentry.api.service.thrift.SentryPolicyServiceClientDefaultImpl.notifyHmsEvent(SentryPolicyServiceClientDefaultImpl.java:1188) ~[sentry-service-api-2.1.0-cdh6.x-SNAPSHOT.jar:?]
> ... 33 more
> Caused by: org.apache.thrift.transport.TTransportException
> at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132) ~[hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
> at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86) ~[hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
> at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429) ~[hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
> at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318) ~[hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
> at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219) ~[hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
> at org.apache.thrift.protocol.TProtocolDecorator.readMessageBegin(TProtocolDecorator.java:135) ~[hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
> at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:77) ~[hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
> at org.apache.sentry.api.service.thrift.SentryPolicyService$Client.recv_sentry_notify_hms_event(SentryPolicyService.java:663) ~[sentry-service-api-2.1.0-cdh6.x-SNAPSHOT.jar:?]
> at org.apache.sentry.api.service.thrift.SentryPolicyService$Client.sentry_notify_hms_event(SentryPolicyService.java:650) ~[sentry-service-api-2.1.0-cdh6.x-SNAPSHOT.jar:?]
> at org.apache.sentry.api.service.thrift.SentryPolicyServiceClientDefaultImpl.notifyHmsEvent(SentryPolicyServiceClientDefaultImpl.java:1184) ~[sentry-service-api-2.1.0-cdh6.x-SNAPSHOT.jar:?]
> ... 33 more
> 2018-12-19T14:29:05,956 ERROR [pool-8-thread-3] transport.RetryClientInvocationHandler: Thrift call failed
> org.apache.thrift.transport.TTransportException: org.apache.thrift.transport.TTransportException
> at org.apache.sentry.core.common.transport.RetryClientInvocationHandler.invokeImpl(RetryClientInvocationHandler.java:110) [sentry-core-common-2.1.0-cdh6.x-SNAPSHOT.jar:?]
> at org.apache.sentry.core.common.transport.SentryClientInvocationHandler.invoke(SentryClientInvocationHandler.java:41) [sentry-core-common-2.1.0-cdh6.x-SNAPSHOT.jar:?]
> at com.sun.proxy.$Proxy38.notifyHmsEvent(Unknown Source) [?:?]
> at org.apache.sentry.binding.metastore.SentrySyncHMSNotificationsPostEventListener.notifyHmsEvent(SentrySyncHMSNotificationsPostEventListener.java:252) [sentry-binding-hive-2.1.0-cdh6.x-SNAPSHOT.jar:?]
> at org.apache.sentry.binding.metastore.SentrySyncHMSNotificationsPostEventListener.onCreateDatabase(SentrySyncHMSNotificationsPostEventListener.java:195) [sentry-binding-hive-2.1.0-cdh6.x-SNAPSHOT.jar:?]
> at org.apache.hadoop.hive.metastore.MetaStoreListenerNotifier$15.notify(MetaStoreListenerNotifier.java:64) [hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
> at org.apache.hadoop.hive.metastore.MetaStoreListenerNotifier.notifyEvent(MetaStoreListenerNotifier.java:175) [hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
> at org.apache.hadoop.hive.metastore.MetaStoreListenerNotifier.notifyEvent(MetaStoreListenerNotifier.java:205) [hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
> at org.apache.hadoop.hive.metastore.MetaStoreListenerNotifier.notifyEvent(MetaStoreListenerNotifier.java:243) [hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
> at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_database_core(HiveMetaStore.java:896) [hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
> at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_database(HiveMetaStore.java:928) [hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
> at sun.reflect.GeneratedMethodAccessor72.invoke(Unknown Source) ~[?:?]
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_144]
> at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_144]
> at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:140) [hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
> at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:99) [hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
> at com.sun.proxy.$Proxy26.create_database(Unknown Source) [?:?]
> at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$create_database.getResult(ThriftHiveMetastore.java:10589) [hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
> at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$create_database.getResult(ThriftHiveMetastore.java:10573) [hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) [hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
> at org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:110) [hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
> at org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:106) [hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
> at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_144]
> at javax.security.auth.Subject.doAs(Subject.java:422) [?:1.8.0_144]
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1875) [hadoop-common-3.0.0-cdh6.x-20181219.140835-1.jar:?]
> at org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:118) [hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
> at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286) [hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_144]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_144]
> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_144]
> Caused by: org.apache.thrift.transport.TTransportException
> at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132) ~[hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
> at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86) ~[hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
> at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429) ~[hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
> at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318) ~[hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
> at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219) ~[hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
> at org.apache.thrift.protocol.TProtocolDecorator.readMessageBegin(TProtocolDecorator.java:135) ~[hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
> at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:77) ~[hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
> at org.apache.sentry.api.service.thrift.SentryPolicyService$Client.recv_sentry_notify_hms_event(SentryPolicyService.java:663) ~[sentry-service-api-2.1.0-cdh6.x-SNAPSHOT.jar:?]
> at org.apache.sentry.api.service.thrift.SentryPolicyService$Client.sentry_notify_hms_event(SentryPolicyService.java:650) ~[sentry-service-api-2.1.0-cdh6.x-SNAPSHOT.jar:?]
> at org.apache.sentry.api.service.thrift.SentryPolicyServiceClientDefaultImpl.notifyHmsEvent(SentryPolicyServiceClientDefaultImpl.java:1184) ~[sentry-service-api-2.1.0-cdh6.x-SNAPSHOT.jar:?]
> at sun.reflect.GeneratedMethodAccessor44.invoke(Unknown Source) ~[?:?]
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_144]
> at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_144]
> at org.apache.sentry.core.common.transport.RetryClientInvocationHandler.invokeImpl(RetryClientInvocationHandler.java:95) ~[sentry-core-common-2.1.0-cdh6.x-SNAPSHOT.jar:?]
> ... 29 more
> 2018-12-19T14:29:06,258 INFO [pool-8-thread-3] metastore.HiveMetaStore: 3: source:127.0.0.1 get_database: test_owner_privileges_without_grant_fd071aba
> 2018-12-19T14:29:06,258 INFO [pool-8-thread-3] HiveMetaStore.audit: ugi=jenkins ip=127.0.0.1 cmd=source:127.0.0.1 get_database: test_owner_privileges_without_grant_fd071aba
> 2018-12-19T14:29:06,271 INFO [pool-8-thread-173] metastore.HiveMetaStore: 54: source:127.0.0.1 get_database: owner_priv_db
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org