You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ambari.apache.org by "Eric Yang (JIRA)" <ji...@apache.org> on 2017/07/27 17:59:00 UTC
[jira] [Commented] (AMBARI-21577) Hive-Service check failing in
post EU validation (BI-HDP)
[ https://issues.apache.org/jira/browse/AMBARI-21577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16103613#comment-16103613 ]
Eric Yang commented on AMBARI-21577:
------------------------------------
In webhcat log file, this shows:
{code}
ERROR | 27 Jul 2017 17:31:27,169 | org.apache.hive.hcatalog.templeton.CatchallExceptionMapper | java.lang.reflect.UndeclaredThrowableException
java.io.IOException: java.lang.reflect.UndeclaredThrowableException
at org.apache.hive.hcatalog.templeton.SecureProxySupport.open(SecureProxySupport.java:91)
at org.apache.hive.hcatalog.templeton.HcatDelegator.run(HcatDelegator.java:63)
at org.apache.hive.hcatalog.templeton.Server.ddl(Server.java:219)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$TypeOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:185)
at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302)
at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1480)
at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1411)
at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1360)
at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1350)
at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416)
at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:538)
at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:716)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:565)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1360)
at org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:617)
at org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:576)
at org.apache.hadoop.hdfs.web.AuthFilter.doFilter(AuthFilter.java:90)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1331)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:477)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1031)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:406)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:965)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:117)
at org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:47)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:111)
at org.eclipse.jetty.server.Server.handle(Server.java:349)
at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:449)
at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:925)
at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:857)
at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)
at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:76)
at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:609)
at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:45)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.reflect.UndeclaredThrowableException
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1884)
at org.apache.hive.hcatalog.templeton.SecureProxySupport.buildHcatDelegationToken(SecureProxySupport.java:208)
at org.apache.hive.hcatalog.templeton.SecureProxySupport.open(SecureProxySupport.java:89)
... 46 more
Caused by: MetaException(message:User: HTTP/eyang-2.openstacklocal@EXAMPLE.COM is not allowed to impersonate ambari-qa)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_delegation_token_result$get_delegation_token_resultStandardScheme.read(ThriftHiveMetastore.java)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_delegation_token_result$get_delegation_token_resultStandardScheme.read(ThriftHiveMetastore.java)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_delegation_token_result.read(ThriftHiveMetastore.java)
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:86)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_delegation_token(ThriftHiveMetastore.java:3793)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_delegation_token(ThriftHiveMetastore.java:3779)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getDelegationToken(HiveMetaStoreClient.java:1824)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:174)
at com.sun.proxy.$Proxy67.getDelegationToken(Unknown Source)
at org.apache.hive.hcatalog.templeton.SecureProxySupport$3.run(SecureProxySupport.java:212)
at org.apache.hive.hcatalog.templeton.SecureProxySupport$3.run(SecureProxySupport.java:208)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866)
... 48 more
{code}
This means HDP is setup HTTP principal to map to a different user. Where IOP is setup to map HTTP principal to hbase user. There is a conflict of interests of what HTTP principal should map to which proxy user. In the ideal case, both component should not use HTTP principal as a proxy user. Internally in the component, it should retrieve end user credential and invoke component's own principal such as hbase, or hcat to run doAs. This will ensure HTTP principal is not over used by multiple service to proxy access.
> Hive-Service check failing in post EU validation (BI-HDP)
> ---------------------------------------------------------
>
> Key: AMBARI-21577
> URL: https://issues.apache.org/jira/browse/AMBARI-21577
> Project: Ambari
> Issue Type: Bug
> Components: stacks
> Affects Versions: 2.5.2
> Environment: OS:- RHEL 7
> Ambari Upgraded 2.2.0 to 2.5.2.0-174
> Express Upgrade:- BigInsights-4.2.0.0 to HDP-2.6.2.0-107
> Reporter: Eric Yang
> Assignee: Siddharth Wagle
> Fix For: 2.5.2
>
> Attachments: AMBARI-21577.patch
>
>
> Steps to reproduce:-
> 1. Installed a IOP cluster ambari-version:- 2.2.0/20160616_1658,BigInsights-4.2.0.0
> 2. Upgrade the ambari from 2.2.0 to 2.5.2.0-174(IOP Clusters)
> 3. Remove IOP Select.
> 4. Register HDP Stack to HDP-2.6.2.0-107.
> 5. EU
> 6. Post EU
> Hive- Service check is failing :-
> {code}
> HTTP/vs-iop420tofnsec-re-2.openstacklocal@EXAMPLE.COM is not allowed to impersonate ambari-qa
> {code}
> stderr:-
> {code}
> Traceback (most recent call last):
> File "/var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/scripts/service_check.py", line 194, in <module>
> HiveServiceCheck().execute()
> File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 329, in execute
> method(env)
> File "/var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/scripts/service_check.py", line 99, in service_check
> webhcat_service_check()
> File "/usr/lib/python2.6/site-packages/ambari_commons/os_family_impl.py", line 89, in thunk
> return fn(*args, **kwargs)
> File "/var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/scripts/webhcat_service_check.py", line 125, in webhcat_service_check
> logoutput=True)
> File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 166, in __init__
> self.env.run()
> File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 160, in run
> self.run_action(resource, action)
> File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 124, in run_action
> provider_action()
> File "/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py", line 262, in action_run
> tries=self.resource.tries, try_sleep=self.resource.try_sleep)
> File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 72, in inner
> result = function(command, **kwargs)
> File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 102, in checked_call
> tries=tries, try_sleep=try_sleep, timeout_kill_strategy=timeout_kill_strategy)
> File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 150, in _call_wrapper
> result = _call(command, **kwargs_copy)
> File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 303, in _call
> raise ExecutionFailed(err_msg, code, out, err)
> resource_management.core.exceptions.ExecutionFailed: Execution of '/var/lib/ambari-agent/tmp/templetonSmoke.sh vs-iop420tofnsec-re-2.openstacklocal ambari-qa 20111 idtest.ambari-qa.1500877355.88.pig /etc/security/keytabs/smokeuser.headless.keytab true /usr/bin/kinit ambari-qa@EXAMPLE.COM /var/lib/ambari-agent/tmp' returned 1. Templeton Smoke Test (ddl cmd): Failed. : {"error":"User: HTTP/vs-iop420tofnsec-re-2.openstacklocal@EXAMPLE.COM is not allowed to impersonate ambari-qa"}http_code <500>
> {code}
> Screenshot:- !Screen Shot 2017-07-24 at 12.04.44 PM.png|thumbnail!
> Live-Server:- http://172.22.115.63:8080.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)