You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@oozie.apache.org by Jiri Kaplan <Ji...@software.dell.com> on 2016/08/25 15:14:12 UTC
Oozie Hive2 Action with Kerberos security and HS2 HTTP transport
mode
Hi,
I'd like to ask for a help with Oozie Hive2 action on HDP-2.3.4.0 cluster with Oozie 4.2.0.2.3 installed and with enabled security over Kerberos. Oozie job always ends up with following exception: HiveSQLException: Delegation token only supported over kerberos authentication. We have HiveServer2 configured with hive.server2.transport.mode=http, hive.server2.thrift.http.path=cliservice and hive.server2.thrift.http.port=10001. I'm not sure if I do something wrong or if this configuration is even supported but when we switch back HS2 transport mode to binary it works. Any kind of help is welcome.
Exception stack trace (from HS2 log):
2016-08-25 11:01:23,337 ERROR [HiveServer2-HttpHandler-Pool: Thread-38]: thrift.ThriftCLIService (ThriftCLIService.java:GetDelegationToken(237)) - Error obtaining delegation token
org.apache.hive.service.cli.HiveSQLException: Delegation token only supported over kerberos authentication
at org.apache.hive.service.auth.HiveAuthFactory.getDelegationToken(HiveAuthFactory.java:283)
at org.apache.hive.service.cli.session.HiveSessionImplwithUGI.getDelegationToken(HiveSessionImplwithUGI.java:192)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:78)
at org.apache.hive.service.cli.session.HiveSessionProxy.access$000(HiveSessionProxy.java:36)
at org.apache.hive.service.cli.session.HiveSessionProxy$1.run(HiveSessionProxy.java:63)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:59)
at com.sun.proxy.$Proxy20.getDelegationToken(Unknown Source)
at org.apache.hive.service.cli.CLIService.getDelegationToken(CLIService.java:484)
at org.apache.hive.service.cli.thrift.ThriftCLIService.GetDelegationToken(ThriftCLIService.java:231)
at org.apache.hive.service.cli.thrift.TCLIService$Processor$GetDelegationToken.getResult(TCLIService.java:1573)
at org.apache.hive.service.cli.thrift.TCLIService$Processor$GetDelegationToken.getResult(TCLIService.java:1558)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
at org.apache.thrift.server.TServlet.doPost(TServlet.java:83)
at org.apache.hive.service.cli.thrift.ThriftHttpServlet.doPost(ThriftHttpServlet.java:171)
Here is my workflow.xml content:
<workflow-app xmlns="uri:oozie:workflow:0.5" name="HIVE2 HTTP Kerberos Test">
<global>
<job-tracker>myrmaddress:8050</job-tracker>
<name-node>hdfs://mynnaddress:8020/</name-node>
</global>
<credentials>
<credential name="hive2creds" type="hive2">
<property>
<name>hive2.jdbc.url</name>
<value>jdbc:hive2://myhiveserver:10001/;sasl.qop=auth-conf;transportMode=http;httpPath=cliservice</value>
</property>
<property>
<name>hive2.server.principal</name>
<value>hive/myhiveserver@mydomain</value>
</property>
</credential>
</credentials>
<start to="MyHiveAction"/>
<action cred="hive2creds" name=" MyHiveAction ">
<hive2 xmlns="uri:oozie:hive2-action:0.1">
<jdbc-url> jdbc:hive2://myhiveserver:10001/;sasl.qop=auth-conf;transportMode=http;httpPath=cliservice </jdbc-url>
<script>script.hql</script>
</hive2>
<ok to="end"/>
<error to="fail"/>
</action>
<kill name="fail">
<message>Action failed, error
message[${wf:errorMessage(wf:lastErrorNode())}]
</message>
</kill>
<end name="end"/>
</workflow-app>
Jiří Kaplan
Software Developer
Dell | R&D Database Management, EMEA
[dell_software]
RE: Oozie Hive2 Action with Kerberos security and HS2 HTTP
transport mode
Posted by Jiri Kaplan <Ji...@software.dell.com>.
Hi,
I've tried that too. Unfortunately, I'm not sure how Oozie creates a JDBC URL for beeline as it fails before a MR job is created. I've tried to put some configuration properties in <configuration> tags but that didn't help either. In case of <credentials> for hive2 there I cannot put principal into property hive2.jdbc.url as it says: "JdbcUriParseException: Bad URL format: Multiple values for property principal". Can you please explain what exactly you mean by ' the auth path'.
With following workflow.xml it didn't work:
<workflow-app xmlns="uri:oozie:workflow:0.5" name="HIVE2 HTTP Kerberos Test">
<global>
<job-tracker>myrmaddress:8050</job-tracker>
<name-node>hdfs://mynnaddress:8020/</name-node>
</global>
<credentials>
<credential name="hive2creds" type="hive2">
<property>
<name>hive2.jdbc.url</name>
<value>jdbc:hive2://myhiveserver:10001/default;sasl.qop=auth-conf;transportMode=http;httpPath=cliservice;kerberoAuthType=fromSubject;auth=kerberos</value>
</property>
<property>
<name>hive2.server.principal</name>
<value>hive/myhiveserver@mydomain</value>
</property>
</credential>
</credentials>
<start to=" MyHiveAction"/>
<action cred="hive2creds" name=" MyHiveAction">
<hive2 xmlns="uri:oozie:hive2-action:0.1">
<jdbc-url>jdbc:hive2://myhiveserver:10001/default;principal=hive/myhiveserver@mydomain;sasl.qop=auth-conf;transportMode=http;httpPath=cliservice;kerberoAuthType=fromSubject;auth=kerberos</jdbc-url>
<script>script.hql</script>
</hive2>
<ok to="end"/>
<error to="fail"/>
</action>
<kill name="fail">
<message>Action failed, error
message[${wf:errorMessage(wf:lastErrorNode())}]
</message>
</kill>
<end name="end"/>
</workflow-app>
-----Original Message-----
From: Peter Cseh [mailto:gezapeti@cloudera.com]
Sent: Monday, August 29, 2016 17:56
To: user@oozie.apache.org
Subject: Re: Oozie Hive2 Action with Kerberos security and HS2 HTTP transport mode
Have you tried including the principal and the auth path <https://cwiki.apache.org/confluence/display/Hive/HiveServer2+Clients#HiveServer2Clients-UsingKerberoswithaPre-AuthenticatedSubject>
in the jdbc url?
Beeline needs that so it has to included in the jdbc-url field in the action too.
Gp
On Thu, Aug 25, 2016 at 5:14 PM, Jiri Kaplan <Ji...@software.dell.com>
wrote:
> Hi,
>
>
>
> I’d like to ask for a help with Oozie Hive2 action on HDP-2.3.4.0
> cluster with Oozie 4.2.0.2.3 installed and with enabled security over Kerberos.
> Oozie job always ends up with following exception: HiveSQLException:
> Delegation token only supported over kerberos authentication. We have
> HiveServer2 configured with hive.server2.transport.mode=http,
> hive.server2.thrift.http.path=cliservice and
> hive.server2.thrift.http.port=10001. I'm not sure if I do something
> wrong or if this configuration is even supported but when we switch
> back HS2 transport mode to binary it works. Any kind of help is welcome.
>
>
>
> Exception stack trace (from HS2 log):
>
> 2016-08-25 11:01:23,337 ERROR [HiveServer2-HttpHandler-Pool: Thread-38]:
> thrift.ThriftCLIService
> (ThriftCLIService.java:GetDelegationToken(237)) - Error obtaining
> delegation token
>
> org.apache.hive.service.cli.HiveSQLException: Delegation token only
> supported over kerberos authentication
>
> at org.apache.hive.service.auth.HiveAuthFactory.
> getDelegationToken(HiveAuthFactory.java:283)
>
> at org.apache.hive.service.cli.session.HiveSessionImplwithUGI.
> getDelegationToken(HiveSessionImplwithUGI.java:192)
>
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>
> at sun.reflect.NativeMethodAccessorImpl.invoke(
> NativeMethodAccessorImpl.java:62)
>
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(
> DelegatingMethodAccessorImpl.java:43)
>
> at java.lang.reflect.Method.invoke(Method.java:497)
>
> at org.apache.hive.service.cli.session.HiveSessionProxy.
> invoke(HiveSessionProxy.java:78)
>
> at org.apache.hive.service.cli.session.HiveSessionProxy.
> access$000(HiveSessionProxy.java:36)
>
> at org.apache.hive.service.cli.session.HiveSessionProxy$1.
> run(HiveSessionProxy.java:63)
>
> at java.security.AccessController.doPrivileged(Native Method)
>
> at javax.security.auth.Subject.doAs(Subject.java:422)
>
> at org.apache.hadoop.security.UserGroupInformation.doAs(
> UserGroupInformation.java:1657)
>
> at org.apache.hive.service.cli.session.HiveSessionProxy.
> invoke(HiveSessionProxy.java:59)
>
> at com.sun.proxy.$Proxy20.getDelegationToken(Unknown Source)
>
> at org.apache.hive.service.cli.CLIService.getDelegationToken(
> CLIService.java:484)
>
> at org.apache.hive.service.cli.thrift.ThriftCLIService.
> GetDelegationToken(ThriftCLIService.java:231)
>
> at org.apache.hive.service.cli.thrift.TCLIService$Processor$
> GetDelegationToken.getResult(TCLIService.java:1573)
>
> at org.apache.hive.service.cli.thrift.TCLIService$Processor$
> GetDelegationToken.getResult(TCLIService.java:1558)
>
> at org.apache.thrift.ProcessFunction.process(
> ProcessFunction.java:39)
>
> at org.apache.thrift.TBaseProcessor.process(
> TBaseProcessor.java:39)
>
> at org.apache.thrift.server.TServlet.doPost(TServlet.java:83)
>
> at org.apache.hive.service.cli.thrift.ThriftHttpServlet.
> doPost(ThriftHttpServlet.java:171)
>
>
>
> Here is my workflow.xml content:
>
> <workflow-app xmlns="uri:oozie:workflow:0.5" name="HIVE2 HTTP Kerberos
> Test">
>
> <global>
>
> <job-tracker>myrmaddress:8050</job-tracker>
>
> <name-node>hdfs://mynnaddress:8020/</name-node>
>
> </global>
>
> <credentials>
>
> <credential name="hive2creds" type="hive2">
>
> <property>
>
> <name>hive2.jdbc.url</name>
>
>
> <value>jdbc:hive2://myhiveserver:10001/;sasl.qop=
> auth-conf;transportMode=http;httpPath=cliservice</value>
>
> </property>
>
> <property>
>
> <name>hive2.server.principal</name>
>
> <value>hive/myhiveserver@mydomain</value>
>
> </property>
>
> </credential>
>
> </credentials>
>
> <start to="MyHiveAction"/>
>
> <action cred="hive2creds" name=" MyHiveAction ">
>
> <hive2 xmlns="uri:oozie:hive2-action:0.1">
>
> <jdbc-url> jdbc:hive2://myhiveserver:
> 10001/;sasl.qop=auth-conf;transportMode=http;httpPath=cliservice
> </jdbc-url>
>
> <script>script.hql</script>
>
> </hive2>
>
> <ok to="end"/>
>
> <error to="fail"/>
>
> </action>
>
> <kill name="fail">
>
> <message>Action failed, error
>
> message[${wf:errorMessage(wf:
> lastErrorNode())}]
>
> </message>
>
> </kill>
>
> <end name="end"/>
>
> </workflow-app>
>
>
>
> *Jiří Kaplan*
> Software Developer
>
> *Dell** | *R&D Database Management, EMEA
>
> [image: dell_software]
>
>
>
--
Peter Cseh
Software Engineer
<http://www.cloudera.com>
Re: Oozie Hive2 Action with Kerberos security and HS2 HTTP transport mode
Posted by Peter Cseh <ge...@cloudera.com>.
Have you tried including the principal and the auth path
<https://cwiki.apache.org/confluence/display/Hive/HiveServer2+Clients#HiveServer2Clients-UsingKerberoswithaPre-AuthenticatedSubject>
in the jdbc url?
Beeline needs that so it has to included in the jdbc-url field in the
action too.
Gp
On Thu, Aug 25, 2016 at 5:14 PM, Jiri Kaplan <Ji...@software.dell.com>
wrote:
> Hi,
>
>
>
> I’d like to ask for a help with Oozie Hive2 action on HDP-2.3.4.0 cluster
> with Oozie 4.2.0.2.3 installed and with enabled security over Kerberos.
> Oozie job always ends up with following exception: HiveSQLException:
> Delegation token only supported over kerberos authentication. We have
> HiveServer2 configured with hive.server2.transport.mode=http,
> hive.server2.thrift.http.path=cliservice and
> hive.server2.thrift.http.port=10001. I'm not sure if I do something wrong
> or if this configuration is even supported but when we switch back HS2
> transport mode to binary it works. Any kind of help is welcome.
>
>
>
> Exception stack trace (from HS2 log):
>
> 2016-08-25 11:01:23,337 ERROR [HiveServer2-HttpHandler-Pool: Thread-38]:
> thrift.ThriftCLIService (ThriftCLIService.java:GetDelegationToken(237)) -
> Error obtaining delegation token
>
> org.apache.hive.service.cli.HiveSQLException: Delegation token only
> supported over kerberos authentication
>
> at org.apache.hive.service.auth.HiveAuthFactory.
> getDelegationToken(HiveAuthFactory.java:283)
>
> at org.apache.hive.service.cli.session.HiveSessionImplwithUGI.
> getDelegationToken(HiveSessionImplwithUGI.java:192)
>
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>
> at sun.reflect.NativeMethodAccessorImpl.invoke(
> NativeMethodAccessorImpl.java:62)
>
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(
> DelegatingMethodAccessorImpl.java:43)
>
> at java.lang.reflect.Method.invoke(Method.java:497)
>
> at org.apache.hive.service.cli.session.HiveSessionProxy.
> invoke(HiveSessionProxy.java:78)
>
> at org.apache.hive.service.cli.session.HiveSessionProxy.
> access$000(HiveSessionProxy.java:36)
>
> at org.apache.hive.service.cli.session.HiveSessionProxy$1.
> run(HiveSessionProxy.java:63)
>
> at java.security.AccessController.doPrivileged(Native Method)
>
> at javax.security.auth.Subject.doAs(Subject.java:422)
>
> at org.apache.hadoop.security.UserGroupInformation.doAs(
> UserGroupInformation.java:1657)
>
> at org.apache.hive.service.cli.session.HiveSessionProxy.
> invoke(HiveSessionProxy.java:59)
>
> at com.sun.proxy.$Proxy20.getDelegationToken(Unknown Source)
>
> at org.apache.hive.service.cli.CLIService.getDelegationToken(
> CLIService.java:484)
>
> at org.apache.hive.service.cli.thrift.ThriftCLIService.
> GetDelegationToken(ThriftCLIService.java:231)
>
> at org.apache.hive.service.cli.thrift.TCLIService$Processor$
> GetDelegationToken.getResult(TCLIService.java:1573)
>
> at org.apache.hive.service.cli.thrift.TCLIService$Processor$
> GetDelegationToken.getResult(TCLIService.java:1558)
>
> at org.apache.thrift.ProcessFunction.process(
> ProcessFunction.java:39)
>
> at org.apache.thrift.TBaseProcessor.process(
> TBaseProcessor.java:39)
>
> at org.apache.thrift.server.TServlet.doPost(TServlet.java:83)
>
> at org.apache.hive.service.cli.thrift.ThriftHttpServlet.
> doPost(ThriftHttpServlet.java:171)
>
>
>
> Here is my workflow.xml content:
>
> <workflow-app xmlns="uri:oozie:workflow:0.5" name="HIVE2 HTTP Kerberos
> Test">
>
> <global>
>
> <job-tracker>myrmaddress:8050</job-tracker>
>
> <name-node>hdfs://mynnaddress:8020/</name-node>
>
> </global>
>
> <credentials>
>
> <credential name="hive2creds" type="hive2">
>
> <property>
>
> <name>hive2.jdbc.url</name>
>
> <value>jdbc:hive2://myhiveserver:10001/;sasl.qop=
> auth-conf;transportMode=http;httpPath=cliservice</value>
>
> </property>
>
> <property>
>
> <name>hive2.server.principal</name>
>
> <value>hive/myhiveserver@mydomain</value>
>
> </property>
>
> </credential>
>
> </credentials>
>
> <start to="MyHiveAction"/>
>
> <action cred="hive2creds" name=" MyHiveAction ">
>
> <hive2 xmlns="uri:oozie:hive2-action:0.1">
>
> <jdbc-url> jdbc:hive2://myhiveserver:
> 10001/;sasl.qop=auth-conf;transportMode=http;httpPath=cliservice
> </jdbc-url>
>
> <script>script.hql</script>
>
> </hive2>
>
> <ok to="end"/>
>
> <error to="fail"/>
>
> </action>
>
> <kill name="fail">
>
> <message>Action failed, error
>
> message[${wf:errorMessage(wf:
> lastErrorNode())}]
>
> </message>
>
> </kill>
>
> <end name="end"/>
>
> </workflow-app>
>
>
>
> *Jiří Kaplan*
> Software Developer
>
> *Dell** | *R&D Database Management, EMEA
>
> [image: dell_software]
>
>
>
--
Peter Cseh
Software Engineer
<http://www.cloudera.com>