You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@livy.apache.org by srungarapu vamsi <sr...@gmail.com> on 2018/08/20 10:35:35 UTC
Impersonation Error
Hi,
We have a secure kerberized CDH cluster.
I am trying to test impersonation through livy rest sever.
I have created the proxy user which can impersonate other users in the
while running the spark jobs.
Here is my core-site.xml for the proxy user "my_proxy_user":
<property>
<name>hadoop.proxyuser.my_proxy_user.hosts</name>
<value>*</value>
</property>
<property>
<name>hadoop.proxyuser. my_proxy_user.groups</name>
<value> my_proxy_user_grp</value>
</property>
<property>
<name>hadoop.proxyuser. my_proxy_user.users</name>
<value>*</value>
</property>
</configuration>
With these configurations, whenever i am trying to create a session or
batch, i am getting the following error.
org.apache.hadoop.yarn.exceptions.YarnException: User my_proxy_user
does not have privilage to see this attempt
appattempt_1533784624146_0323_-000001
at org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.getApplicationAttemptReport(ClientRMService.java:375)
at org.apache.hadoop.yarn.api.impl.pb.service.ApplicationClientProtocolPBServiceImpl.getApplicationAttemptReport(ApplicationClientProtocolPBServiceImpl.java:355)
at org.apache.hadoop.yarn.proto.ApplicationClientProtocol$ApplicationClientProtocolService$2.callBlockingMethod(ApplicationClientProtocol.java:425)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2217)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2213)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1917)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2211)
I am able to run the spark jobs by passsing the --proxyUser flag.
Also, i am successfully able to run yarn jobs with HADOOP_PROXY_USER.
But, on trying to create a session or batch, i am not getting in stderr of
the session/batch logs. But "Yarn Diagnostics" show the above error.
Can you please help me in solving this issue.
--
/Vamsi
Re: Impersonation Error
Posted by srungarapu vamsi <sr...@gmail.com>.
Hi Tim,
Yes, I have referred the thread on user impersonation from Tuesday, July
10, 2018 (
https://www.mail-archive.com/user@livy.incubator.apache.org/msg00294.html )
Yes, I have enabled hive context.
Please find below configurations set in livy.conf .
livy.keystore = /path/to/server.jks
livy.keystore.password = password
livy.server.port = 8998
livy.spark.master = yarn
livy.spark.deploy-mode = cluster
livy.impersonation.enabled = true
livy.repl.enable-hive-context = true
livy.server.access-control.enabled = false
livy.server.auth.type = kerberos
livy.server.auth.kerberos.keytab=/path/to/http/keytab
livy.server.auth.kerberos.principal=HTTP/<principal>
livy.server.launch.kerberos.keytab= /path/to/my_proxy_user/keytab
livy.server.launch.kerberos.principal=my_proxy_user/<principal>
livy.superusers = my_proxy_user
Whenever I try to create a batch, the batch is successfully executing.
However, yarn diagnostics is throwing the above YarnException.
When I try to create a session, the session is finally ending up in a dead
state. with the above-mentioned Yarn Exception in the Yarn Diagnostics.
Thanks,
Vamsi
On Tue, 21 Aug 2018 at 00:49, Harsch, Tim <Ti...@teradata.com> wrote:
> Vamsi,
>
> It may be helpful if you read the thread on user impersonation from
> Tuesday, July 10, 2018.
>
>
>
> Do you have hive sql enabled? What related configurations do you have set
> in livy.conf ? Are you using kerberos as well?
>
>
>
>
>
> *From: *srungarapu vamsi <sr...@gmail.com>
> *Reply-To: *"user@livy.incubator.apache.org" <
> user@livy.incubator.apache.org>
> *Date: *Monday, August 20, 2018 at 3:36 AM
> *To: *"user@livy.incubator.apache.org" <us...@livy.incubator.apache.org>
> *Subject: *Impersonation Error
>
>
>
> [External Email]
> ------------------------------
>
> Hi,
>
>
>
> We have a secure kerberized CDH cluster.
>
> I am trying to test impersonation through livy rest sever.
>
> I have created the proxy user which can impersonate other users in the
> while running the spark jobs.
>
> Here is my core-site.xml for the proxy user "my_proxy_user":
>
>
>
> <property>
>
> <name>hadoop.proxyuser.my_proxy_user.hosts</name>
>
> <value>*</value>
>
> </property>
>
> <property>
>
> <name>hadoop.proxyuser. my_proxy_user.groups</name>
>
> <value> my_proxy_user_grp</value>
>
> </property>
>
> <property>
>
> <name>hadoop.proxyuser. my_proxy_user.users</name>
>
> <value>*</value>
>
> </property>
>
> </configuration>
>
>
>
> With these configurations, whenever i am trying to create a session or
> batch, i am getting the following error.
>
>
>
> org.apache.hadoop.yarn.exceptions.YarnException: User my_proxy_user does not have privilage to see this attempt appattempt_1533784624146_0323_-000001
>
> at org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.getApplicationAttemptReport(ClientRMService.java:375)
>
> at org.apache.hadoop.yarn.api.impl.pb.service.ApplicationClientProtocolPBServiceImpl.getApplicationAttemptReport(ApplicationClientProtocolPBServiceImpl.java:355)
>
> at org.apache.hadoop.yarn.proto.ApplicationClientProtocol$ApplicationClientProtocolService$2.callBlockingMethod(ApplicationClientProtocol.java:425)
>
> at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617)
>
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073)
>
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2217)
>
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2213)
>
> at java.security.AccessController.doPrivileged(Native Method)
>
> at javax.security.auth.Subject.doAs(Subject.java:422)
>
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1917)
>
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2211)
>
>
>
> I am able to run the spark jobs by passsing the --proxyUser flag.
>
> Also, i am successfully able to run yarn jobs with HADOOP_PROXY_USER.
>
>
>
> But, on trying to create a session or batch, i am not getting in stderr of
> the session/batch logs. But "Yarn Diagnostics" show the above error.
>
>
>
> Can you please help me in solving this issue.
>
> --
>
> /Vamsi
>
--
/Vamsi
Re: Impersonation Error
Posted by "Harsch, Tim" <Ti...@Teradata.com>.
Vamsi,
It may be helpful if you read the thread on user impersonation from Tuesday, July 10, 2018.
Do you have hive sql enabled? What related configurations do you have set in livy.conf ? Are you using kerberos as well?
From: srungarapu vamsi <sr...@gmail.com>
Reply-To: "user@livy.incubator.apache.org" <us...@livy.incubator.apache.org>
Date: Monday, August 20, 2018 at 3:36 AM
To: "user@livy.incubator.apache.org" <us...@livy.incubator.apache.org>
Subject: Impersonation Error
[External Email]
________________________________
Hi,
We have a secure kerberized CDH cluster.
I am trying to test impersonation through livy rest sever.
I have created the proxy user which can impersonate other users in the while running the spark jobs.
Here is my core-site.xml for the proxy user "my_proxy_user":
<property>
<name>hadoop.proxyuser.my_proxy_user.hosts</name>
<value>*</value>
</property>
<property>
<name>hadoop.proxyuser. my_proxy_user.groups</name>
<value> my_proxy_user_grp</value>
</property>
<property>
<name>hadoop.proxyuser. my_proxy_user.users</name>
<value>*</value>
</property>
</configuration>
With these configurations, whenever i am trying to create a session or batch, i am getting the following error.
org.apache.hadoop.yarn.exceptions.YarnException: User my_proxy_user does not have privilage to see this attempt appattempt_1533784624146_0323_-000001
at org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.getApplicationAttemptReport(ClientRMService.java:375)
at org.apache.hadoop.yarn.api.impl.pb.service.ApplicationClientProtocolPBServiceImpl.getApplicationAttemptReport(ApplicationClientProtocolPBServiceImpl.java:355)
at org.apache.hadoop.yarn.proto.ApplicationClientProtocol$ApplicationClientProtocolService$2.callBlockingMethod(ApplicationClientProtocol.java:425)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2217)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2213)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1917)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2211)
I am able to run the spark jobs by passsing the --proxyUser flag.
Also, i am successfully able to run yarn jobs with HADOOP_PROXY_USER.
But, on trying to create a session or batch, i am not getting in stderr of the session/batch logs. But "Yarn Diagnostics" show the above error.
Can you please help me in solving this issue.
--
/Vamsi