You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@livy.apache.org by "Harsch, Tim" <Ti...@Teradata.com> on 2018/07/10 15:52:51 UTC
user impersonation in Livy not working
Hi,
I have been unable to get proxyUsers to work with Livy. I seem to have Kerberos configured correctly, as evidenced by the last beeline command. When I start a livy session the server log will usually say something like: “owner: kylo, proxyUser: Some(dladmin)” so it seems the server should know what to do. But no matter what I try I always get “org.apache.spark.sql.AnalysisException: Table not found”
# Ambari config
[/var/folders/_n/k2nq_2fx5t5__fcxqdl2tc6j_p8_px/T/com.microsoft.Outlook/WebArchiveCopyPasteTempFiles/p1426]
# Livy Configuration
livy.impersonation.enabled = true
livy.server.auth.type = kerberos
livy.server.launch.kerberos.principal = kylo/sandbox.kylo.io@KYLO
livy.server.launch.kerberos.keytab = /etc/security/keytabs/kylo.keytab
livy.server.auth.kerberos.principal = HTTP/sandbox.kylo.io@KYLO
livy.server.auth.kerberos.keytab = /etc/security/keytabs/spnego.service.keytab
livy.server.access_control.enabled = true
livy.server.access_control.users = kylo,dladmin
livy.server.access_control.modify-users = kylo
livy.superusers = kylo
# Curl connect
kinit -kt /etc/security/keytabs/kylo.keytab kylo/sandbox.kylo.io@KYLO
klist
curl --negotiate -u : -s -X POST --data '{"kind": "spark", "proxyUser": "dladmin"}' -H "Content-Type: application/json" sandbox.kylo.io:8998/sessions
curl --negotiate -u : -s -X GET sandbox.kylo.io:8998/sessions
* NOTE: verify proxyUser="dladmin" in response. it will be null if impersonation is not enabled
# submit this code to Livy:
sqlContext.sql("select * from default.d4").show()
# Response:
{
"code": "sqlContext.sql(\"select * from default.d4\").show()",
"id": 0,
"output": {
"ename": "Error",
"evalue": "org.apache.spark.sql.AnalysisException: Table not found: `default`.`d4`;",
....
# Can "dladmin" see table data?
[root@sandbox more]# beeline -u "jdbc:hive2://localhost:10000/;principal=hive/sandbox.kylo.io@KYLO;;hive.server2.proxy.user=dladmin"
Connecting to jdbc:hive2://localhost:10000/;principal=hive/sandbox.kylo.io@KYLO;;hive.server2.proxy.user=dladmin
Connected to: Apache Hive (version 1.2.1000.2.5.6.0-40)
Driver: Hive JDBC (version 1.2.1000.2.5.6.0-40)
Transaction isolation: TRANSACTION_REPEATABLE_READ
Beeline version 1.2.1000.2.5.6.0-40 by Apache Hive
0: jdbc:hive2://localhost:10000/> select * from default.d4;
+--------+--+
| d4.d4 |
+--------+--+
| d4 |
+--------+--+
1 row selected (0.132 seconds)
Re: user impersonation in Livy not working
Posted by "Harsch, Tim" <Ti...@Teradata.com>.
Thanks Jeff,
Getting back to this now. I will try your suggestion when I get my environment running again. I did run into something today that I missed earlier and now I have some questions:
I noticed a setting in livy-client.conf.template for livy.rsc.proxy-user::
# The user that should be impersonated when requesting a Livy session
# livy.rsc.proxy-user =
What effect would that property have when I'm doing user impersonation? Is it necessary? Also, I see this at the top of livy-client.conf.template. Does that mean I should put the file in the 'conf' folder or some other folder accessible to the spark executors, like repl_2.10-jars ?
# Configurations for a Livy Client, any configurations set here will override any
# livy or spark-default configurations.
#
# Before a Livy Client is able to load these configurations the folder containing
# this file must be added to the application classpath
#
________________________________
From: Jeff Zhang <zj...@gmail.com>
Sent: Tuesday, July 10, 2018 5:27:52 PM
To: user@livy.incubator.apache.org
Subject: Re: user impersonation in Livy not working
Make sure you enable hiveContext. set livy.repl.enable-hive-context to true in livy.conf
Harsch, Tim <Ti...@teradata.com>>于2018年7月10日周二 下午11:53写道：
Hi,
I have been unable to get proxyUsers to work with Livy. I seem to have Kerberos configured correctly, as evidenced by the last beeline command. When I start a livy session the server log will usually say something like: “owner: kylo, proxyUser: Some(dladmin)” so it seems the server should know what to do. But no matter what I try I always get “org.apache.spark.sql.AnalysisException: Table not found”
# Ambari config
[/var/folders/_n/k2nq_2fx5t5__fcxqdl2tc6j_p8_px/T/com.microsoft.Outlook/WebArchiveCopyPasteTempFiles/p1426]
# Livy Configuration
livy.impersonation.enabled = true
livy.server.auth.type = kerberos
livy.server.launch.kerberos.principal = kylo/sandbox.kylo.io@KYLO
livy.server.launch.kerberos.keytab = /etc/security/keytabs/kylo.keytab
livy.server.auth.kerberos.principal = HTTP/sandbox.kylo.io@KYLO
livy.server.auth.kerberos.keytab = /etc/security/keytabs/spnego.service.keytab
livy.server.access_control.enabled = true
livy.server.access_control.users = kylo,dladmin
livy.server.access_control.modify-users = kylo
livy.superusers = kylo
# Curl connect
kinit -kt /etc/security/keytabs/kylo.keytab kylo/sandbox.kylo.io@KYLO
klist
curl --negotiate -u : -s -X POST --data '{"kind": "spark", "proxyUser": "dladmin"}' -H "Content-Type: application/json" sandbox.kylo.io:8998/sessions<http://sandbox.kylo.io:8998/sessions>
curl --negotiate -u : -s -X GET sandbox.kylo.io:8998/sessions<http://sandbox.kylo.io:8998/sessions>
* NOTE: verify proxyUser="dladmin" in response. it will be null if impersonation is not enabled
# submit this code to Livy:
sqlContext.sql("select * from default.d4").show()
# Response:
{
"code": "sqlContext.sql(\"select * from default.d4\").show()",
"id": 0,
"output": {
"ename": "Error",
"evalue": "org.apache.spark.sql.AnalysisException: Table not found: `default`.`d4`;",
....
# Can "dladmin" see table data?
[root@sandbox more]# beeline -u "jdbc:hive2://localhost:10000/;principal=hive/sandbox.kylo.io@KYLO;;hive.server2.proxy.user=dladmin"
Connecting to jdbc:hive2://localhost:10000/;principal=hive/sandbox.kylo.io@KYLO;;hive.server2.proxy.user=dladmin
Connected to: Apache Hive (version 1.2.1000.2.5.6.0-40)
Driver: Hive JDBC (version 1.2.1000.2.5.6.0-40)
Transaction isolation: TRANSACTION_REPEATABLE_READ
Beeline version 1.2.1000.2.5.6.0-40 by Apache Hive
0: jdbc:hive2://localhost:10000/> select * from default.d4;
+--------+--+
| d4.d4 |
+--------+--+
| d4 |
+--------+--+
1 row selected (0.132 seconds)
Re: user impersonation in Livy not working
Posted by Jeff Zhang <zj...@gmail.com>.
Make sure you enable hiveContext. set livy.repl.enable-hive-context to true
in livy.conf
Harsch, Tim <Ti...@teradata.com>于2018年7月10日周二 下午11:53写道：
> Hi,
>
> I have been unable to get proxyUsers to work with Livy. I seem to have
> Kerberos configured correctly, as evidenced by the last beeline command.
> When I start a livy session the server log will usually say something like:
> “owner: kylo, proxyUser: Some(dladmin)” so it seems the server should know
> what to do. But no matter what I try I always get
> “org.apache.spark.sql.AnalysisException: Table not found”
>
>
>
> # Ambari config
>
> [image:
> /var/folders/_n/k2nq_2fx5t5__fcxqdl2tc6j_p8_px/T/com.microsoft.Outlook/WebArchiveCopyPasteTempFiles/p1426]
>
>
>
> # Livy Configuration
>
> livy.impersonation.enabled = true
>
> livy.server.auth.type = kerberos
>
> livy.server.launch.kerberos.principal = kylo/sandbox.kylo.io@KYLO
>
> livy.server.launch.kerberos.keytab = /etc/security/keytabs/kylo.keytab
>
> livy.server.auth.kerberos.principal = HTTP/sandbox.kylo.io@KYLO
>
> livy.server.auth.kerberos.keytab =
> /etc/security/keytabs/spnego.service.keytab
>
> livy.server.access_control.enabled = true
>
> livy.server.access_control.users = kylo,dladmin
>
> livy.server.access_control.modify-users = kylo
>
> livy.superusers = kylo
>
>
>
> # Curl connect
>
> kinit -kt /etc/security/keytabs/kylo.keytab kylo/sandbox.kylo.io@KYLO
>
> klist
>
> curl --negotiate -u : -s -X POST --data '{"kind": "spark", "proxyUser":
> "dladmin"}' -H "Content-Type: application/json"
> sandbox.kylo.io:8998/sessions
>
> curl --negotiate -u : -s -X GET sandbox.kylo.io:8998/sessions
>
> * NOTE: verify proxyUser="dladmin" in response. it will be null if
> impersonation is not enabled
>
>
>
> # submit this code to Livy:
>
> sqlContext.sql("select * from default.d4").show()
>
>
>
> # Response:
>
> {
>
> "code": "sqlContext.sql(\"select * from default.d4\").show()",
>
> "id": 0,
>
> "output": {
>
> "ename": "Error",
>
> "evalue": "org.apache.spark.sql.AnalysisException: Table not found:
> `default`.`d4`;",
>
> ....
>
>
>
> # Can "dladmin" see table data?
>
> [root@sandbox more]# beeline -u
> "jdbc:hive2://localhost:10000/;principal=hive/sandbox.kylo.io@KYLO
> ;;hive.server2.proxy.user=dladmin"
>
> Connecting to
> jdbc:hive2://localhost:10000/;principal=hive/sandbox.kylo.io@KYLO
> ;;hive.server2.proxy.user=dladmin
>
> Connected to: Apache Hive (version 1.2.1000.2.5.6.0-40)
>
> Driver: Hive JDBC (version 1.2.1000.2.5.6.0-40)
>
> Transaction isolation: TRANSACTION_REPEATABLE_READ
>
> Beeline version 1.2.1000.2.5.6.0-40 by Apache Hive
>
> 0: jdbc:hive2://localhost:10000/> select * from default.d4;
>
> +--------+--+
>
> | d4.d4 |
>
> +--------+--+
>
> | d4 |
>
> +--------+--+
>
> 1 row selected (0.132 seconds)
>
>
>
>
>
>
>
>
>