You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@zeppelin.apache.org by Michał Kabocik <mi...@gmail.com> on 2017/01/27 14:59:42 UTC
Connection refused when trying to run livy.spark interpterer in
kerberized HDP
Dears,
I'm running Zeppelin 0.6.2 on HDP 2.5.3. My Zeppelin has also configured
shiro to authenticate users from AD.
Now I'm trying to configure livy.spark interpreter with user impersonation.
I used livy 0.2 and 0.3 but in both cases I cannot start livy.spark
interpterer.
Here's the config of livy 0.3:
livy.impersonation.enabled true
livy.server.csrf_protection.enabled true
livy.server.port 8998
livy.server.session.timeout 3600000
livy.server.auth.type = kerberos
livy.server.launch.kerberos.principal = livy/hostname@DOMAIN.COM
livy.server.launch.kerberos.keytab = /etc/security/keytabs/livy.
service.keytab
livy.server.auth.kerberos.principal = HTTP/hostname@DOMAIN.COM
livy.server.auth.kerberos.keytab = /etc/security/keytabs/spnego.
service.keytab
livy.server.access_control.enabled true
livy.server.access_control.users livy,zeppelin
Here's the livy.spark interpreter config on Zeppelin:
zeppelin.interpreter.localRepo /usr/hdp/current/zeppelin-
server/local-repo/2C42AQ9SU
zeppelin.livy.concurrentSQL false
zeppelin.livy.create.session.retries 120
zeppelin.livy.keytab /etc/security/keytabs/zeppelin.service.keytab
zeppelin.livy.principal zeppelin_acc/hostname@DOMAIN.COM
zeppelin.livy.spark.sql.maxResult 1000
zeppelin.livy.url http://hostname:8998
When running livy.spark with user impersonation enabled on Livy and
disabled on Zeppelin livy.spark interpreter, I'm getting response , for
example:
%livy.spark
sc.version
res0: String = 1.6.2
But when I enable user impersonation on livy.spark interpterer, I'm getting
an error:
INFO [2017-01-27 14:14:40,794] ({pool-1-thread-12}
SchedulerFactory.java[jobStarted]:131) - Job
paragraph_1484571361183_-784415977 started by scheduler
org.apache.zeppelin.interpreter.remote.RemoteInterpretershared_session1803759183
INFO [2017-01-27 14:14:40,796] ({pool-1-thread-12}
Paragraph.java[jobRun]:254) - run paragraph 20170116-135601_2032386468
using livy.spark
org.apache.zeppelin.interpreter.LazyOpenInterpreter@7d89c50f
ERROR [2017-01-27 14:14:40,797] ({pool-1-thread-12} Job.java[run]:189) -
Job failed
org.apache.zeppelin.interpreter.InterpreterException:
org.apache.zeppelin.interpreter.InterpreterException:
org.apache.thrift.transport.TTransportException: java.net.ConnectException:
Connection refused (Connection refused)
at
org.apache.zeppelin.interpreter.remote.RemoteInterpreter.init(RemoteInterpreter.java:175)
at
org.apache.zeppelin.interpreter.remote.RemoteInterpreter.getFormType(RemoteInterpreter.java:338)
at
org.apache.zeppelin.interpreter.LazyOpenInterpreter.getFormType(LazyOpenInterpreter.java:105)
at org.apache.zeppelin.notebook.Paragraph.jobRun(Paragraph.java:262)
at org.apache.zeppelin.scheduler.Job.run(Job.java:176)
at
org.apache.zeppelin.scheduler.RemoteScheduler$JobRunner.run(RemoteScheduler.java:328)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.zeppelin.interpreter.InterpreterException:
org.apache.thrift.transport.TTransportException: java.net.ConnectException:
Connection refused (Connection refused)
at
org.apache.zeppelin.interpreter.remote.ClientFactory.create(ClientFactory.java:53)
at
org.apache.zeppelin.interpreter.remote.ClientFactory.create(ClientFactory.java:37)
at
org.apache.commons.pool2.BasePooledObjectFactory.makeObject(BasePooledObjectFactory.java:60)
at
org.apache.commons.pool2.impl.GenericObjectPool.create(GenericObjectPool.java:861)
at
org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:435)
at
org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:363)
at
org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.getClient(RemoteInterpreterProcess.java:189)
at
org.apache.zeppelin.interpreter.remote.RemoteInterpreter.init(RemoteInterpreter.java:173)
... 12 more
Caused by: org.apache.thrift.transport.TTransportException:
java.net.ConnectException: Connection refused (Connection refused)
at org.apache.thrift.transport.TSocket.open(TSocket.java:187)
at
org.apache.zeppelin.interpreter.remote.ClientFactory.create(ClientFactory.java:51)
... 19 more
Caused by: java.net.ConnectException: Connection refused (Connection
refused)
at java.net.PlainSocketImpl.socketConnect(Native Method)
at
java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at
java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at
java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at org.apache.thrift.transport.TSocket.open(TSocket.java:182)
... 20 more
Any support will be highly appreciated.
Kind regards,
--
Michał Kabocik
Posted by Michał Kabocik <mi...@gmail.com>.
Hello,
Current configuration looks like that:
livy.conf:
livy.impersonation.enabled = true
livy.server.csrf_protection.enabled = true
livy.server.port = 8998
livy.server.session.timeout = 3600000
livy.server.auth.type = kerberos
livy.server.launch.kerberos.principal = livy/hostname@domain.com
livy.server.launch.kerberos.keytab = /etc/security/keytabs/livy.service.keytab
livy.server.auth.kerberos.principal = HTTP/hostname@domain.com
livy.server.auth.kerberos.keytab = /etc/security/keytabs/spnego.service.keytab
livy.server.access_control.enabled true
livy.server.access_control.users livy,zeppelin
livy.superusers = zeppelin
livy.server.session.factory = yarn
livy interpreter on zeppelin:
livy.impersonation.enabled true
livy.spark.master yarn-client
livy.superusers zeppelin
zeppelin.interpreter.localRepo /usr/hdp/current/zeppelin-server/local-repo/2C42AQ9SU
zeppelin.livy.keytab /etc/security/keytabs/zeppelin.service.keytab
zeppelin.livy.principal zeppelin/hostname@domain.com
zeppelin.livy.url http://hostname:8998
And when I try to execute:
%livy.spark
sc.textFile("hdfs:///user/my_user/file.txt")
file.take(1)
I get:
Error running rest call; nested exception is org.springframework.web.client.HttpClientErrorException: 401 User not authorised to use Livy.
When I look into zeppelin-interpterer-livy log file I see:
- http-outgoing-1 >> "}, "proxyUser": "MY_USER@DOMAIN.COM"}"
- http-outgoing-1 << "HTTP/1.1 401 User not authorised to use Livy.[\r][\n]"
MY_USER@DOMAIN.COM is my user from AD which I'm logged into Zeppelin.
What am I doing wrong?
I'll appreciate your help.
Micha\u0142
Re:
Posted by Jeff Zhang <zj...@gmail.com>.
If you enable impersonation in livy server side, the spark application
should be launched as the user who login zeppelin. Did you enable
impersonation in livy server ?
Michał Kabocik <mi...@gmail.com>于2017年1月30日周一 下午7:01写道：
> Thank you for the reply.
>
> I know that impersonation is done by livy, but when I have user
> impersonation disabled on livy.spark interpreter, then livy impersonates
> zeppelin user, not my user. On ranger audit and in yarn history manager I
> see zeppelin via livy-session.
> What I want to achieve is to have my user (logged in to zeppelin after AD
> authentication) impersonated. I need this to be able to apply ranger data
> access policies based on the users/groups from AD.
> Is there a way to implement this scenario?
>
> I'll appreciate any suggestions.
> Kind regards,
> Michał
>
>
Posted by Michał Kabocik <mi...@gmail.com>.
Thank you for the reply.
I know that impersonation is done by livy, but when I have user impersonation disabled on livy.spark interpreter, then livy impersonates zeppelin user, not my user. On ranger audit and in yarn history manager I see zeppelin via livy-session.
What I want to achieve is to have my user (logged in to zeppelin after AD authentication) impersonated. I need this to be able to apply ranger data access policies based on the users/groups from AD.
Is there a way to implement this scenario?
I'll appreciate any suggestions.
Kind regards,
Micha\u0142
Re: Connection refused when trying to run livy.spark interpterer in
kerberized HDP
Posted by "Jianfeng (Jeff) Zhang" <jz...@hortonworks.com>.
>>> But when I enable user impersonation on ivy.spark interpterer
Do you mean you enable the option of user impersonation on livy interpreter setting of zeppelin ? You don't need to do that as impersonation is done by livy, the option there is a little confusing, it should not be displayed.
Best Regard,
Jeff Zhang
From: Michał Kabocik <mi...@gmail.com>>
Reply-To: "users@zeppelin.apache.org<ma...@zeppelin.apache.org>" <us...@zeppelin.apache.org>>
Date: Friday, January 27, 2017 at 10:59 PM
To: "users@zeppelin.apache.org<ma...@zeppelin.apache.org>" <us...@zeppelin.apache.org>>
Subject: Connection refused when trying to run livy.spark interpterer in kerberized HDP
But when I enable user impersonation on livy.spark interpterer