You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by Zoltán Zvara <zo...@gmail.com> on 2015/04/09 13:57:13 UTC
Connect to remote YARN cluster
I'm trying to debug Spark in yarn-client mode. On my local, single node
cluster everything works fine, but the remote YARN resource manager throws
away my request because of authentication error. I'm running IntelliJ 14 on
Ubuntu and the driver tries to connect to YARN with my local user name. How
can I force IntelliJ to run my code with a different user? Or how can I set
up the connection to YARN RM with auth data?
Thanks!
Zvara Zoltán
mail, hangout, skype: zoltan.zvara@gmail.com
mobile, viber: +36203129543
bank: 10918001-00000021-50480008
address: Hungary, 2475 Kápolnásnyék, Kossuth 6/a
elte: HSKSJZ (ZVZOAAI.ELTE)
Re: Connect to remote YARN cluster
Posted by Zoltán Zvara <zo...@gmail.com>.
Wow, very useful comments! Thanks! Now, just HDFS permission problems.
Zvara Zoltán
mail, hangout, skype: zoltan.zvara@gmail.com
mobile, viber: +36203129543
bank: 10918001-00000021-50480008
address: Hungary, 2475 Kápolnásnyék, Kossuth 6/a
elte: HSKSJZ (ZVZOAAI.ELTE)
2015-04-09 21:48 GMT+02:00 Steve Loughran <st...@hortonworks.com>:
>
> > On 9 Apr 2015, at 17:42, Marcelo Vanzin <va...@cloudera.com> wrote:
> >
> > If YARN is authenticating users it's probably running on kerberos, so
> > you need to log in with your kerberos credentials (kinit) before
> > submitting an application.
>
> also: make sure that you have the full JCE and not the crippled crypto;
> every time you upgrade the JDK you are likely to have to re-install it.
> Java gives no useful error messages on this or any other Kerberos problem
>
Re: Connect to remote YARN cluster
Posted by Steve Loughran <st...@hortonworks.com>.
> On 9 Apr 2015, at 17:42, Marcelo Vanzin <va...@cloudera.com> wrote:
>
> If YARN is authenticating users it's probably running on kerberos, so
> you need to log in with your kerberos credentials (kinit) before
> submitting an application.
also: make sure that you have the full JCE and not the crippled crypto; every time you upgrade the JDK you are likely to have to re-install it. Java gives no useful error messages on this or any other Kerberos problem
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org
Re: Connect to remote YARN cluster
Posted by Marcelo Vanzin <va...@cloudera.com>.
If YARN is authenticating users it's probably running on kerberos, so
you need to log in with your kerberos credentials (kinit) before
submitting an application.
On Thu, Apr 9, 2015 at 4:57 AM, Zoltán Zvara <zo...@gmail.com> wrote:
> I'm trying to debug Spark in yarn-client mode. On my local, single node
> cluster everything works fine, but the remote YARN resource manager throws
> away my request because of authentication error. I'm running IntelliJ 14 on
> Ubuntu and the driver tries to connect to YARN with my local user name. How
> can I force IntelliJ to run my code with a different user? Or how can I set
> up the connection to YARN RM with auth data?
>
> Thanks!
>
> Zvara Zoltán
>
>
>
> mail, hangout, skype: zoltan.zvara@gmail.com
>
> mobile, viber: +36203129543
>
> bank: 10918001-00000021-50480008
>
> address: Hungary, 2475 Kápolnásnyék, Kossuth 6/a
>
> elte: HSKSJZ (ZVZOAAI.ELTE)
--
Marcelo
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org