You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Zhanfeng Huo <hu...@gmail.com> on 2014/09/03 20:00:13 UTC
How can I start history-server with kerberos HDFS ?
Hi,
I have seted properties in conf/spark-defaults.conf and start with command "./sbin/start-history-server.sh /tmp/spark-events". It get errors and seems that the properties in spark-defaults.conf file doesn't effect. How can I solve this problem(Enable properties in spark-defaults.conf when start spark history-server) ?
14/09/04 01:44:04 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
14/09/04 01:44:05 WARN UserGroupInformation: PriviledgedActionException as:root (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
14/09/04 01:44:05 WARN Client: Exception encountered while connecting to the server : javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
14/09/04 01:44:05 WARN UserGroupInformation: PriviledgedActionException as:root (auth:KERBEROS) cause:java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
Exception in thread "main" java.io.IOException: Failed on local exception: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]; Host Details :
#history-server
spark.history.kerberos.enabled true
park.history.kerberos.principal test/spark@test
spark.history.kerberos.keytab /home/test/test_spark.keytab
spark.eventLog.enabled true
Zhanfeng Huo
Re: Re: How can I start history-server with kerberos HDFS ?
Posted by Zhanfeng Huo <hu...@gmail.com>.
Thanks for your help.
It works after setting SPARK_HISTORY_OPTS.
Zhanfeng Huo
From: Andrew Or
Date: 2014-09-04 07:52
To: Marcelo Vanzin
CC: Zhanfeng Huo; user
Subject: Re: How can I start history-server with kerberos HDFS ?
Hi Zhanfeng,
You will need to set these through SPARK_HISTORY_OPTS in conf/spark-env.sh. This is documented here: http://spark.apache.org/docs/latest/monitoring.html.
Let me know if you have it working,
-Andrew
2014-09-03 11:14 GMT-07:00 Marcelo Vanzin <va...@cloudera.com>:
The history server (and other Spark daemons) do not read
spark-defaults.conf. There's a bug open to implement that
(SPARK-2098), and an open PR to fix it, but it's still not in Spark.
On Wed, Sep 3, 2014 at 11:00 AM, Zhanfeng Huo <hu...@gmail.com> wrote:
> Hi,
>
> I have seted properties in conf/spark-defaults.conf and start with
> command "./sbin/start-history-server.sh /tmp/spark-events". It get errors
> and seems that the properties in spark-defaults.conf file doesn't effect.
> How can I solve this problem(Enable properties in spark-defaults.conf when
> start spark history-server) ?
>
> 14/09/04 01:44:04 WARN NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> 14/09/04 01:44:05 WARN UserGroupInformation: PriviledgedActionException
> as:root (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS
> initiate failed [Caused by GSSException: No valid credentials provided
> (Mechanism level: Failed to find any Kerberos tgt)]
> 14/09/04 01:44:05 WARN Client: Exception encountered while connecting to the
> server : javax.security.sasl.SaslException: GSS initiate failed [Caused by
> GSSException: No valid credentials provided (Mechanism level: Failed to find
> any Kerberos tgt)]
> 14/09/04 01:44:05 WARN UserGroupInformation: PriviledgedActionException
> as:root (auth:KERBEROS) cause:java.io.IOException:
> javax.security.sasl.SaslException: GSS initiate failed [Caused by
> GSSException: No valid credentials provided (Mechanism level: Failed to find
> any Kerberos tgt)]
> Exception in thread "main" java.io.IOException: Failed on local exception:
> java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed
> [Caused by GSSException: No valid credentials provided (Mechanism level:
> Failed to find any Kerberos tgt)]; Host Details :
>
> #history-server
> spark.history.kerberos.enabled true
> park.history.kerberos.principal test/spark@test
> spark.history.kerberos.keytab /home/test/test_spark.keytab
> spark.eventLog.enabled true
>
> ________________________________
> Zhanfeng Huo
--
Marcelo
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: How can I start history-server with kerberos HDFS ?
Posted by Andrew Or <an...@databricks.com>.
Hi Zhanfeng,
You will need to set these through SPARK_HISTORY_OPTS in conf/spark-env.sh.
This is documented here: http://spark.apache.org/docs/latest/monitoring.html
.
Let me know if you have it working,
-Andrew
2014-09-03 11:14 GMT-07:00 Marcelo Vanzin <va...@cloudera.com>:
> The history server (and other Spark daemons) do not read
> spark-defaults.conf. There's a bug open to implement that
> (SPARK-2098), and an open PR to fix it, but it's still not in Spark.
>
> On Wed, Sep 3, 2014 at 11:00 AM, Zhanfeng Huo <hu...@gmail.com>
> wrote:
> > Hi,
> >
> > I have seted properties in conf/spark-defaults.conf and start with
> > command "./sbin/start-history-server.sh /tmp/spark-events". It get errors
> > and seems that the properties in spark-defaults.conf file doesn't
> effect.
> > How can I solve this problem(Enable properties in spark-defaults.conf
> when
> > start spark history-server) ?
> >
> > 14/09/04 01:44:04 WARN NativeCodeLoader: Unable to load native-hadoop
> > library for your platform... using builtin-java classes where applicable
> > 14/09/04 01:44:05 WARN UserGroupInformation: PriviledgedActionException
> > as:root (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS
> > initiate failed [Caused by GSSException: No valid credentials provided
> > (Mechanism level: Failed to find any Kerberos tgt)]
> > 14/09/04 01:44:05 WARN Client: Exception encountered while connecting to
> the
> > server : javax.security.sasl.SaslException: GSS initiate failed [Caused
> by
> > GSSException: No valid credentials provided (Mechanism level: Failed to
> find
> > any Kerberos tgt)]
> > 14/09/04 01:44:05 WARN UserGroupInformation: PriviledgedActionException
> > as:root (auth:KERBEROS) cause:java.io.IOException:
> > javax.security.sasl.SaslException: GSS initiate failed [Caused by
> > GSSException: No valid credentials provided (Mechanism level: Failed to
> find
> > any Kerberos tgt)]
> > Exception in thread "main" java.io.IOException: Failed on local
> exception:
> > java.io.IOException: javax.security.sasl.SaslException: GSS initiate
> failed
> > [Caused by GSSException: No valid credentials provided (Mechanism level:
> > Failed to find any Kerberos tgt)]; Host Details :
> >
> > #history-server
> > spark.history.kerberos.enabled true
> > park.history.kerberos.principal test/spark@test
> > spark.history.kerberos.keytab /home/test/test_spark.keytab
> > spark.eventLog.enabled true
> >
> > ________________________________
> > Zhanfeng Huo
>
>
>
> --
> Marcelo
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>
Re: How can I start history-server with kerberos HDFS ?
Posted by Marcelo Vanzin <va...@cloudera.com>.
The history server (and other Spark daemons) do not read
spark-defaults.conf. There's a bug open to implement that
(SPARK-2098), and an open PR to fix it, but it's still not in Spark.
On Wed, Sep 3, 2014 at 11:00 AM, Zhanfeng Huo <hu...@gmail.com> wrote:
> Hi,
>
> I have seted properties in conf/spark-defaults.conf and start with
> command "./sbin/start-history-server.sh /tmp/spark-events". It get errors
> and seems that the properties in spark-defaults.conf file doesn't effect.
> How can I solve this problem(Enable properties in spark-defaults.conf when
> start spark history-server) ?
>
> 14/09/04 01:44:04 WARN NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> 14/09/04 01:44:05 WARN UserGroupInformation: PriviledgedActionException
> as:root (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS
> initiate failed [Caused by GSSException: No valid credentials provided
> (Mechanism level: Failed to find any Kerberos tgt)]
> 14/09/04 01:44:05 WARN Client: Exception encountered while connecting to the
> server : javax.security.sasl.SaslException: GSS initiate failed [Caused by
> GSSException: No valid credentials provided (Mechanism level: Failed to find
> any Kerberos tgt)]
> 14/09/04 01:44:05 WARN UserGroupInformation: PriviledgedActionException
> as:root (auth:KERBEROS) cause:java.io.IOException:
> javax.security.sasl.SaslException: GSS initiate failed [Caused by
> GSSException: No valid credentials provided (Mechanism level: Failed to find
> any Kerberos tgt)]
> Exception in thread "main" java.io.IOException: Failed on local exception:
> java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed
> [Caused by GSSException: No valid credentials provided (Mechanism level:
> Failed to find any Kerberos tgt)]; Host Details :
>
> #history-server
> spark.history.kerberos.enabled true
> park.history.kerberos.principal test/spark@test
> spark.history.kerberos.keytab /home/test/test_spark.keytab
> spark.eventLog.enabled true
>
> ________________________________
> Zhanfeng Huo
--
Marcelo
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org