You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Vinay Kashyap <vi...@gmail.com> on 2016/01/13 14:09:40 UTC
Error in Spark Executors when trying to read HBase table from Spark
with Kerberos enabled
Hi all,
I am using *Spark 1.5.1 in YARN cluster mode in CDH 5.5.*
I am trying to create an RDD by reading HBase table with kerberos enabled.
I am able to launch the spark job to read the HBase table, but I notice
that the executors launched for the job cannot proceed due to an issue with
Kerberos and they are stuck indefinitely.
Below is my code to read a HBase table.
*Configuration configuration = HBaseConfiguration.create();*
* configuration.set(TableInputFormat.INPUT_TABLE,
frameStorage.getHbaseStorage().getTableId());*
* String hbaseKerberosUser = "sparkUser";*
* String hbaseKerberosKeytab = "";*
* if (!hbaseKerberosUser.trim().isEmpty() &&
!hbaseKerberosKeytab.trim().isEmpty()) {*
* configuration.set("hadoop.security.authentication", "kerberos");*
* configuration.set("hbase.security.authentication", "kerberos");*
* configuration.set("hbase.security.authorization", "true");*
* configuration.set("hbase.rpc.protection", "authentication");*
* configuration.set("hbase.master.kerberos.principal",
"hbase/_HOST@CERT.LOCAL");*
* configuration.set("hbase.regionserver.kerberos.principal",
"hbase/_HOST@CERT.LOCAL");*
* configuration.set("hbase.rest.kerberos.principal",
"hbase/_HOST@CERT.LOCAL");*
* configuration.set("hbase.thrift.kerberos.principal",
"hbase/_HOST@CERT.LOCAL");*
* configuration.set("hbase.master.keytab.file",
hbaseKerberosKeytab);*
* configuration.set("hbase.regionserver.keytab.file",
hbaseKerberosKeytab);*
* configuration.set("hbase.rest.authentication.kerberos.keytab",
hbaseKerberosKeytab);*
* configuration.set("hbase.thrift.keytab.file",
hbaseKerberosKeytab);*
* UserGroupInformation.setConfiguration(configuration);*
* if (UserGroupInformation.isSecurityEnabled()) {*
* UserGroupInformation ugi = UserGroupInformation*
* .loginUserFromKeytabAndReturnUGI(hbaseKerberosUser,
hbaseKerberosKeytab);*
* TokenUtil.obtainAndCacheToken(configuration, ugi);*
* }*
* }*
* System.out.println("loading HBase Table RDD ...");*
* JavaPairRDD<ImmutableBytesWritable, Result> hbaseTableRDD =
this.sparkContext.newAPIHadoopRDD(*
* configuration, TableInputFormat.class,
ImmutableBytesWritable.class, Result.class);*
* JavaRDD<Row> tableRDD = getTableRDD(hbaseTableRDD, dataFrameModel);*
* System.out.println("Count :: " + tableRDD.count());*
Following is the error which I can see in the container logs
*16/01/13 10:01:42 WARN security.UserGroupInformation:
PriviledgedActionException as:sparkUser (auth:SIMPLE)
cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by
GSSException: No valid credentials provided (Mechanism level: Failed to
find any Kerberos tgt)]*
*16/01/13 10:01:42 WARN ipc.RpcClient: Exception encountered while
connecting to the server : javax.security.sasl.SaslException: GSS initiate
failed [Caused by GSSException: No valid credentials provided (Mechanism
level: Failed to find any Kerberos tgt)]*
*16/01/13 10:01:42 ERROR ipc.RpcClient: SASL authentication failed. The
most likely cause is missing or invalid credentials. Consider 'kinit'.*
*javax.security.sasl.SaslException: GSS initiate failed [Caused by
GSSException: No valid credentials provided (Mechanism level: Failed to
find any Kerberos tgt)]*
* at
com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:212)*
* at
org.apache.hadoop.hbase.security.HBaseSaslRpcClient.saslConnect(HBaseSaslRpcClient.java:179)*
* at
org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupSaslConnection(RpcClient.java:770)*
* at
org.apache.hadoop.hbase.ipc.RpcClient$Connection.access$600(RpcClient.java:357)*
* at
org.apache.hadoop.hbase.ipc.RpcClient$Connection$2.run(RpcClient.java:891)*
* at
org.apache.hadoop.hbase.ipc.RpcClient$Connection$2.run(RpcClient.java:888)*
* at java.security.AccessController.doPrivileged(Native Method)*
* at javax.security.auth.Subject.doAs(Subject.java:415)*
Have valid Kerberos Token as can be seen below:
sparkUser@infra:/ebs1/agent$ klist
Ticket cache: FILE:/tmp/krb5cc_1001
Default principal: sparkUser@CERT.LOCAL
Valid starting Expires Service principal
13/01/2016 12:07 14/01/2016 12:07 krbtgt/CERT.LOCAL@CERT.LOCAL
Also, I confirmed that only reading from HBase is giving this problem.
Because I can read a simple file in HDFS and I am able to create the RDD as
required.
After digging through some contents in the net, found that there is a
ticket in JIRA which is logged which is similar to what I am experiencing
*https://issues.apache.org/jira/browse/SPARK-12279
<https://issues.apache.org/jira/browse/SPARK-12279>*
Wanted to know if the issue is the same as I am facing..??
And any workaround for the same so that I can proceed with my requirement
reading from HBase table.??
--
*Thanks and regards*
*Vinay Kashyap*
Re: Error in Spark Executors when trying to read HBase table from
Spark with Kerberos enabled
Posted by Vinay Kashyap <vi...@gmail.com>.
Hi Guys,
Any help regarding this issue..??
On Wed, Jan 13, 2016 at 6:39 PM, Vinay Kashyap <vi...@gmail.com> wrote:
> Hi all,
>
> I am using *Spark 1.5.1 in YARN cluster mode in CDH 5.5.*
> I am trying to create an RDD by reading HBase table with kerberos enabled.
> I am able to launch the spark job to read the HBase table, but I notice
> that the executors launched for the job cannot proceed due to an issue with
> Kerberos and they are stuck indefinitely.
>
> Below is my code to read a HBase table.
>
>
> *Configuration configuration = HBaseConfiguration.create();*
> * configuration.set(TableInputFormat.INPUT_TABLE,
> frameStorage.getHbaseStorage().getTableId());*
> * String hbaseKerberosUser = "sparkUser";*
> * String hbaseKerberosKeytab = "";*
> * if (!hbaseKerberosUser.trim().isEmpty() &&
> !hbaseKerberosKeytab.trim().isEmpty()) {*
> * configuration.set("hadoop.security.authentication", "kerberos");*
> * configuration.set("hbase.security.authentication", "kerberos");*
> * configuration.set("hbase.security.authorization", "true");*
> * configuration.set("hbase.rpc.protection", "authentication");*
> * configuration.set("hbase.master.kerberos.principal",
> "hbase/_HOST@CERT.LOCAL");*
> * configuration.set("hbase.regionserver.kerberos.principal",
> "hbase/_HOST@CERT.LOCAL");*
> * configuration.set("hbase.rest.kerberos.principal",
> "hbase/_HOST@CERT.LOCAL");*
> * configuration.set("hbase.thrift.kerberos.principal",
> "hbase/_HOST@CERT.LOCAL");*
> * configuration.set("hbase.master.keytab.file",
> hbaseKerberosKeytab);*
> * configuration.set("hbase.regionserver.keytab.file",
> hbaseKerberosKeytab);*
> * configuration.set("hbase.rest.authentication.kerberos.keytab",
> hbaseKerberosKeytab);*
> * configuration.set("hbase.thrift.keytab.file",
> hbaseKerberosKeytab);*
> * UserGroupInformation.setConfiguration(configuration);*
> * if (UserGroupInformation.isSecurityEnabled()) {*
> * UserGroupInformation ugi = UserGroupInformation*
> * .loginUserFromKeytabAndReturnUGI(hbaseKerberosUser,
> hbaseKerberosKeytab);*
> * TokenUtil.obtainAndCacheToken(configuration, ugi);*
> * }*
> * }*
>
> * System.out.println("loading HBase Table RDD ...");*
> * JavaPairRDD<ImmutableBytesWritable, Result> hbaseTableRDD =
> this.sparkContext.newAPIHadoopRDD(*
> * configuration, TableInputFormat.class,
> ImmutableBytesWritable.class, Result.class);*
> * JavaRDD<Row> tableRDD = getTableRDD(hbaseTableRDD, dataFrameModel);*
> * System.out.println("Count :: " + tableRDD.count());*
> Following is the error which I can see in the container logs
>
> *16/01/13 10:01:42 WARN security.UserGroupInformation:
> PriviledgedActionException as:sparkUser (auth:SIMPLE)
> cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by
> GSSException: No valid credentials provided (Mechanism level: Failed to
> find any Kerberos tgt)]*
> *16/01/13 10:01:42 WARN ipc.RpcClient: Exception encountered while
> connecting to the server : javax.security.sasl.SaslException: GSS initiate
> failed [Caused by GSSException: No valid credentials provided (Mechanism
> level: Failed to find any Kerberos tgt)]*
> *16/01/13 10:01:42 ERROR ipc.RpcClient: SASL authentication failed. The
> most likely cause is missing or invalid credentials. Consider 'kinit'.*
> *javax.security.sasl.SaslException: GSS initiate failed [Caused by
> GSSException: No valid credentials provided (Mechanism level: Failed to
> find any Kerberos tgt)]*
> * at
> com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:212)*
> * at
> org.apache.hadoop.hbase.security.HBaseSaslRpcClient.saslConnect(HBaseSaslRpcClient.java:179)*
> * at
> org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupSaslConnection(RpcClient.java:770)*
> * at
> org.apache.hadoop.hbase.ipc.RpcClient$Connection.access$600(RpcClient.java:357)*
> * at
> org.apache.hadoop.hbase.ipc.RpcClient$Connection$2.run(RpcClient.java:891)*
> * at
> org.apache.hadoop.hbase.ipc.RpcClient$Connection$2.run(RpcClient.java:888)*
> * at java.security.AccessController.doPrivileged(Native Method)*
> * at javax.security.auth.Subject.doAs(Subject.java:415)*
>
> Have valid Kerberos Token as can be seen below:
>
> sparkUser@infra:/ebs1/agent$ klist
> Ticket cache: FILE:/tmp/krb5cc_1001
> Default principal: sparkUser@CERT.LOCAL
>
> Valid starting Expires Service principal
> 13/01/2016 12:07 14/01/2016 12:07 krbtgt/CERT.LOCAL@CERT.LOCAL
>
> Also, I confirmed that only reading from HBase is giving this problem.
> Because I can read a simple file in HDFS and I am able to create the RDD as
> required.
> After digging through some contents in the net, found that there is a
> ticket in JIRA which is logged which is similar to what I am experiencing
> *https://issues.apache.org/jira/browse/SPARK-12279
> <https://issues.apache.org/jira/browse/SPARK-12279>*
>
> Wanted to know if the issue is the same as I am facing..??
> And any workaround for the same so that I can proceed with my requirement
> reading from HBase table.??
>
> --
> *Thanks and regards*
> *Vinay Kashyap*
>
--
*Thanks and regards*
*Vinay Kashyap*