You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Partha Pratim Ghosh (JIRA)" <ji...@apache.org> on 2016/07/04 11:51:11 UTC
[jira] [Comment Edited] (SPARK-16298) spark.yarn.principal not
working
[ https://issues.apache.org/jira/browse/SPARK-16298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15358814#comment-15358814 ]
Partha Pratim Ghosh edited comment on SPARK-16298 at 7/4/16 11:50 AM:
----------------------------------------------------------------------
I have tried with spark-submit -
spark-submit --class connectivity.spark.java.Wordcount --executor-memory 12g --principal <principal name> --keytab /app/<keytab file name>.keytab --master yarn --deploy-mode cluster /app/java/spark-connectivity/target/spark-connectivity-0.0.1-SNAPSHOT.jar
To be sure that it is using keytab I removed the kerberos cache file from /tmp so that klist -f provides the following output -
klist: No credentials cache found (ticket cache FILE:/tmp/krb5cc_515)
To be sure that the key is valid I checked the key with klist -k -t <keytab file name>.keytab
Now, when I ran the above piece I received the following -
16/07/01 07:07:45 INFO DFSClient: Created HDFS_DELEGATION_TOKEN token 5958319 for <principal> on ha-hdfs:<unix env>
Exception in thread "main" java.io.IOException: org.apache.hadoop.security.authentication.client.AuthenticationException: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)
at org.apache.hadoop.crypto.key.kms.KMSClientProvider.addDelegationTokens(KMSClientProvider.java:794)
at org.apache.hadoop.crypto.key.KeyProviderDelegationTokenExtension.addDelegationTokens(KeyProviderDelegationTokenExtension.java:86)
...
Caused by: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)
at sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147)
at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:122)
at sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187)
Am I missing something here?
Seems that even if I provide the principal and keytab spark-submit tries to read from the default ticket cache in the unix session.
In that case, is there any requirement that the default unix keytab shall be related to the principal and keytab I provide in some way?
was (Author: partha):
I have tried with spark-submit -
spark-submit --class connectivity.spark.java.Wordcount --executor-memory 12g --principal <principal name> --keytab /app/<keytab file name>.keytab --master yarn --deploy-mode cluster /app/java/spark-connectivity/target/spark-connectivity-0.0.1-SNAPSHOT.jar
To be sure that it is using keytab I removed the kerberos cache file from /tmp so that klist -f provides the following output -
klist: No credentials cache found (ticket cache FILE:/tmp/krb5cc_515)
To be sure that the key is valid I checked the key with klist -k -t <keytab file name>.keytab
Now, when I ran the above piece I received the following -
16/07/01 07:07:45 INFO DFSClient: Created HDFS_DELEGATION_TOKEN token 5958319 for pg18084 on ha-hdfs:FajitaDevelopment
Exception in thread "main" java.io.IOException: org.apache.hadoop.security.authentication.client.AuthenticationException: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)
at org.apache.hadoop.crypto.key.kms.KMSClientProvider.addDelegationTokens(KMSClientProvider.java:794)
at org.apache.hadoop.crypto.key.KeyProviderDelegationTokenExtension.addDelegationTokens(KeyProviderDelegationTokenExtension.java:86)
...
Caused by: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)
at sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147)
at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:122)
at sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187)
Am I missing something here?
Seems that even if I provide the principal and keytab spark-submit tries to read from the default ticket cache in the unix session.
In that case, is there any requirement that the default unix keytab shall be related to the principal and keytab I provide in some way?
> spark.yarn.principal not working
> --------------------------------
>
> Key: SPARK-16298
> URL: https://issues.apache.org/jira/browse/SPARK-16298
> Project: Spark
> Issue Type: Bug
> Reporter: Partha Pratim Ghosh
>
> I am opening a Spark configuration with spark.yarn.principal and spark.yarn.keytab. However, this is not authenticating the underlying HDFS with the same principal and keytab. Instead, seems it is picking up from ticket cache. Without this feature the spark.yarn.principal and spark.yarn.keytab doesn't seem to be logical.
> Sample code -
> SparkConf conf = new SparkConf().setMaster("yarn-client").setAppName("spark-test")
> .set("spark.repl.class.uri", classServerUri);
> conf.set("spark.yarn.principal", principal);
> conf.set("spark.yarn.keytab", keytab);
> conf.setSparkHome(sparkBasePath);
>
> if (execUri != null) {
> conf.set("spark.executor.uri", execUri);
> }
> conf.set("spark.executor.memory", "8g");
> conf.set("spark.scheduler.mode", "FAIR");
> SparkContext sparkContext = new SparkContext(conf);
> Please advise how this can be achieved.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org