You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Yuming Wang (JIRA)" <ji...@apache.org> on 2018/02/28 15:12:00 UTC
[jira] [Commented] (SPARK-23527) Error with spark-submit and
kerberos with TLS-enabled Hadoop cluster
[ https://issues.apache.org/jira/browse/SPARK-23527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16380440#comment-16380440 ]
Yuming Wang commented on SPARK-23527:
-------------------------------------
It isn't bug, please check your hostname: {{host1.domain.com;host2.domain.com}}
> Error with spark-submit and kerberos with TLS-enabled Hadoop cluster
> --------------------------------------------------------------------
>
> Key: SPARK-23527
> URL: https://issues.apache.org/jira/browse/SPARK-23527
> Project: Spark
> Issue Type: Bug
> Components: Spark Submit
> Affects Versions: 2.2.1
> Environment: core-site.xml
> <property>
> <name>hadoop.security.key.provider.path</name>
> <value>kms://https@host1.domain.com;host2.domain.com:16000/kms</value>
> </property>
> hdfs-site.xml
> <property>
> <name>dfs.encryption.key.provider.uri</name>
> <value>kms://https@host1.domain.com;host2.domain.com:16000/kms</value>
> </property>
> Reporter: Ron Gonzalez
> Priority: Critical
>
> For current configuration of our enterprise cluster, I submit using spark-submit:
> ./spark-submit --master yarn --deploy-mode cluster --class org.apache.spark.examples.SparkPi --conf spark.yarn.jars=hdfs:/user/user1/spark/lib/*.jar ../examples/jars/spark-examples_2.11-2.2.1.jar 10
> I am getting the following problem:
>
> 18/02/27 21:03:48 INFO hdfs.DFSClient: Created HDFS_DELEGATION_TOKEN token 3351181 for svchdc236d on ha-hdfs:nameservice1
> Exception in thread "main" java.lang.IllegalArgumentException: java.net.UnknownHostException: host1.domain.com;host2.domain.com
> at org.apache.hadoop.security.SecurityUtil.buildTokenService(SecurityUtil.java:374)
> at org.apache.hadoop.crypto.key.kms.KMSClientProvider.getDelegationTokenService(KMSClientProvider.java:825)
> at org.apache.hadoop.crypto.key.kms.KMSClientProvider.addDelegationTokens(KMSClientProvider.java:781)
> at org.apache.hadoop.crypto.key.KeyProviderDelegationTokenExtension.addDelegationTokens(KeyProviderDelegationTokenExtension.java:86)
> at org.apache.hadoop.hdfs.DistributedFileSystem.addDelegationTokens(DistributedFileSystem.java:2046)
> at org.apache.spark.deploy.yarn.security.HadoopFSCredentialProvider$$anonfun$obtainCredentials$1.apply(HadoopFSCredentialProvider.scala:52)
>
> If I get rid of the other host for the properties so instead of kms://https@host1.domain.com;host2.domain.com:16000/kms, I convert it to:
> kms://https@host1.domain.com:16000/kms
> it fails with a different error:
> java.io.IOException: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
> If I do the same thing using spark 1.6, it works so it seems like a regression...
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org