You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Ron Gonzalez (JIRA)" <ji...@apache.org> on 2018/02/27 21:12:00 UTC
[jira] [Created] (SPARK-23527) Error with spark-submit and kerberos
with TLS-enabled Hadoop cluster
Ron Gonzalez created SPARK-23527:
------------------------------------
Summary: Error with spark-submit and kerberos with TLS-enabled Hadoop cluster
Key: SPARK-23527
URL: https://issues.apache.org/jira/browse/SPARK-23527
Project: Spark
Issue Type: Bug
Components: Spark Submit
Affects Versions: 2.2.1
Environment: core-site.xml
<property>
<name>hadoop.security.key.provider.path</name>
<value>kms://https@host1.domain.com;host2.domain.com:16000/kms</value>
</property>
hdfs-site.xml
<property>
<name>dfs.encryption.key.provider.uri</name>
<value>kms://https@host1.domain.com;host2.domain.com:16000/kms</value>
</property>
Reporter: Ron Gonzalez
For current configuration of our enterprise cluster, I submit using spark-submit:
./spark-submit --master yarn --deploy-mode cluster --class org.apache.spark.examples.SparkPi --conf spark.yarn.jars=hdfs:/user/user1/spark/lib/*.jar ../examples/jars/spark-examples_2.11-2.2.1.jar 10
I am getting the following problem:
18/02/27 21:03:48 INFO hdfs.DFSClient: Created HDFS_DELEGATION_TOKEN token 3351181 for svchdc236d on ha-hdfs:nameservice1
Exception in thread "main" java.lang.IllegalArgumentException: java.net.UnknownHostException: host1.domain.com;host2.domain.com
at org.apache.hadoop.security.SecurityUtil.buildTokenService(SecurityUtil.java:374)
at org.apache.hadoop.crypto.key.kms.KMSClientProvider.getDelegationTokenService(KMSClientProvider.java:825)
at org.apache.hadoop.crypto.key.kms.KMSClientProvider.addDelegationTokens(KMSClientProvider.java:781)
at org.apache.hadoop.crypto.key.KeyProviderDelegationTokenExtension.addDelegationTokens(KeyProviderDelegationTokenExtension.java:86)
at org.apache.hadoop.hdfs.DistributedFileSystem.addDelegationTokens(DistributedFileSystem.java:2046)
at org.apache.spark.deploy.yarn.security.HadoopFSCredentialProvider$$anonfun$obtainCredentials$1.apply(HadoopFSCredentialProvider.scala:52)
If I get rid of the other host for the properties so instead of kms://https@host1.domain.com;host2.domain.com:16000/kms, I convert it to:
kms://https@host1.domain.com:16000/kms
it fails with a different error:
java.io.IOException: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
If I do the same thing using spark 1.6, it works so it seems like a regression...
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org