You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Jeremy (Jira)" <ji...@apache.org> on 2019/11/13 20:22:00 UTC
[jira] [Created] (SPARK-29884) spark-Submit to kuberentes can not
parse valid ca certificate
Jeremy created SPARK-29884:
------------------------------
Summary: spark-Submit to kuberentes can not parse valid ca certificate
Key: SPARK-29884
URL: https://issues.apache.org/jira/browse/SPARK-29884
Project: Spark
Issue Type: Bug
Components: Kubernetes
Affects Versions: 2.4.4
Environment: A kuberentes cluster that has been in use for over 2 years and handles large amounts of production payloads.
Reporter: Jeremy
spark submit can not be used to to schedule to kuberentes with oauth token and cacert
{code:java}
spark-submit \
--deploy-mode cluster \
--class org.apache.spark.examples.SparkPi \
--master k8s://https://api.borg-dev-1-aws-eu-west-1.k8s.in.here.com \
--conf spark.kubernetes.authenticate.submission.oauthToken=$TOKEN \
--conf spark.kubernetes.authenticate.driver.serviceAccountName=spark \
--conf spark.kubernetes.authenticate.submission.caCertFile=/home/jeremybr/.kube/borg-dev-1-aws-eu-west-1.crt \
--conf spark.kubernetes.namespace=here-olp-3dds-sit \
--conf spark.executor.instances=1 \
--conf spark.app.name=spark-pi \
--conf spark.kubernetes.driver.docker.image=kubespark/spark-driver:v2.2.0-kubernetes-0.5.0 \
--conf spark.kubernetes.executor.docker.image=kubespark/spark-executor:v2.2.0-kubernetes-0.5.0 \
local:///opt/spark/examples/jars/spark-examples_2.11-2.2.0-k8s-0.5.0.jar
{code}
returns
{code:java}
log4j:WARN No appenders could be found for logger (io.fabric8.kubernetes.client.Config).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
Exception in thread "main" io.fabric8.kubernetes.client.KubernetesClientException: An error has occurred.
at io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:64)
at io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:53)
at io.fabric8.kubernetes.client.utils.HttpClientUtils.createHttpClient(HttpClientUtils.java:183)
at org.apache.spark.deploy.k8s.SparkKubernetesClientFactory$.createKubernetesClient(SparkKubernetesClientFactory.scala:84)
at org.apache.spark.deploy.k8s.submit.KubernetesClientApplication$$anonfun$run$4.apply(KubernetesClientApplication.scala:235)
at org.apache.spark.deploy.k8s.submit.KubernetesClientApplication$$anonfun$run$4.apply(KubernetesClientApplication.scala:235)
at org.apache.spark.util.Utils$.tryWithResource(Utils.scala:2542)
at org.apache.spark.deploy.k8s.submit.KubernetesClientApplication.run(KubernetesClientApplication.scala:241)
at org.apache.spark.deploy.k8s.submit.KubernetesClientApplication.start(KubernetesClientApplication.scala:204)
at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:845)
at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:161)
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:184)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:920)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:929)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.security.cert.CertificateException: Could not parse certificate: java.io.IOException: Empty input
at sun.security.provider.X509Factory.engineGenerateCertificate(X509Factory.java:110)
at java.security.cert.CertificateFactory.generateCertificate(CertificateFactory.java:339)
at io.fabric8.kubernetes.client.internal.CertUtils.createKeyStore(CertUtils.java:104)
at io.fabric8.kubernetes.client.internal.CertUtils.createKeyStore(CertUtils.java:197)
at io.fabric8.kubernetes.client.internal.SSLUtils.keyManagers(SSLUtils.java:128)
at io.fabric8.kubernetes.client.internal.SSLUtils.keyManagers(SSLUtils.java:122)
at io.fabric8.kubernetes.client.utils.HttpClientUtils.createHttpClient(HttpClientUtils.java:78)
... 13 more
Caused by: java.io.IOException: Empty input
at sun.security.provider.X509Factory.engineGenerateCertificate(X509Factory.java:106)
... 19 more
{code}
The cacert and token are both valid and work even with curl
{code:java}
curl --cacert /home/jeremybr/.kube/borg-dev-1-aws-eu-west-1.crt -H "Authorization: bearer $TOKEN" -v https://api.borg-dev-1-aws-eu-west-1.k8s.in.here.com/api/v1/namespaces/here-olp-3dds-sit/pods -o out
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Trying 10.117.233.37:443...
* TCP_NODELAY set
* Connected to api.borg-dev-1-aws-eu-west-1.k8s.in.here.com (10.117.233.37) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /home/jeremybr/.kube/borg-dev-1-aws-eu-west-1.crt
CApath: none
} [5 bytes data]
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
* TLSv1.3 (IN), TLS handshake, Server hello (2):
{ [58 bytes data]
* TLSv1.2 (IN), TLS handshake, Certificate (11):
{ [1565 bytes data]
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
{ [556 bytes data]
* TLSv1.2 (IN), TLS handshake, Request CERT (13):
{ [588 bytes data]
* TLSv1.2 (IN), TLS handshake, Server finished (14):
{ [4 bytes data]
* TLSv1.2 (OUT), TLS handshake, Certificate (11):
} [7 bytes data]
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
} [37 bytes data]
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
} [1 bytes data]
* TLSv1.2 (OUT), TLS handshake, Finished (20):
} [16 bytes data]
* TLSv1.2 (IN), TLS handshake, Finished (20):
{ [16 bytes data]
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server accepted to use h2
* Server certificate:
* subject: CN=kubernetes-master
* start date: Apr 2 13:30:06 2019 GMT
* expire date: Apr 1 13:30:06 2020 GMT
* subjectAltName: host "api.borg-dev-1-aws-eu-west-1.k8s.in.here.com" matched cert's "api.borg-dev-1-aws-eu-west-1.k8s.in.here.com"
* issuer: C=DE; ST=Berlin; O=HERE Global BV; OU=OLP Engineering Infrastructure Development; CN=borg-dev-1-aws-eu-west-1; emailAddress=sdp_ops_team@here.com
* SSL certificate verify ok.
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
} [5 bytes data]
* Using Stream ID: 1 (easy handle 0x558a2ede87b0)
} [5 bytes data]
> GET /api/v1/namespaces/here-olp-3dds-sit/pods HTTP/2
> Host: api.borg-dev-1-aws-eu-west-1.k8s.in.here.com
> User-Agent: curl/7.66.0
> Accept: */*
> Authorization: bearer eyJhbGciOiJSUzI1NiIsImtpZCI6Im9pZDEifQ.eyJpc3MiOiJodHRwczovL3N0LnAuYWNjb3VudC5oZXJlLmNvbSIsInN1YiI6IlpkYU4wQ3h6WWdxTHpaUG1CNEFYIiwiYXVkIjoiaW50LTEtYXdzLWV1LXdlc3QtMS5rOHMuaW4uaGVyZS5jb20iLCJleHAiOjE1NzM3NjI0ODQsImlhdCI6MTU3MzY3NjA4NCwibm9uY2UiOiI0YjhGU2kiLCJzZHAiOlsiR1JPVVAtMDM1NzY2MGMtOTkwMS00NTEwLTkzZWUtMTdlYTY1NDE2NTViIl19.SIXRdOzeTRYlK5WGO4r0ND9tGBaVhJlq05-dYJxfSlt_JO77FBPK0YDUNlsdCJD-F2QrTtHtF_iiyT7eQRv53LPSWeUHZPzP4zDCkufFNvHBZGdr5cxvQfnh4FntENXAeR8mQA-dOkMUaRS4mFitaeBc4tBDXFTNFAOhh2PN3_3WXg-j5cbNFZDD13GVdI4cYqDEjA94ltMRhTt4WYlVWMJbgo8zjFZAiMDrwR1VmS1AHSDtARc6nZaN534DEC289tBoH8NbpMNP9RYB_AuERSlql4MVeoT7TzuUgCFtC-MWiJvyZ6qfA5jWgMJuLj0Bf0nfcC2ON4HVi1rfecarDQ
>
{ [5 bytes data]
* Connection state changed (MAX_CONCURRENT_STREAMS == 250)!
} [5 bytes data]
< HTTP/2 200
< audit-id: a26869e9-7b42-4013-b1fa-839e963c6b78
< content-type: application/json
< date: Wed, 13 Nov 2019 20:20:18 GMT
<
{ [5 bytes data]
100 56466 0 56466 0 0 64020 0 --:--:-- --:--:-- --:--:-- 63947
* Connection #0 to host api.borg-dev-1-aws-eu-west-1.k8s.in.here.com left intact
{code}
{code:java}
// code placeholder
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org