You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Jeremy (Jira)" <ji...@apache.org> on 2019/11/14 21:35:00 UTC
[jira] [Comment Edited] (SPARK-29884) spark-submit to kuberentes
can not parse valid ca certificate
[ https://issues.apache.org/jira/browse/SPARK-29884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16974621#comment-16974621 ]
Jeremy edited comment on SPARK-29884 at 11/14/19 9:34 PM:
----------------------------------------------------------
After doing some debugging it seams like this might be in fabric k8s client. It tries to use .kube/config even if it gets all the parameters is needs from arguments.
was (Author: jeremyjjbrown):
After doing some debugging it seams like this might be in fabric k8s client. I tries to use .kube/config even if it gets all the parameters is needs from arguments.
> spark-submit to kuberentes can not parse valid ca certificate
> -------------------------------------------------------------
>
> Key: SPARK-29884
> URL: https://issues.apache.org/jira/browse/SPARK-29884
> Project: Spark
> Issue Type: Bug
> Components: Kubernetes
> Affects Versions: 2.4.4
> Environment: A kuberentes cluster that has been in use for over 2 years and handles large amounts of production payloads.
> Reporter: Jeremy
> Priority: Major
>
> spark submit can not be used to to schedule to kuberentes with oauth token and cacert
> {code:java}
> spark-submit \
> --deploy-mode cluster \
> --class org.apache.spark.examples.SparkPi \
> --master k8s://https://api.borg-dev-1-aws-eu-west-1.k8s.in.here.com \
> --conf spark.kubernetes.authenticate.submission.oauthToken=$TOKEN \
> --conf spark.kubernetes.authenticate.driver.serviceAccountName=spark \
> --conf spark.kubernetes.authenticate.submission.caCertFile=/home/jeremybr/.kube/borg-dev-1-aws-eu-west-1.crt \
> --conf spark.kubernetes.namespace=here-olp-3dds-sit \
> --conf spark.executor.instances=1 \
> --conf spark.app.name=spark-pi \
> --conf spark.kubernetes.driver.docker.image=kubespark/spark-driver:v2.2.0-kubernetes-0.5.0 \
> --conf spark.kubernetes.executor.docker.image=kubespark/spark-executor:v2.2.0-kubernetes-0.5.0 \
> local:///opt/spark/examples/jars/spark-examples_2.11-2.2.0-k8s-0.5.0.jar
> {code}
> returns
> {code:java}
> log4j:WARN No appenders could be found for logger (io.fabric8.kubernetes.client.Config).
> log4j:WARN Please initialize the log4j system properly.
> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
> Exception in thread "main" io.fabric8.kubernetes.client.KubernetesClientException: An error has occurred.
> at io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:64)
> at io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:53)
> at io.fabric8.kubernetes.client.utils.HttpClientUtils.createHttpClient(HttpClientUtils.java:183)
> at org.apache.spark.deploy.k8s.SparkKubernetesClientFactory$.createKubernetesClient(SparkKubernetesClientFactory.scala:84)
> at org.apache.spark.deploy.k8s.submit.KubernetesClientApplication$$anonfun$run$4.apply(KubernetesClientApplication.scala:235)
> at org.apache.spark.deploy.k8s.submit.KubernetesClientApplication$$anonfun$run$4.apply(KubernetesClientApplication.scala:235)
> at org.apache.spark.util.Utils$.tryWithResource(Utils.scala:2542)
> at org.apache.spark.deploy.k8s.submit.KubernetesClientApplication.run(KubernetesClientApplication.scala:241)
> at org.apache.spark.deploy.k8s.submit.KubernetesClientApplication.start(KubernetesClientApplication.scala:204)
> at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:845)
> at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:161)
> at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:184)
> at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
> at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:920)
> at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:929)
> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
> Caused by: java.security.cert.CertificateException: Could not parse certificate: java.io.IOException: Empty input
> at sun.security.provider.X509Factory.engineGenerateCertificate(X509Factory.java:110)
> at java.security.cert.CertificateFactory.generateCertificate(CertificateFactory.java:339)
> at io.fabric8.kubernetes.client.internal.CertUtils.createKeyStore(CertUtils.java:104)
> at io.fabric8.kubernetes.client.internal.CertUtils.createKeyStore(CertUtils.java:197)
> at io.fabric8.kubernetes.client.internal.SSLUtils.keyManagers(SSLUtils.java:128)
> at io.fabric8.kubernetes.client.internal.SSLUtils.keyManagers(SSLUtils.java:122)
> at io.fabric8.kubernetes.client.utils.HttpClientUtils.createHttpClient(HttpClientUtils.java:78)
> ... 13 more
> Caused by: java.io.IOException: Empty input
> at sun.security.provider.X509Factory.engineGenerateCertificate(X509Factory.java:106)
> ... 19 more
> {code}
> The cacert and token are both valid and work even with curl
> {code:java}
> curl --cacert /home/jeremybr/.kube/borg-dev-1-aws-eu-west-1.crt -H "Authorization: bearer $TOKEN" -v https://api.borg-dev-1-aws-eu-west-1.k8s.in.here.com/api/v1/namespaces/here-olp-3dds-sit/pods -o out
> % Total % Received % Xferd Average Speed Time Time Time Current
> Dload Upload Total Spent Left Speed
> 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Trying 10.117.233.37:443...
> * TCP_NODELAY set
> * Connected to api.borg-dev-1-aws-eu-west-1.k8s.in.here.com (10.117.233.37) port 443 (#0)
> * ALPN, offering h2
> * ALPN, offering http/1.1
> * successfully set certificate verify locations:
> * CAfile: /home/jeremybr/.kube/borg-dev-1-aws-eu-west-1.crt
> CApath: none
> } [5 bytes data]
> * TLSv1.3 (OUT), TLS handshake, Client hello (1):
> } [512 bytes data]
> * TLSv1.3 (IN), TLS handshake, Server hello (2):
> { [58 bytes data]
> * TLSv1.2 (IN), TLS handshake, Certificate (11):
> { [1565 bytes data]
> * TLSv1.2 (IN), TLS handshake, Server key exchange (12):
> { [556 bytes data]
> * TLSv1.2 (IN), TLS handshake, Request CERT (13):
> { [588 bytes data]
> * TLSv1.2 (IN), TLS handshake, Server finished (14):
> { [4 bytes data]
> * TLSv1.2 (OUT), TLS handshake, Certificate (11):
> } [7 bytes data]
> * TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
> } [37 bytes data]
> * TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
> } [1 bytes data]
> * TLSv1.2 (OUT), TLS handshake, Finished (20):
> } [16 bytes data]
> * TLSv1.2 (IN), TLS handshake, Finished (20):
> { [16 bytes data]
> * SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
> * ALPN, server accepted to use h2
> * Server certificate:
> * subject: CN=kubernetes-master
> * start date: Apr 2 13:30:06 2019 GMT
> * expire date: Apr 1 13:30:06 2020 GMT
> * subjectAltName: host "api.borg-dev-1-aws-eu-west-1.k8s.in.here.com" matched cert's "api.borg-dev-1-aws-eu-west-1.k8s.in.here.com"
> * issuer: C=DE; ST=Berlin; O=HERE Global BV; OU=OLP Engineering Infrastructure Development; CN=borg-dev-1-aws-eu-west-1; emailAddress=sdp_ops_team@here.com
> * SSL certificate verify ok.
> 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Using HTTP2, server supports multi-use
> * Connection state changed (HTTP/2 confirmed)
> * Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
> } [5 bytes data]
> * Using Stream ID: 1 (easy handle 0x558a2ede87b0)
> } [5 bytes data]
> > GET /api/v1/namespaces/here-olp-3dds-sit/pods HTTP/2
> > Host: api.borg-dev-1-aws-eu-west-1.k8s.in.here.com
> > User-Agent: curl/7.66.0
> > Accept: */*
> > Authorization: bearer *****************************************
> >
> { [5 bytes data]
> * Connection state changed (MAX_CONCURRENT_STREAMS == 250)!
> } [5 bytes data]
> < HTTP/2 200
> < audit-id: a26869e9-7b42-4013-b1fa-839e963c6b78
> < content-type: application/json
> < date: Wed, 13 Nov 2019 20:20:18 GMT
> <
> { [5 bytes data]
> 100 56466 0 56466 0 0 64020 0 --:--:-- --:--:-- --:--:-- 63947
> * Connection #0 to host api.borg-dev-1-aws-eu-west-1.k8s.in.here.com left intact
> {code}
>
> {code:java}
> // code placeholder
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org