You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Steven Stetzler <st...@gmail.com> on 2018/12/13 21:48:51 UTC

Problem running Spark on Kubernetes: Certificate error

Hello,

I am following the tutorial here (
https://spark.apache.org/docs/latest/running-on-kubernetes.html) to get
spark running on a Kubernetes cluster. My Kubernetes cluster is hosted with
Digital Ocean's kubernetes cluster manager. I have change the KUBECONFIG
environment variable to point to my cluster access credentials, so both
Spark and kubectl can speak with the nodes.

I am running into an issue when trying to run the SparkPi example as
described in the Spark on Kubernetes tutorials. The command I am running
is:

./bin/spark-submit --master k8s://$CLUSTERIP --deploy-mode cluster --name
spark-pi --class org.apache.spark.examples.SparkPi --conf
spark.executor.instances=1 --conf
spark.kubernetes.container.image=$IMAGEURL --conf
spark.kubernetes.authenticate.driver.serviceAccountName=spark
local:///opt/spark/examples/jars/spark-examples_2.11-2.4.0.jar

where CLUSTERIP contains the ip of my cluster and IMAGEURL contains the URL
of the Spark docker image I am using (
https://hub.docker.com/r/stevenstetzler/spark/). This docker image was
built and pushed with the script included in the Spark 2.4 distribution. I
have created a service account for Spark to ensure that it has proper
permissions to create pods etc., which I checked using

kubectl auth can-i create pods --as=system:serviceaccount:default:spark

When I try to run the SparkPi example using the above command, I get the
following output:

2018-12-12 06:26:15 WARN  Utils:66 - Your hostname, docker-test resolves to
a loopback address: 127.0.1.1; using 10.46.0.6 instead (on interface eth0)
2018-12-12 06:26:15 WARN  Utils:66 - Set SPARK_LOCAL_IP if you need to bind
to another address
2018-12-12 06:26:19 INFO  LoggingPodStatusWatcherImpl:54 - State changed,
new state:
         pod name: spark-pi-1544595975520-driver
         namespace: default
         labels: spark-app-selector ->
spark-ec5eb54644d348e7a213f8178b8ef61f, spark-role -> driver
         pod uid: d5d6bdc7-fdd6-11e8-b666-8e815d3815b2
         creation time: 2018-12-12T06:26:18Z
         service account name: spark
         volumes: spark-local-dir-1, spark-conf-volume, spark-token-qf9dt
         node name: N/A
         start time: N/A
         container images: N/A
         phase: Pending
         status: []
2018-12-12 06:26:19 INFO  LoggingPodStatusWatcherImpl:54 - State changed,
new state:
         pod name: spark-pi-1544595975520-driver
         namespace: default
         labels: spark-app-selector ->
spark-ec5eb54644d348e7a213f8178b8ef61f, spark-role -> driver
         pod uid: d5d6bdc7-fdd6-11e8-b666-8e815d3815b2
         creation time: 2018-12-12T06:26:18Z
         service account name: spark
         volumes: spark-local-dir-1, spark-conf-volume, spark-token-qf9dt
         node name: flamboyant-darwin-3rhc
         start time: N/A
         container images: N/A
         phase: Pending
         status: []
2018-12-12 06:26:19 INFO  LoggingPodStatusWatcherImpl:54 - State changed,
new state:
         pod name: spark-pi-1544595975520-driver
         namespace: default
         labels: spark-app-selector ->
spark-ec5eb54644d348e7a213f8178b8ef61f, spark-role -> driver
         pod uid: d5d6bdc7-fdd6-11e8-b666-8e815d3815b2
         creation time: 2018-12-12T06:26:18Z
         service account name: spark
         volumes: spark-local-dir-1, spark-conf-volume, spark-token-qf9dt
         node name: flamboyant-darwin-3rhc
         start time: 2018-12-12T06:26:18Z
         container images: docker.io/stevenstetzler/spark:v1
         phase: Pending
         status: [ContainerStatus(containerID=null, image=
docker.io/stevenstetzler/spark:v1, imageID=,
lastState=ContainerState(running=null, terminated=null, waiting=null,
additionalProperties={}), name=spark-kubernetes-driver, ready=false,
restartCount=0, state=ContainerState(running=null, terminated=null,
waiting=ContainerStateWaiting(message=null, reason=ContainerCreating,
additionalProperties={}), additionalProperties={}),
additionalProperties={})]
2018-12-12 06:26:19 INFO  Client:54 - Waiting for application spark-pi to
finish...
2018-12-12 06:26:21 INFO  LoggingPodStatusWatcherImpl:54 - State changed,
new state:
         pod name: spark-pi-1544595975520-driver
         namespace: default
         labels: spark-app-selector ->
spark-ec5eb54644d348e7a213f8178b8ef61f, spark-role -> driver
         pod uid: d5d6bdc7-fdd6-11e8-b666-8e815d3815b2
         creation time: 2018-12-12T06:26:18Z
         service account name: spark
         volumes: spark-local-dir-1, spark-conf-volume, spark-token-qf9dt
         node name: flamboyant-darwin-3rhc
         start time: 2018-12-12T06:26:18Z
         container images: stevenstetzler/spark:v1
         phase: Running
         status:
[ContainerStatus(containerID=docker://b923c6ff02b93557c8c104c01a4eeb1c05f3d0c0123ec4e5895bfd6be398a03a,
image=stevenstetzler/spark:v1,
imageID=docker-pullable://stevenstetzler/spark@sha256:dc4bce1e410ebd7b14a88caa46a4282a61ff058c6374b7cf721b7498829bb041,
lastState=ContainerState(running=null, terminated=null, waiting=null,
additionalProperties={}), name=spark-kubernetes-driver, ready=true,
restartCount=0,
state=ContainerState(running=ContainerStateRunning(startedAt=Time(time=2018-12-12T06:26:21Z,
additionalProperties={}), additionalProperties={}), terminated=null,
waiting=null, additionalProperties={}), additionalProperties={})]
2018-12-12 06:26:24 INFO  LoggingPodStatusWatcherImpl:54 - State changed,
new state:
         pod name: spark-pi-1544595975520-driver
         namespace: default
         labels: spark-app-selector ->
spark-ec5eb54644d348e7a213f8178b8ef61f, spark-role -> driver
         pod uid: d5d6bdc7-fdd6-11e8-b666-8e815d3815b2
         creation time: 2018-12-12T06:26:18Z
         service account name: spark
         volumes: spark-local-dir-1, spark-conf-volume, spark-token-qf9dt
         node name: flamboyant-darwin-3rhc
         start time: 2018-12-12T06:26:18Z
         container images: stevenstetzler/spark:v1
         phase: Failed
         status:
[ContainerStatus(containerID=docker://b923c6ff02b93557c8c104c01a4eeb1c05f3d0c0123ec4e5895bfd6be398a03a,
image=stevenstetzler/spark:v1,
imageID=docker-pullable://stevenstetzler/spark@sha256:dc4bce1e410ebd7b14a88caa46a4282a61ff058c6374b7cf721b7498829bb041,
lastState=ContainerState(running=null, terminated=null, waiting=null,
additionalProperties={}), name=spark-kubernetes-driver, ready=false,
restartCount=0, state=ContainerState(running=null,
terminated=ContainerStateTerminated(containerID=docker://b923c6ff02b93557c8c104c01a4eeb1c05f3d0c0123ec4e5895bfd6be398a03a,
exitCode=1, finishedAt=Time(time=2018-12-12T06:26:24Z,
additionalProperties={}), message=null, reason=Error, signal=null,
startedAt=Time(time=2018-12-12T06:26:21Z, additionalProperties={}),
additionalProperties={}), waiting=null, additionalProperties={}),
additionalProperties={})]
2018-12-12 06:26:24 INFO  LoggingPodStatusWatcherImpl:54 - Container final
statuses:


         Container name: spark-kubernetes-driver
         Container image: stevenstetzler/spark:v1
         Container state: Terminated
         Exit code: 1
2018-12-12 06:26:24 INFO  Client:54 - Application spark-pi finished.
2018-12-12 06:26:24 INFO  ShutdownHookManager:54 - Shutdown hook called
2018-12-12 06:26:24 INFO  ShutdownHookManager:54 - Deleting directory
/tmp/spark-ca66ad4b-6526-4053-9ae3-f9ceb2e41cfd

When I check the logs with

kubectl logs spark-pi-1544595975520-driver

I get the following:

++ id -u
+ myuid=0
++ id -g
+ mygid=0
+ set +e
++ getent passwd 0
+ uidentry=root:x:0:0:root:/root:/bin/ash
+ set -e
+ '[' -z root:x:0:0:root:/root:/bin/ash ']'
+ SPARK_K8S_CMD=driver
+ case "$SPARK_K8S_CMD" in
+ shift 1
+ SPARK_CLASSPATH=':/opt/spark/jars/*'
+ env
+ grep SPARK_JAVA_OPT_
+ sed 's/[^=]*=\(.*\)/\1/g'
+ sort -t_ -k4 -n
+ readarray -t SPARK_EXECUTOR_JAVA_OPTS
+ '[' -n '' ']'
+ '[' -n '' ']'
+ PYSPARK_ARGS=
+ '[' -n '' ']'
+ R_ARGS=
+ '[' -n '' ']'
+ '[' '' == 2 ']'
+ '[' '' == 3 ']'
+ case "$SPARK_K8S_CMD" in
+ CMD=("$SPARK_HOME/bin/spark-submit" --conf
"spark.driver.bindAddress=$SPARK_DRIVER_BIND_ADDRESS" --deploy-mode client
"$@")
+ exec /sbin/tini -s -- /opt/spark/bin/spark-submit --conf
spark.driver.bindAddress=10.244.96.3 --deploy-mode client --properties-file
/opt/spark/conf/spark.properties --class org.apache.spark.examples.SparkPi
spark-internal
2018-12-12 06:26:22 WARN  NativeCodeLoader:62 - Unable to load
native-hadoop library for your platform... using builtin-java classes where
applicable
2018-12-12 06:26:22 INFO  SparkContext:54 - Running Spark version 2.4.0
2018-12-12 06:26:22 INFO  SparkContext:54 - Submitted application: Spark Pi
2018-12-12 06:26:22 INFO  SecurityManager:54 - Changing view acls to: root
2018-12-12 06:26:22 INFO  SecurityManager:54 - Changing modify acls to:
root
2018-12-12 06:26:22 INFO  SecurityManager:54 - Changing view acls groups
to:
2018-12-12 06:26:22 INFO  SecurityManager:54 - Changing modify acls groups
to:
2018-12-12 06:26:22 INFO  SecurityManager:54 - SecurityManager:
authentication disabled; ui acls disabled; users  with view permissions:
Set(root); groups with view permissions: Set(); users  with modify
permissions: Set(root); groups with modify permissions: Set()
2018-12-12 06:26:23 INFO  Utils:54 - Successfully started service
'sparkDriver' on port 7078.
2018-12-12 06:26:23 INFO  SparkEnv:54 - Registering MapOutputTracker
2018-12-12 06:26:23 INFO  SparkEnv:54 - Registering BlockManagerMaster
2018-12-12 06:26:23 INFO  BlockManagerMasterEndpoint:54 - Using
org.apache.spark.storage.DefaultTopologyMapper for getting topology
information
2018-12-12 06:26:23 INFO  BlockManagerMasterEndpoint:54 -
BlockManagerMasterEndpoint up
2018-12-12 06:26:23 INFO  DiskBlockManager:54 - Created local directory at
/var/data/spark-c306f3bc-cc7c-4090-a7f7-22fbc02b72bd/blockmgr-63358d1a-1e7e-493e-accd-7992da9b0adf
2018-12-12 06:26:23 INFO  MemoryStore:54 - MemoryStore started with
capacity 413.9 MB
2018-12-12 06:26:23 INFO  SparkEnv:54 - Registering OutputCommitCoordinator
2018-12-12 06:26:23 INFO  log:192 - Logging initialized @2059ms
2018-12-12 06:26:23 INFO  Server:351 - jetty-9.3.z-SNAPSHOT, build
timestamp: unknown, git hash: unknown
2018-12-12 06:26:23 INFO  Server:419 - Started @2155ms
2018-12-12 06:26:23 INFO  AbstractConnector:278 - Started
ServerConnector@7180e701{HTTP/1.1,[http/1.1]}{0.0.0.0:4040}
2018-12-12 06:26:23 INFO  Utils:54 - Successfully started service 'SparkUI'
on port 4040.
2018-12-12 06:26:23 INFO  ContextHandler:781 - Started
o.s.j.s.ServletContextHandler@9f6e406{/jobs,null,AVAILABLE,@Spark}
2018-12-12 06:26:23 INFO  ContextHandler:781 - Started
o.s.j.s.ServletContextHandler@3a60c416{/jobs/json,null,AVAILABLE,@Spark}
2018-12-12 06:26:23 INFO  ContextHandler:781 - Started
o.s.j.s.ServletContextHandler@60f2e0bd{/jobs/job,null,AVAILABLE,@Spark}
2018-12-12 06:26:23 INFO  ContextHandler:781 - Started
o.s.j.s.ServletContextHandler@6b410923
{/jobs/job/json,null,AVAILABLE,@Spark}
2018-12-12 06:26:23 INFO  ContextHandler:781 - Started
o.s.j.s.ServletContextHandler@57bd2029{/stages,null,AVAILABLE,@Spark}
2018-12-12 06:26:23 INFO  ContextHandler:781 - Started
o.s.j.s.ServletContextHandler@203dd56b{/stages/json,null,AVAILABLE,@Spark}
2018-12-12 06:26:23 INFO  ContextHandler:781 - Started
o.s.j.s.ServletContextHandler@e8fadb0{/stages/stage,null,AVAILABLE,@Spark}
2018-12-12 06:26:23 INFO  ContextHandler:781 - Started
o.s.j.s.ServletContextHandler@1d3e6d34
{/stages/stage/json,null,AVAILABLE,@Spark}
2018-12-12 06:26:23 INFO  ContextHandler:781 - Started
o.s.j.s.ServletContextHandler@6eafb10e{/stages/pool,null,AVAILABLE,@Spark}
2018-12-12 06:26:23 INFO  ContextHandler:781 - Started
o.s.j.s.ServletContextHandler@26a94fa5
{/stages/pool/json,null,AVAILABLE,@Spark}
2018-12-12 06:26:23 INFO  ContextHandler:781 - Started
o.s.j.s.ServletContextHandler@464a4442{/storage,null,AVAILABLE,@Spark}
2018-12-12 06:26:23 INFO  ContextHandler:781 - Started
o.s.j.s.ServletContextHandler@2873d672{/storage/json,null,AVAILABLE,@Spark}
2018-12-12 06:26:23 INFO  ContextHandler:781 - Started
o.s.j.s.ServletContextHandler@3bc735b3{/storage/rdd,null,AVAILABLE,@Spark}
2018-12-12 06:26:23 INFO  ContextHandler:781 - Started
o.s.j.s.ServletContextHandler@577f9109
{/storage/rdd/json,null,AVAILABLE,@Spark}
2018-12-12 06:26:23 INFO  ContextHandler:781 - Started
o.s.j.s.ServletContextHandler@4303b7f0{/environment,null,AVAILABLE,@Spark}
2018-12-12 06:26:23 INFO  ContextHandler:781 - Started
o.s.j.s.ServletContextHandler@757529a4
{/environment/json,null,AVAILABLE,@Spark}
2018-12-12 06:26:23 INFO  ContextHandler:781 - Started
o.s.j.s.ServletContextHandler@779de014{/executors,null,AVAILABLE,@Spark}
2018-12-12 06:26:23 INFO  ContextHandler:781 - Started
o.s.j.s.ServletContextHandler@5c41d037
{/executors/json,null,AVAILABLE,@Spark}
2018-12-12 06:26:23 INFO  ContextHandler:781 - Started
o.s.j.s.ServletContextHandler@2234078
{/executors/threadDump,null,AVAILABLE,@Spark}
2018-12-12 06:26:23 INFO  ContextHandler:781 - Started
o.s.j.s.ServletContextHandler@5ec77191
{/executors/threadDump/json,null,AVAILABLE,@Spark}
2018-12-12 06:26:23 INFO  ContextHandler:781 - Started
o.s.j.s.ServletContextHandler@4642b71d{/static,null,AVAILABLE,@Spark}
2018-12-12 06:26:23 INFO  ContextHandler:781 - Started
o.s.j.s.ServletContextHandler@7ed9ae94{/,null,AVAILABLE,@Spark}
2018-12-12 06:26:23 INFO  ContextHandler:781 - Started
o.s.j.s.ServletContextHandler@66908383{/api,null,AVAILABLE,@Spark}
2018-12-12 06:26:23 INFO  ContextHandler:781 - Started
o.s.j.s.ServletContextHandler@534ca02b
{/jobs/job/kill,null,AVAILABLE,@Spark}
2018-12-12 06:26:23 INFO  ContextHandler:781 - Started
o.s.j.s.ServletContextHandler@29a23c3d
{/stages/stage/kill,null,AVAILABLE,@Spark}
2018-12-12 06:26:23 INFO  SparkUI:54 - Bound SparkUI to 0.0.0.0, and
started at http://spark-pi-1544595975520-driver-svc.default.svc:4040
2018-12-12 06:26:23 INFO  SparkContext:54 - Added JAR
file:///opt/spark/examples/jars/spark-examples_2.11-2.4.0.jar at
spark://spark-pi-1544595975520-driver-svc.default.svc:7078/jars/spark-examples_2.11-2.4.0.jar
with timestamp 1544595983645
2018-12-12 06:26:23 ERROR SparkContext:91 - Error initializing
SparkContext.
org.apache.spark.SparkException: External scheduler cannot be instantiated
        at
org.apache.spark.SparkContext$.org$apache$spark$SparkContext$$createTaskScheduler(SparkContext.scala:2794)
        at org.apache.spark.SparkContext.<init>(SparkContext.scala:493)
        at
org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2520)
        at
org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:935)
        at
org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:926)
        at scala.Option.getOrElse(Option.scala:121)
        at
org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:926)
        at org.apache.spark.examples.SparkPi$.main(SparkPi.scala:31)
        at org.apache.spark.examples.SparkPi.main(SparkPi.scala)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at
org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
        at org.apache.spark.deploy.SparkSubmit.org
$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:849)
        at
org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:167)
        at
org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:195)
        at
org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
        at
org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:924)
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:933)
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: io.fabric8.kubernetes.client.KubernetesClientException: An error
has occurred.
        at
io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:62)
        at
io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:53)
        at
io.fabric8.kubernetes.client.utils.HttpClientUtils.createHttpClient(HttpClientUtils.java:167)
        at
org.apache.spark.deploy.k8s.SparkKubernetesClientFactory$.createKubernetesClient(SparkKubernetesClientFactory.scala:84)
        at
org.apache.spark.scheduler.cluster.k8s.KubernetesClusterManager.createSchedulerBackend(KubernetesClusterManager.scala:64)
        at
org.apache.spark.SparkContext$.org$apache$spark$SparkContext$$createTaskScheduler(SparkContext.scala:2788)
        ... 20 more
Caused by: java.security.cert.CertificateException: Could not parse
certificate: java.io.IOException: Empty input
        at
sun.security.provider.X509Factory.engineGenerateCertificate(X509Factory.java:110)
        at
java.security.cert.CertificateFactory.generateCertificate(CertificateFactory.java:339)
        at
io.fabric8.kubernetes.client.internal.CertUtils.createTrustStore(CertUtils.java:93)
        at
io.fabric8.kubernetes.client.internal.CertUtils.createTrustStore(CertUtils.java:71)
        at
io.fabric8.kubernetes.client.internal.SSLUtils.trustManagers(SSLUtils.java:114)
        at
io.fabric8.kubernetes.client.internal.SSLUtils.trustManagers(SSLUtils.java:93)
        at
io.fabric8.kubernetes.client.utils.HttpClientUtils.createHttpClient(HttpClientUtils.java:63)
        ... 23 more
Caused by: java.io.IOException: Empty input
        at
sun.security.provider.X509Factory.engineGenerateCertificate(X509Factory.java:106)
        ... 29 more
2018-12-12 06:26:23 INFO  AbstractConnector:318 - Stopped Spark@7180e701
{HTTP/1.1,[http/1.1]}{0.0.0.0:4040}
2018-12-12 06:26:23 INFO  SparkUI:54 - Stopped Spark web UI at
http://spark-pi-1544595975520-driver-svc.default.svc:4040
2018-12-12 06:26:23 INFO  MapOutputTrackerMasterEndpoint:54 -
MapOutputTrackerMasterEndpoint stopped!
2018-12-12 06:26:23 INFO  MemoryStore:54 - MemoryStore cleared
2018-12-12 06:26:23 INFO  BlockManager:54 - BlockManager stopped
2018-12-12 06:26:23 INFO  BlockManagerMaster:54 - BlockManagerMaster
stopped
2018-12-12 06:26:23 WARN  MetricsSystem:66 - Stopping a MetricsSystem that
is not running
2018-12-12 06:26:23 INFO
 OutputCommitCoordinator$OutputCommitCoordinatorEndpoint:54 -
OutputCommitCoordinator stopped!
2018-12-12 06:26:23 INFO  SparkContext:54 - Successfully stopped
SparkContext
Exception in thread "main" org.apache.spark.SparkException: External
scheduler cannot be instantiated
        at
org.apache.spark.SparkContext$.org$apache$spark$SparkContext$$createTaskScheduler(SparkContext.scala:2794)
        at org.apache.spark.SparkContext.<init>(SparkContext.scala:493)
        at
org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2520)
        at
org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:935)
        at
org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:926)
        at scala.Option.getOrElse(Option.scala:121)
        at
org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:926)
        at org.apache.spark.examples.SparkPi$.main(SparkPi.scala:31)
        at org.apache.spark.examples.SparkPi.main(SparkPi.scala)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at
org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
        at org.apache.spark.deploy.SparkSubmit.org
$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:849)
        at
org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:167)
        at
org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:195)
        at
org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
        at
org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:924)
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:933)
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: io.fabric8.kubernetes.client.KubernetesClientException: An error
has occurred.
        at
io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:62)
        at
io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:53)
        at
io.fabric8.kubernetes.client.utils.HttpClientUtils.createHttpClient(HttpClientUtils.java:167)
        at
org.apache.spark.deploy.k8s.SparkKubernetesClientFactory$.createKubernetesClient(SparkKubernetesClientFactory.scala:84)
        at
org.apache.spark.scheduler.cluster.k8s.KubernetesClusterManager.createSchedulerBackend(KubernetesClusterManager.scala:64)
        at
org.apache.spark.SparkContext$.org$apache$spark$SparkContext$$createTaskScheduler(SparkContext.scala:2788)
        ... 20 more
Caused by: java.security.cert.CertificateException: Could not parse
certificate: java.io.IOException: Empty input
        at
sun.security.provider.X509Factory.engineGenerateCertificate(X509Factory.java:110)
        at
java.security.cert.CertificateFactory.generateCertificate(CertificateFactory.java:339)
        at
io.fabric8.kubernetes.client.internal.CertUtils.createTrustStore(CertUtils.java:93)
        at
io.fabric8.kubernetes.client.internal.CertUtils.createTrustStore(CertUtils.java:71)
        at
io.fabric8.kubernetes.client.internal.SSLUtils.trustManagers(SSLUtils.java:114)
        at
io.fabric8.kubernetes.client.internal.SSLUtils.trustManagers(SSLUtils.java:93)
        at
io.fabric8.kubernetes.client.utils.HttpClientUtils.createHttpClient(HttpClientUtils.java:63)
        ... 23 more
Caused by: java.io.IOException: Empty input
        at
sun.security.provider.X509Factory.engineGenerateCertificate(X509Factory.java:106)
        ... 29 more
2018-12-12 06:26:23 INFO  ShutdownHookManager:54 - Shutdown hook called
2018-12-12 06:26:23 INFO  ShutdownHookManager:54 - Deleting directory
/var/data/spark-c306f3bc-cc7c-4090-a7f7-22fbc02b72bd/spark-157bde7a-d0e1-4092-9e39-5ab2b727e3e7
2018-12-12 06:26:23 INFO  ShutdownHookManager:54 - Deleting directory
/tmp/spark-07910ccb-c212-4622-9c1b-3e8b772f3faa

It looks to me like there is a communication / authentication issue during
pod creation, and it looks like there might be a certificate missing that
Spark needs to communicate with Kubernetes, which I infer from the
"java.security.cert.CertificateException: Could not parse certificate:
java.io.IOException: Empty input" error message.

Any ideas on what I can do to resolve this? Thanks!

Re: Problem running Spark on Kubernetes: Certificate error

Posted by Steven Stetzler <st...@gmail.com>.
Hi Matt,

Thank your for the help. This worked for me. For posterity: convert the
data in the certificate-authority-data field of your DigitalOcean
Kubernetes configuration file, which is downloaded from their site, from
base64 to PEM format and save it to a file ca.crt, then submit with --conf
spark.kubernetes.authenticate.driver.caCertFile=ca.crt.

As an extension to this question, I am also interested in using PySpark on
a JupyterHub deployment to interact with my cluster. The JupyterHub
deployment provides a Python (with PySpark) environment in a pod on a node
in the cluster, and I would like to create a SparkContext for computations
that uses the same cluster. I have attempted to replicate the above
solution there using:

from pyspark import *
> conf = SparkConf()
> conf.setMaster('k8s://<CLUSTERIP>')
> conf.set('spark.kubernetes.container.image', '<IMAGEURL>')
> conf.set('spark.kubernetes.authenticate.driver.caCertFile', 'ca.crt')
> sc = SparkContext(conf=conf)


which returns an identical error:

---------------------------------------------------------------------------
> Py4JJavaError Traceback (most recent call last)
> <ipython-input-4-bfc2d0364053> in <module>
> 22
> 23
> ---> 24 sc = SparkContext(conf=conf)
> /usr/local/spark/python/pyspark/context.py in __init__(self, master,
> appName, sparkHome, pyFiles, environment, batchSize, serializer, conf,
> gateway, jsc, profiler_cls)
> 116 try:
> 117 self._do_init(master, appName, sparkHome, pyFiles, environment,
> batchSize, serializer,
> --> 118 conf, jsc, profiler_cls) 119 except:
> 120 # If an error occurs, clean up in order to allow future SparkContext
> creation: /usr/local/spark/python/pyspark/context.py in _do_init(self,
> master, appName, sparkHome, pyFiles, environment, batchSize, serializer,
> conf, jsc, profiler_cls)
> 178
> 179 # Create the Java SparkContext through Py4J
> --> 180 self._jsc = jsc or self._initialize_context(self._conf._jconf)
> 181 # Reset the SparkConf to the one actually used by the SparkContext in
> JVM.
> 182 self._conf = SparkConf(_jconf=self._jsc.sc().conf())
> /usr/local/spark/python/pyspark/context.py in _initialize_context(self,
> jconf)
> 286 Initialize SparkContext in function to allow subclass specific
> initialization
> 287 """
> --> 288 return self._jvm.JavaSparkContext(jconf)
> 289
> 290 @classmethod
> /usr/local/spark/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py in
> __call__(self, *args)
> 1523 answer = self._gateway_client.send_command(command)
> 1524 return_value = get_return_value(
> -> 1525 answer, self._gateway_client, None, self._fqn) 1526
> 1527 for temp_arg in temp_args:
> /usr/local/spark/python/lib/py4j-0.10.7-src.zip/py4j/protocol.py in
> get_return_value(answer, gateway_client, target_id, name)
> 326 raise Py4JJavaError(
> 327 "An error occurred while calling {0}{1}{2}.\n".
> --> 328 format(target_id, ".", name), value) 329 else:
> 330 raise Py4JError( Py4JJavaError: An error occurred while calling
> None.org.apache.spark.api.java.JavaSparkContext.
> : org.apache.spark.SparkException: External scheduler cannot be
> instantiated
> at
> org.apache.spark.SparkContext$.org$apache$spark$SparkContext$$createTaskScheduler(SparkContext.scala:2794)
> at org.apache.spark.SparkContext.<init>(SparkContext.scala:493)
> at
> org.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:58)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:247)
> at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
> at py4j.Gateway.invoke(Gateway.java:238)
> at
> py4j.commands.ConstructorCommand.invokeConstructor(ConstructorCommand.java:80)
> at py4j.commands.ConstructorCommand.execute(ConstructorCommand.java:69)
> at py4j.GatewayConnection.run(GatewayConnection.java:238)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: io.fabric8.kubernetes.client.KubernetesClientException: An
> error has occurred.
> at
> io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:62)
> at
> io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:53)
> at
> io.fabric8.kubernetes.client.utils.HttpClientUtils.createHttpClient(HttpClientUtils.java:167)
> at
> org.apache.spark.deploy.k8s.SparkKubernetesClientFactory$.createKubernetesClient(SparkKubernetesClientFactory.scala:84)
> at
> org.apache.spark.scheduler.cluster.k8s.KubernetesClusterManager.createSchedulerBackend(KubernetesClusterManager.scala:64)
> at
> org.apache.spark.SparkContext$.org$apache$spark$SparkContext$$createTaskScheduler(SparkContext.scala:2788)
> ... 13 more
> Caused by: java.security.cert.CertificateException: Could not parse
> certificate: java.io.IOException: Empty input
> at
> sun.security.provider.X509Factory.engineGenerateCertificate(X509Factory.java:110)
> at
> java.security.cert.CertificateFactory.generateCertificate(CertificateFactory.java:339)
> at
> io.fabric8.kubernetes.client.internal.CertUtils.createTrustStore(CertUtils.java:93)
> at
> io.fabric8.kubernetes.client.internal.CertUtils.createTrustStore(CertUtils.java:71)
> at
> io.fabric8.kubernetes.client.internal.SSLUtils.trustManagers(SSLUtils.java:114)
> at
> io.fabric8.kubernetes.client.internal.SSLUtils.trustManagers(SSLUtils.java:93)
> at
> io.fabric8.kubernetes.client.utils.HttpClientUtils.createHttpClient(HttpClientUtils.java:63)
> ... 16 more
> Caused by: java.io.IOException: Empty input
> at
> sun.security.provider.X509Factory.engineGenerateCertificate(X509Factory.java:106)
> ... 22 more


On the other hand, I have no problem running

from pyspark import *
> sc = SparkContext()


and then performing a computation. However I believe that this will only be
able to use the resources provided to the Kubernetes pod that the Jupyter
environment is deployed on and not the whole cluster, so it's not really
what I'm looking for. I have tried running the same Python code from
outside of the cluster (on my own machine with PySpark installed). Note
that it doesn't matter what IP I use for the master, the same error occurs.
Running from outside of the cluster does seem to allow PySpark to
communicate with the cluster. However, there appear to be issues allocating
resources / creating workers. (But that seems like an issue for another
thread.)

Any idea what can be done here? Note that if it was really passing the file
and the file was not found, it would raise an error to indicate that. (I
have tried.) The error raised indicates instead that no file was ever
passed. I would assume that
conf.set('spark.kubernetes.authenticate.driver.caCertFile',
'ca.crt') would have identical behavior to --conf
spark.kubernetes.authenticate.driver.caCertFile=ca.crt in spark-submit.
Sorry if I am misunderstanding something basic in how PySpark and
spark-submit work.

Thanks,
Steven


On Thu, Dec 13, 2018 at 7:59 PM Matt Cheah <mc...@palantir.com> wrote:

> Hi Steven,
>
>
>
> What I think is happening is that your machine has a CA certificate that
> is used for communicating with your API server, particularly because you’re
> using Digital Ocean’s cluster manager. However, it’s unclear if your pod
> has the same CA certificate or if the pod needs that certificate file. You
> can use configurations to have your pod use a particular CA certificate
> file to communicate with the APi server. If you set spark.kubernetes.authenticate.driver.caCertFile
> to the path of your CA certificate on your local disk, spark-submit will
> create a secret that contains that certificate file and use that
> certificate to configure TLS for the driver pod’s communication with the
> API server.
>
>
>
> It's clear that your driver pod doesn’t have the right TLS certificate to
> communicate with the API server, so I would try to introspect the driver
> pod and see what certificate it’s using for that communication. If there’s
> a fix that needs to happen in Spark, feel free to indicate as such.
>
>
>
> -Matt Cheah
>
>
>
> *From: *Steven Stetzler <st...@gmail.com>
> *Date: *Thursday, December 13, 2018 at 1:49 PM
> *To: *"user@spark.apache.org" <us...@spark.apache.org>
> *Subject: *Problem running Spark on Kubernetes: Certificate error
>
>
>
> Hello,
>
> I am following the tutorial here (https://spark.apache.org/docs/latest/running-on-kubernetes.html
> [spark.apache.org]
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__spark.apache.org_docs_latest_running-2Don-2Dkubernetes.html&d=DwMFaQ&c=izlc9mHr637UR4lpLEZLFFS3Vn2UXBrZ4tFb6oOnmz8&r=hzwIMNQ9E99EMYGuqHI0kXhVbvX3nU3OSDadUnJxjAs&m=koBmBBRy1RJd_Z00W1RO9fi0a72z2vYhFoj-O0FN3RQ&s=9gQXKMnjixgZA7jGBSWFYdhjnBsgzdy51B5bZJ5znQ8&e=>)
> to get spark running on a Kubernetes cluster. My Kubernetes cluster is
> hosted with Digital Ocean's kubernetes cluster manager. I have change the
> KUBECONFIG environment variable to point to my cluster access credentials,
> so both Spark and kubectl can speak with the nodes.
>
> I am running into an issue when trying to run the SparkPi example as
> described in the Spark on Kubernetes tutorials. The command I am running
> is:
>
> ./bin/spark-submit --master k8s://$CLUSTERIP --deploy-mode cluster --name
> spark-pi --class org.apache.spark.examples.SparkPi --conf
> spark.executor.instances=1 --conf
> spark.kubernetes.container.image=$IMAGEURL --conf
> spark.kubernetes.authenticate.driver.serviceAccountName=spark
> local:///opt/spark/examples/jars/spark-examples_2.11-2.4.0.jar
>
> where CLUSTERIP contains the ip of my cluster and IMAGEURL contains the
> URL of the Spark docker image I am using (https://hub.docker.com/r/stevenstetzler/spark/
> [hub.docker.com]
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__hub.docker.com_r_stevenstetzler_spark_&d=DwMFaQ&c=izlc9mHr637UR4lpLEZLFFS3Vn2UXBrZ4tFb6oOnmz8&r=hzwIMNQ9E99EMYGuqHI0kXhVbvX3nU3OSDadUnJxjAs&m=koBmBBRy1RJd_Z00W1RO9fi0a72z2vYhFoj-O0FN3RQ&s=lgzK2iTO0oHaHnD6w81b-Cwl8YAAK5XyddxSBCi35j0&e=>).
> This docker image was built and pushed with the script included in the
> Spark 2.4 distribution. I have created a service account for Spark to
> ensure that it has proper permissions to create pods etc., which I checked
> using
>
> kubectl auth can-i create pods --as=system:serviceaccount:default:spark
>
> When I try to run the SparkPi example using the above command, I get the
> following output:
>
> 2018-12-12 06:26:15 WARN  Utils:66 - Your hostname, docker-test resolves
> to a loopback address: 127.0.1.1; using 10.46.0.6 instead (on interface
> eth0)
> 2018-12-12 06:26:15 WARN  Utils:66 - Set SPARK_LOCAL_IP if you need to
> bind to another address
> 2018-12-12 06:26:19 INFO  LoggingPodStatusWatcherImpl:54 - State changed,
> new state:
>          pod name: spark-pi-1544595975520-driver
>          namespace: default
>          labels: spark-app-selector ->
> spark-ec5eb54644d348e7a213f8178b8ef61f, spark-role -> driver
>          pod uid: d5d6bdc7-fdd6-11e8-b666-8e815d3815b2
>          creation time: 2018-12-12T06:26:18Z
>          service account name: spark
>          volumes: spark-local-dir-1, spark-conf-volume, spark-token-qf9dt
>          node name: N/A
>          start time: N/A
>          container images: N/A
>          phase: Pending
>          status: []
> 2018-12-12 06:26:19 INFO  LoggingPodStatusWatcherImpl:54 - State changed,
> new state:
>          pod name: spark-pi-1544595975520-driver
>          namespace: default
>          labels: spark-app-selector ->
> spark-ec5eb54644d348e7a213f8178b8ef61f, spark-role -> driver
>          pod uid: d5d6bdc7-fdd6-11e8-b666-8e815d3815b2
>          creation time: 2018-12-12T06:26:18Z
>          service account name: spark
>          volumes: spark-local-dir-1, spark-conf-volume, spark-token-qf9dt
>          node name: flamboyant-darwin-3rhc
>          start time: N/A
>          container images: N/A
>          phase: Pending
>          status: []
> 2018-12-12 06:26:19 INFO  LoggingPodStatusWatcherImpl:54 - State changed,
> new state:
>          pod name: spark-pi-1544595975520-driver
>          namespace: default
>          labels: spark-app-selector ->
> spark-ec5eb54644d348e7a213f8178b8ef61f, spark-role -> driver
>          pod uid: d5d6bdc7-fdd6-11e8-b666-8e815d3815b2
>          creation time: 2018-12-12T06:26:18Z
>          service account name: spark
>          volumes: spark-local-dir-1, spark-conf-volume, spark-token-qf9dt
>          node name: flamboyant-darwin-3rhc
>          start time: 2018-12-12T06:26:18Z
>          container images: docker.io/stevenstetzler/spark:v1 [docker.io]
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__docker.io_stevenstetzler_spark-3Av1&d=DwMFaQ&c=izlc9mHr637UR4lpLEZLFFS3Vn2UXBrZ4tFb6oOnmz8&r=hzwIMNQ9E99EMYGuqHI0kXhVbvX3nU3OSDadUnJxjAs&m=koBmBBRy1RJd_Z00W1RO9fi0a72z2vYhFoj-O0FN3RQ&s=O4l8wZMrO52IV576ItzHzN_REr0NCrRNnZ-0ZCB30a0&e=>
>
>          phase: Pending
>          status: [ContainerStatus(containerID=null, image=docker.io/stevenstetzler/spark:v1
> [docker.io]
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__docker.io_stevenstetzler_spark-3Av1&d=DwMFaQ&c=izlc9mHr637UR4lpLEZLFFS3Vn2UXBrZ4tFb6oOnmz8&r=hzwIMNQ9E99EMYGuqHI0kXhVbvX3nU3OSDadUnJxjAs&m=koBmBBRy1RJd_Z00W1RO9fi0a72z2vYhFoj-O0FN3RQ&s=O4l8wZMrO52IV576ItzHzN_REr0NCrRNnZ-0ZCB30a0&e=>,
> imageID=, lastState=ContainerState(running=null, terminated=null,
> waiting=null, additionalProperties={}), name=spark-kubernetes-driver,
> ready=false, restartCount=0, state=ContainerState(running=null,
> terminated=null, waiting=ContainerStateWaiting(message=null,
> reason=ContainerCreating, additionalProperties={}),
> additionalProperties={}), additionalProperties={})]
> 2018-12-12 06:26:19 INFO  Client:54 - Waiting for application spark-pi to
> finish...
> 2018-12-12 06:26:21 INFO  LoggingPodStatusWatcherImpl:54 - State changed,
> new state:
>          pod name: spark-pi-1544595975520-driver
>          namespace: default
>          labels: spark-app-selector ->
> spark-ec5eb54644d348e7a213f8178b8ef61f, spark-role -> driver
>          pod uid: d5d6bdc7-fdd6-11e8-b666-8e815d3815b2
>          creation time: 2018-12-12T06:26:18Z
>          service account name: spark
>          volumes: spark-local-dir-1, spark-conf-volume, spark-token-qf9dt
>          node name: flamboyant-darwin-3rhc
>          start time: 2018-12-12T06:26:18Z
>          container images: stevenstetzler/spark:v1
>          phase: Running
>          status:
> [ContainerStatus(containerID=docker://b923c6ff02b93557c8c104c01a4eeb1c05f3d0c0123ec4e5895bfd6be398a03a,
> image=stevenstetzler/spark:v1,
> imageID=docker-pullable://stevenstetzler/spark@sha256:dc4bce1e410ebd7b14a88caa46a4282a61ff058c6374b7cf721b7498829bb041,
> lastState=ContainerState(running=null, terminated=null, waiting=null,
> additionalProperties={}), name=spark-kubernetes-driver, ready=true,
> restartCount=0,
> state=ContainerState(running=ContainerStateRunning(startedAt=Time(time=2018-12-12T06:26:21Z,
> additionalProperties={}), additionalProperties={}), terminated=null,
> waiting=null, additionalProperties={}), additionalProperties={})]
> 2018-12-12 06:26:24 INFO  LoggingPodStatusWatcherImpl:54 - State changed,
> new state:
>          pod name: spark-pi-1544595975520-driver
>          namespace: default
>          labels: spark-app-selector ->
> spark-ec5eb54644d348e7a213f8178b8ef61f, spark-role -> driver
>          pod uid: d5d6bdc7-fdd6-11e8-b666-8e815d3815b2
>          creation time: 2018-12-12T06:26:18Z
>          service account name: spark
>          volumes: spark-local-dir-1, spark-conf-volume, spark-token-qf9dt
>          node name: flamboyant-darwin-3rhc
>          start time: 2018-12-12T06:26:18Z
>          container images: stevenstetzler/spark:v1
>          phase: Failed
>          status:
> [ContainerStatus(containerID=docker://b923c6ff02b93557c8c104c01a4eeb1c05f3d0c0123ec4e5895bfd6be398a03a,
> image=stevenstetzler/spark:v1,
> imageID=docker-pullable://stevenstetzler/spark@sha256:dc4bce1e410ebd7b14a88caa46a4282a61ff058c6374b7cf721b7498829bb041,
> lastState=ContainerState(running=null, terminated=null, waiting=null,
> additionalProperties={}), name=spark-kubernetes-driver, ready=false,
> restartCount=0, state=ContainerState(running=null,
> terminated=ContainerStateTerminated(containerID=docker://b923c6ff02b93557c8c104c01a4eeb1c05f3d0c0123ec4e5895bfd6be398a03a,
> exitCode=1, finishedAt=Time(time=2018-12-12T06:26:24Z,
> additionalProperties={}), message=null, reason=Error, signal=null,
> startedAt=Time(time=2018-12-12T06:26:21Z, additionalProperties={}),
> additionalProperties={}), waiting=null, additionalProperties={}),
> additionalProperties={})]
> 2018-12-12 06:26:24 INFO  LoggingPodStatusWatcherImpl:54 - Container final
> statuses:
>
>
>          Container name: spark-kubernetes-driver
>          Container image: stevenstetzler/spark:v1
>          Container state: Terminated
>          Exit code: 1
> 2018-12-12 06:26:24 INFO  Client:54 - Application spark-pi finished.
> 2018-12-12 06:26:24 INFO  ShutdownHookManager:54 - Shutdown hook called
> 2018-12-12 06:26:24 INFO  ShutdownHookManager:54 - Deleting directory
> /tmp/spark-ca66ad4b-6526-4053-9ae3-f9ceb2e41cfd
>
> When I check the logs with
>
> kubectl logs spark-pi-1544595975520-driver
>
> I get the following:
>
> ++ id -u
> + myuid=0
> ++ id -g
> + mygid=0
> + set +e
> ++ getent passwd 0
> + uidentry=root:x:0:0:root:/root:/bin/ash
> + set -e
> + '[' -z root:x:0:0:root:/root:/bin/ash ']'
> + SPARK_K8S_CMD=driver
> + case "$SPARK_K8S_CMD" in
> + shift 1
> + SPARK_CLASSPATH=':/opt/spark/jars/*'
> + env
> + grep SPARK_JAVA_OPT_
> + sed 's/[^=]*=\(.*\)/\1/g'
> + sort -t_ -k4 -n
> + readarray -t SPARK_EXECUTOR_JAVA_OPTS
> + '[' -n '' ']'
> + '[' -n '' ']'
> + PYSPARK_ARGS=
> + '[' -n '' ']'
> + R_ARGS=
> + '[' -n '' ']'
> + '[' '' == 2 ']'
> + '[' '' == 3 ']'
> + case "$SPARK_K8S_CMD" in
> + CMD=("$SPARK_HOME/bin/spark-submit" --conf
> "spark.driver.bindAddress=$SPARK_DRIVER_BIND_ADDRESS" --deploy-mode client
> "$@")
> + exec /sbin/tini -s -- /opt/spark/bin/spark-submit --conf
> spark.driver.bindAddress=10.244.96.3 --deploy-mode client --properties-file
> /opt/spark/conf/spark.properties --class org.apache.spark.examples.SparkPi
> spark-internal
> 2018-12-12 06:26:22 WARN  NativeCodeLoader:62 - Unable to load
> native-hadoop library for your platform... using builtin-java classes where
> applicable
> 2018-12-12 06:26:22 INFO  SparkContext:54 - Running Spark version 2.4.0
> 2018-12-12 06:26:22 INFO  SparkContext:54 - Submitted application: Spark
> Pi
> 2018-12-12 06:26:22 INFO  SecurityManager:54 - Changing view acls to: root
> 2018-12-12 06:26:22 INFO  SecurityManager:54 - Changing modify acls to:
> root
> 2018-12-12 06:26:22 INFO  SecurityManager:54 - Changing view acls groups
> to:
> 2018-12-12 06:26:22 INFO  SecurityManager:54 - Changing modify acls groups
> to:
> 2018-12-12 06:26:22 INFO  SecurityManager:54 - SecurityManager:
> authentication disabled; ui acls disabled; users  with view permissions:
> Set(root); groups with view permissions: Set(); users  with modify
> permissions: Set(root); groups with modify permissions: Set()
> 2018-12-12 06:26:23 INFO  Utils:54 - Successfully started service
> 'sparkDriver' on port 7078.
> 2018-12-12 06:26:23 INFO  SparkEnv:54 - Registering MapOutputTracker
> 2018-12-12 06:26:23 INFO  SparkEnv:54 - Registering BlockManagerMaster
> 2018-12-12 06:26:23 INFO  BlockManagerMasterEndpoint:54 - Using
> org.apache.spark.storage.DefaultTopologyMapper for getting topology
> information
> 2018-12-12 06:26:23 INFO  BlockManagerMasterEndpoint:54 -
> BlockManagerMasterEndpoint up
> 2018-12-12 06:26:23 INFO  DiskBlockManager:54 - Created local directory at
> /var/data/spark-c306f3bc-cc7c-4090-a7f7-22fbc02b72bd/blockmgr-63358d1a-1e7e-493e-accd-7992da9b0adf
> 2018-12-12 06:26:23 INFO  MemoryStore:54 - MemoryStore started with
> capacity 413.9 MB
> 2018-12-12 06:26:23 INFO  SparkEnv:54 - Registering
> OutputCommitCoordinator
> 2018-12-12 06:26:23 INFO  log:192 - Logging initialized @2059ms
> 2018-12-12 06:26:23 INFO  Server:351 - jetty-9.3.z-SNAPSHOT, build
> timestamp: unknown, git hash: unknown
> 2018-12-12 06:26:23 INFO  Server:419 - Started @2155ms
> 2018-12-12 06:26:23 INFO  AbstractConnector:278 - Started
> ServerConnector@7180e701{HTTP/1.1,[http/1.1]}{0.0.0.0:4040 [0.0.0.0]
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__0.0.0.0-3A4040&d=DwMFaQ&c=izlc9mHr637UR4lpLEZLFFS3Vn2UXBrZ4tFb6oOnmz8&r=hzwIMNQ9E99EMYGuqHI0kXhVbvX3nU3OSDadUnJxjAs&m=koBmBBRy1RJd_Z00W1RO9fi0a72z2vYhFoj-O0FN3RQ&s=NEpJ4w980MUskRPIbAZm04Qx67LafCpnymbJwyzat4Y&e=>
> }
> 2018-12-12 06:26:23 INFO  Utils:54 - Successfully started service
> 'SparkUI' on port 4040.
> 2018-12-12 06:26:23 INFO  ContextHandler:781 - Started
> o.s.j.s.ServletContextHandler@9f6e406{/jobs,null,AVAILABLE,@Spark}
> 2018-12-12 06:26:23 INFO  ContextHandler:781 - Started
> o.s.j.s.ServletContextHandler@3a60c416{/jobs/json,null,AVAILABLE,@Spark}
> 2018-12-12 06:26:23 INFO  ContextHandler:781 - Started
> o.s.j.s.ServletContextHandler@60f2e0bd{/jobs/job,null,AVAILABLE,@Spark}
> 2018-12-12 06:26:23 INFO  ContextHandler:781 - Started
> o.s.j.s.ServletContextHandler@6b410923
> {/jobs/job/json,null,AVAILABLE,@Spark}
> 2018-12-12 06:26:23 INFO  ContextHandler:781 - Started
> o.s.j.s.ServletContextHandler@57bd2029{/stages,null,AVAILABLE,@Spark}
> 2018-12-12 06:26:23 INFO  ContextHandler:781 - Started
> o.s.j.s.ServletContextHandler@203dd56b
> {/stages/json,null,AVAILABLE,@Spark}
> 2018-12-12 06:26:23 INFO  ContextHandler:781 - Started
> o.s.j.s.ServletContextHandler@e8fadb0
> {/stages/stage,null,AVAILABLE,@Spark}
> 2018-12-12 06:26:23 INFO  ContextHandler:781 - Started
> o.s.j.s.ServletContextHandler@1d3e6d34
> {/stages/stage/json,null,AVAILABLE,@Spark}
> 2018-12-12 06:26:23 INFO  ContextHandler:781 - Started
> o.s.j.s.ServletContextHandler@6eafb10e
> {/stages/pool,null,AVAILABLE,@Spark}
> 2018-12-12 06:26:23 INFO  ContextHandler:781 - Started
> o.s.j.s.ServletContextHandler@26a94fa5
> {/stages/pool/json,null,AVAILABLE,@Spark}
> 2018-12-12 06:26:23 INFO  ContextHandler:781 - Started
> o.s.j.s.ServletContextHandler@464a4442{/storage,null,AVAILABLE,@Spark}
> 2018-12-12 06:26:23 INFO  ContextHandler:781 - Started
> o.s.j.s.ServletContextHandler@2873d672
> {/storage/json,null,AVAILABLE,@Spark}
> 2018-12-12 06:26:23 INFO  ContextHandler:781 - Started
> o.s.j.s.ServletContextHandler@3bc735b3
> {/storage/rdd,null,AVAILABLE,@Spark}
> 2018-12-12 06:26:23 INFO  ContextHandler:781 - Started
> o.s.j.s.ServletContextHandler@577f9109
> {/storage/rdd/json,null,AVAILABLE,@Spark}
> 2018-12-12 06:26:23 INFO  ContextHandler:781 - Started
> o.s.j.s.ServletContextHandler@4303b7f0
> {/environment,null,AVAILABLE,@Spark}
> 2018-12-12 06:26:23 INFO  ContextHandler:781 - Started
> o.s.j.s.ServletContextHandler@757529a4
> {/environment/json,null,AVAILABLE,@Spark}
> 2018-12-12 06:26:23 INFO  ContextHandler:781 - Started
> o.s.j.s.ServletContextHandler@779de014{/executors,null,AVAILABLE,@Spark}
> 2018-12-12 06:26:23 INFO  ContextHandler:781 - Started
> o.s.j.s.ServletContextHandler@5c41d037
> {/executors/json,null,AVAILABLE,@Spark}
> 2018-12-12 06:26:23 INFO  ContextHandler:781 - Started
> o.s.j.s.ServletContextHandler@2234078
> {/executors/threadDump,null,AVAILABLE,@Spark}
> 2018-12-12 06:26:23 INFO  ContextHandler:781 - Started
> o.s.j.s.ServletContextHandler@5ec77191
> {/executors/threadDump/json,null,AVAILABLE,@Spark}
> 2018-12-12 06:26:23 INFO  ContextHandler:781 - Started
> o.s.j.s.ServletContextHandler@4642b71d{/static,null,AVAILABLE,@Spark}
> 2018-12-12 06:26:23 INFO  ContextHandler:781 - Started
> o.s.j.s.ServletContextHandler@7ed9ae94{/,null,AVAILABLE,@Spark}
> 2018-12-12 06:26:23 INFO  ContextHandler:781 - Started
> o.s.j.s.ServletContextHandler@66908383{/api,null,AVAILABLE,@Spark}
> 2018-12-12 06:26:23 INFO  ContextHandler:781 - Started
> o.s.j.s.ServletContextHandler@534ca02b
> {/jobs/job/kill,null,AVAILABLE,@Spark}
> 2018-12-12 06:26:23 INFO  ContextHandler:781 - Started
> o.s.j.s.ServletContextHandler@29a23c3d
> {/stages/stage/kill,null,AVAILABLE,@Spark}
> 2018-12-12 06:26:23 INFO  SparkUI:54 - Bound SparkUI to 0.0.0.0, and
> started at http://spark-pi-1544595975520-driver-svc.default.svc:4040
> [spark-pi-1544595975520-driver-svc.default.svc]
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__spark-2Dpi-2D1544595975520-2Ddriver-2Dsvc.default.svc-3A4040_&d=DwMFaQ&c=izlc9mHr637UR4lpLEZLFFS3Vn2UXBrZ4tFb6oOnmz8&r=hzwIMNQ9E99EMYGuqHI0kXhVbvX3nU3OSDadUnJxjAs&m=koBmBBRy1RJd_Z00W1RO9fi0a72z2vYhFoj-O0FN3RQ&s=7_pmhkpOYkR7IRMQolPOXsfzMG9byW5WEPVrfs4UgCg&e=>
> 2018-12-12 06:26:23 INFO  SparkContext:54 - Added JAR
> file:///opt/spark/examples/jars/spark-examples_2.11-2.4.0.jar at
> spark://spark-pi-1544595975520-driver-svc.default.svc:7078/jars/spark-examples_2.11-2.4.0.jar
> with timestamp 1544595983645
> 2018-12-12 06:26:23 ERROR SparkContext:91 - Error initializing
> SparkContext.
> org.apache.spark.SparkException: External scheduler cannot be instantiated
>         at
> org.apache.spark.SparkContext$.org$apache$spark$SparkContext$$createTaskScheduler(SparkContext.scala:2794)
>         at org.apache.spark.SparkContext.<init>(SparkContext.scala:493)
>         at
> org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2520)
>         at
> org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:935)
>         at
> org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:926)
>         at scala.Option.getOrElse(Option.scala:121)
>         at
> org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:926)
>         at org.apache.spark.examples.SparkPi$.main(SparkPi.scala:31)
>         at org.apache.spark.examples.SparkPi.main(SparkPi.scala)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>         at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:498)
>         at
> org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
>         at org.apache.spark.deploy.SparkSubmit.org
> [org.apache.spark.deploy.sparksubmit.org]
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__org.apache.spark.deploy.SparkSubmit.org&d=DwMFaQ&c=izlc9mHr637UR4lpLEZLFFS3Vn2UXBrZ4tFb6oOnmz8&r=hzwIMNQ9E99EMYGuqHI0kXhVbvX3nU3OSDadUnJxjAs&m=koBmBBRy1RJd_Z00W1RO9fi0a72z2vYhFoj-O0FN3RQ&s=UFf1bgM3AcQSaqrUz94wpKROIXo4lHTVmIa6KAOv2vo&e=>
> $apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:849)
>         at
> org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:167)
>         at
> org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:195)
>         at
> org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
>         at
> org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:924)
>         at
> org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:933)
>         at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
> Caused by: io.fabric8.kubernetes.client.KubernetesClientException: An
> error has occurred.
>         at
> io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:62)
>         at
> io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:53)
>         at
> io.fabric8.kubernetes.client.utils.HttpClientUtils.createHttpClient(HttpClientUtils.java:167)
>         at
> org.apache.spark.deploy.k8s.SparkKubernetesClientFactory$.createKubernetesClient(SparkKubernetesClientFactory.scala:84)
>         at
> org.apache.spark.scheduler.cluster.k8s.KubernetesClusterManager.createSchedulerBackend(KubernetesClusterManager.scala:64)
>         at
> org.apache.spark.SparkContext$.org$apache$spark$SparkContext$$createTaskScheduler(SparkContext.scala:2788)
>         ... 20 more
> Caused by: java.security.cert.CertificateException: Could not parse
> certificate: java.io.IOException: Empty input
>         at
> sun.security.provider.X509Factory.engineGenerateCertificate(X509Factory.java:110)
>         at
> java.security.cert.CertificateFactory.generateCertificate(CertificateFactory.java:339)
>         at
> io.fabric8.kubernetes.client.internal.CertUtils.createTrustStore(CertUtils.java:93)
>         at
> io.fabric8.kubernetes.client.internal.CertUtils.createTrustStore(CertUtils.java:71)
>         at
> io.fabric8.kubernetes.client.internal.SSLUtils.trustManagers(SSLUtils.java:114)
>         at
> io.fabric8.kubernetes.client.internal.SSLUtils.trustManagers(SSLUtils.java:93)
>         at
> io.fabric8.kubernetes.client.utils.HttpClientUtils.createHttpClient(HttpClientUtils.java:63)
>         ... 23 more
> Caused by: java.io.IOException: Empty input
>         at
> sun.security.provider.X509Factory.engineGenerateCertificate(X509Factory.java:106)
>         ... 29 more
> 2018-12-12 06:26:23 INFO  AbstractConnector:318 - Stopped Spark@7180e701
> {HTTP/1.1,[http/1.1]}{0.0.0.0:4040 [0.0.0.0]
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__0.0.0.0-3A4040&d=DwMFaQ&c=izlc9mHr637UR4lpLEZLFFS3Vn2UXBrZ4tFb6oOnmz8&r=hzwIMNQ9E99EMYGuqHI0kXhVbvX3nU3OSDadUnJxjAs&m=koBmBBRy1RJd_Z00W1RO9fi0a72z2vYhFoj-O0FN3RQ&s=NEpJ4w980MUskRPIbAZm04Qx67LafCpnymbJwyzat4Y&e=>
> }
> 2018-12-12 06:26:23 INFO  SparkUI:54 - Stopped Spark web UI at http://spark-pi-1544595975520-driver-svc.default.svc:4040
> [spark-pi-1544595975520-driver-svc.default.svc]
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__spark-2Dpi-2D1544595975520-2Ddriver-2Dsvc.default.svc-3A4040_&d=DwMFaQ&c=izlc9mHr637UR4lpLEZLFFS3Vn2UXBrZ4tFb6oOnmz8&r=hzwIMNQ9E99EMYGuqHI0kXhVbvX3nU3OSDadUnJxjAs&m=koBmBBRy1RJd_Z00W1RO9fi0a72z2vYhFoj-O0FN3RQ&s=7_pmhkpOYkR7IRMQolPOXsfzMG9byW5WEPVrfs4UgCg&e=>
> 2018-12-12 06:26:23 INFO  MapOutputTrackerMasterEndpoint:54 -
> MapOutputTrackerMasterEndpoint stopped!
> 2018-12-12 06:26:23 INFO  MemoryStore:54 - MemoryStore cleared
> 2018-12-12 06:26:23 INFO  BlockManager:54 - BlockManager stopped
> 2018-12-12 06:26:23 INFO  BlockManagerMaster:54 - BlockManagerMaster
> stopped
> 2018-12-12 06:26:23 WARN  MetricsSystem:66 - Stopping a MetricsSystem that
> is not running
> 2018-12-12 06:26:23 INFO
>  OutputCommitCoordinator$OutputCommitCoordinatorEndpoint:54 -
> OutputCommitCoordinator stopped!
> 2018-12-12 06:26:23 INFO  SparkContext:54 - Successfully stopped
> SparkContext
> Exception in thread "main" org.apache.spark.SparkException: External
> scheduler cannot be instantiated
>         at
> org.apache.spark.SparkContext$.org$apache$spark$SparkContext$$createTaskScheduler(SparkContext.scala:2794)
>         at org.apache.spark.SparkContext.<init>(SparkContext.scala:493)
>         at
> org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2520)
>         at
> org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:935)
>         at
> org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:926)
>         at scala.Option.getOrElse(Option.scala:121)
>         at
> org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:926)
>         at org.apache.spark.examples.SparkPi$.main(SparkPi.scala:31)
>         at org.apache.spark.examples.SparkPi.main(SparkPi.scala)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>         at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:498)
>         at
> org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
>         at org.apache.spark.deploy.SparkSubmit.org
> [org.apache.spark.deploy.sparksubmit.org]
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__org.apache.spark.deploy.SparkSubmit.org&d=DwMFaQ&c=izlc9mHr637UR4lpLEZLFFS3Vn2UXBrZ4tFb6oOnmz8&r=hzwIMNQ9E99EMYGuqHI0kXhVbvX3nU3OSDadUnJxjAs&m=koBmBBRy1RJd_Z00W1RO9fi0a72z2vYhFoj-O0FN3RQ&s=UFf1bgM3AcQSaqrUz94wpKROIXo4lHTVmIa6KAOv2vo&e=>
> $apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:849)
>         at
> org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:167)
>         at
> org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:195)
>         at
> org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
>         at
> org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:924)
>         at
> org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:933)
>         at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
> Caused by: io.fabric8.kubernetes.client.KubernetesClientException: An
> error has occurred.
>         at
> io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:62)
>         at
> io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:53)
>         at
> io.fabric8.kubernetes.client.utils.HttpClientUtils.createHttpClient(HttpClientUtils.java:167)
>         at
> org.apache.spark.deploy.k8s.SparkKubernetesClientFactory$.createKubernetesClient(SparkKubernetesClientFactory.scala:84)
>         at
> org.apache.spark.scheduler.cluster.k8s.KubernetesClusterManager.createSchedulerBackend(KubernetesClusterManager.scala:64)
>         at
> org.apache.spark.SparkContext$.org$apache$spark$SparkContext$$createTaskScheduler(SparkContext.scala:2788)
>         ... 20 more
> Caused by: java.security.cert.CertificateException: Could not parse
> certificate: java.io.IOException: Empty input
>         at
> sun.security.provider.X509Factory.engineGenerateCertificate(X509Factory.java:110)
>         at
> java.security.cert.CertificateFactory.generateCertificate(CertificateFactory.java:339)
>         at
> io.fabric8.kubernetes.client.internal.CertUtils.createTrustStore(CertUtils.java:93)
>         at
> io.fabric8.kubernetes.client.internal.CertUtils.createTrustStore(CertUtils.java:71)
>         at
> io.fabric8.kubernetes.client.internal.SSLUtils.trustManagers(SSLUtils.java:114)
>         at
> io.fabric8.kubernetes.client.internal.SSLUtils.trustManagers(SSLUtils.java:93)
>         at
> io.fabric8.kubernetes.client.utils.HttpClientUtils.createHttpClient(HttpClientUtils.java:63)
>         ... 23 more
> Caused by: java.io.IOException: Empty input
>         at
> sun.security.provider.X509Factory.engineGenerateCertificate(X509Factory.java:106)
>         ... 29 more
> 2018-12-12 06:26:23 INFO  ShutdownHookManager:54 - Shutdown hook called
> 2018-12-12 06:26:23 INFO  ShutdownHookManager:54 - Deleting directory
> /var/data/spark-c306f3bc-cc7c-4090-a7f7-22fbc02b72bd/spark-157bde7a-d0e1-4092-9e39-5ab2b727e3e7
> 2018-12-12 06:26:23 INFO  ShutdownHookManager:54 - Deleting directory
> /tmp/spark-07910ccb-c212-4622-9c1b-3e8b772f3faa
>
> It looks to me like there is a communication / authentication issue during
> pod creation, and it looks like there might be a certificate missing that
> Spark needs to communicate with Kubernetes, which I infer from the
> "java.security.cert.CertificateException: Could not parse certificate:
> java.io.IOException: Empty input" error message.
>
> Any ideas on what I can do to resolve this? Thanks!
>

Re: Problem running Spark on Kubernetes: Certificate error

Posted by Matt Cheah <mc...@palantir.com>.
Hi Steven,

 

What I think is happening is that your machine has a CA certificate that is used for communicating with your API server, particularly because you’re using Digital Ocean’s cluster manager. However, it’s unclear if your pod has the same CA certificate or if the pod needs that certificate file. You can use configurations to have your pod use a particular CA certificate file to communicate with the APi server. If you set spark.kubernetes.authenticate.driver.caCertFile to the path of your CA certificate on your local disk, spark-submit will create a secret that contains that certificate file and use that certificate to configure TLS for the driver pod’s communication with the API server.

 

It's clear that your driver pod doesn’t have the right TLS certificate to communicate with the API server, so I would try to introspect the driver pod and see what certificate it’s using for that communication. If there’s a fix that needs to happen in Spark, feel free to indicate as such.

 

-Matt Cheah

 

From: Steven Stetzler <st...@gmail.com>
Date: Thursday, December 13, 2018 at 1:49 PM
To: "user@spark.apache.org" <us...@spark.apache.org>
Subject: Problem running Spark on Kubernetes: Certificate error

 

Hello, 

I am following the tutorial here (https://spark.apache.org/docs/latest/running-on-kubernetes.html [spark.apache.org]) to get spark running on a Kubernetes cluster. My Kubernetes cluster is hosted with Digital Ocean's kubernetes cluster manager. I have change the KUBECONFIG environment variable to point to my cluster access credentials, so both Spark and kubectl can speak with the nodes. 

I am running into an issue when trying to run the SparkPi example as described in the Spark on Kubernetes tutorials. The command I am running is: 

./bin/spark-submit --master k8s://$CLUSTERIP --deploy-mode cluster --name spark-pi --class org.apache.spark.examples.SparkPi --conf spark.executor.instances=1 --conf spark.kubernetes.container.image=$IMAGEURL --conf spark.kubernetes.authenticate.driver.serviceAccountName=spark local:///opt/spark/examples/jars/spark-examples_2.11-2.4.0.jar 

where CLUSTERIP contains the ip of my cluster and IMAGEURL contains the URL of the Spark docker image I am using (https://hub.docker.com/r/stevenstetzler/spark/ [hub.docker.com]). This docker image was built and pushed with the script included in the Spark 2.4 distribution. I have created a service account for Spark to ensure that it has proper permissions to create pods etc., which I checked using 

kubectl auth can-i create pods --as=system:serviceaccount:default:spark 

When I try to run the SparkPi example using the above command, I get the following output: 

2018-12-12 06:26:15 WARN  Utils:66 - Your hostname, docker-test resolves to a loopback address: 127.0.1.1; using 10.46.0.6 instead (on interface eth0) 
2018-12-12 06:26:15 WARN  Utils:66 - Set SPARK_LOCAL_IP if you need to bind to another address 
2018-12-12 06:26:19 INFO  LoggingPodStatusWatcherImpl:54 - State changed, new state: 
         pod name: spark-pi-1544595975520-driver 
         namespace: default 
         labels: spark-app-selector -> spark-ec5eb54644d348e7a213f8178b8ef61f, spark-role -> driver 
         pod uid: d5d6bdc7-fdd6-11e8-b666-8e815d3815b2 
         creation time: 2018-12-12T06:26:18Z 
         service account name: spark 
         volumes: spark-local-dir-1, spark-conf-volume, spark-token-qf9dt 
         node name: N/A 
         start time: N/A 
         container images: N/A 
         phase: Pending 
         status: [] 
2018-12-12 06:26:19 INFO  LoggingPodStatusWatcherImpl:54 - State changed, new state: 
         pod name: spark-pi-1544595975520-driver 
         namespace: default 
         labels: spark-app-selector -> spark-ec5eb54644d348e7a213f8178b8ef61f, spark-role -> driver 
         pod uid: d5d6bdc7-fdd6-11e8-b666-8e815d3815b2 
         creation time: 2018-12-12T06:26:18Z 
         service account name: spark 
         volumes: spark-local-dir-1, spark-conf-volume, spark-token-qf9dt 
         node name: flamboyant-darwin-3rhc 
         start time: N/A 
         container images: N/A 
         phase: Pending 
         status: [] 
2018-12-12 06:26:19 INFO  LoggingPodStatusWatcherImpl:54 - State changed, new state: 
         pod name: spark-pi-1544595975520-driver 
         namespace: default 
         labels: spark-app-selector -> spark-ec5eb54644d348e7a213f8178b8ef61f, spark-role -> driver 
         pod uid: d5d6bdc7-fdd6-11e8-b666-8e815d3815b2 
         creation time: 2018-12-12T06:26:18Z 
         service account name: spark 
         volumes: spark-local-dir-1, spark-conf-volume, spark-token-qf9dt 
         node name: flamboyant-darwin-3rhc 
         start time: 2018-12-12T06:26:18Z 
         container images: docker.io/stevenstetzler/spark:v1 [docker.io] 
         phase: Pending 
         status: [ContainerStatus(containerID=null, image=docker.io/stevenstetzler/spark:v1 [docker.io], imageID=, lastState=ContainerState(running=null, terminated=null, waiting=null, additionalProperties={}), name=spark-kubernetes-driver, ready=false, restartCount=0, state=ContainerState(running=null, terminated=null, waiting=ContainerStateWaiting(message=null, reason=ContainerCreating, additionalProperties={}), additionalProperties={}), additionalProperties={})] 
2018-12-12 06:26:19 INFO  Client:54 - Waiting for application spark-pi to finish... 
2018-12-12 06:26:21 INFO  LoggingPodStatusWatcherImpl:54 - State changed, new state: 
         pod name: spark-pi-1544595975520-driver 
         namespace: default 
         labels: spark-app-selector -> spark-ec5eb54644d348e7a213f8178b8ef61f, spark-role -> driver 
         pod uid: d5d6bdc7-fdd6-11e8-b666-8e815d3815b2 
         creation time: 2018-12-12T06:26:18Z 
         service account name: spark 
         volumes: spark-local-dir-1, spark-conf-volume, spark-token-qf9dt 
         node name: flamboyant-darwin-3rhc 
         start time: 2018-12-12T06:26:18Z 
         container images: stevenstetzler/spark:v1 
         phase: Running 
         status: [ContainerStatus(containerID=docker://b923c6ff02b93557c8c104c01a4eeb1c05f3d0c0123ec4e5895bfd6be398a03a, image=stevenstetzler/spark:v1, imageID=docker-pullable://stevenstetzler/spark@sha256:dc4bce1e410ebd7b14a88caa46a4282a61ff058c6374b7cf721b7498829bb041, lastState=ContainerState(running=null, terminated=null, waiting=null, additionalProperties={}), name=spark-kubernetes-driver, ready=true, restartCount=0, state=ContainerState(running=ContainerStateRunning(startedAt=Time(time=2018-12-12T06:26:21Z, additionalProperties={}), additionalProperties={}), terminated=null, waiting=null, additionalProperties={}), additionalProperties={})] 
2018-12-12 06:26:24 INFO  LoggingPodStatusWatcherImpl:54 - State changed, new state: 
         pod name: spark-pi-1544595975520-driver 
         namespace: default 
         labels: spark-app-selector -> spark-ec5eb54644d348e7a213f8178b8ef61f, spark-role -> driver 
         pod uid: d5d6bdc7-fdd6-11e8-b666-8e815d3815b2 
         creation time: 2018-12-12T06:26:18Z 
         service account name: spark 
         volumes: spark-local-dir-1, spark-conf-volume, spark-token-qf9dt 
         node name: flamboyant-darwin-3rhc 
         start time: 2018-12-12T06:26:18Z 
         container images: stevenstetzler/spark:v1 
         phase: Failed 
         status: [ContainerStatus(containerID=docker://b923c6ff02b93557c8c104c01a4eeb1c05f3d0c0123ec4e5895bfd6be398a03a, image=stevenstetzler/spark:v1, imageID=docker-pullable://stevenstetzler/spark@sha256:dc4bce1e410ebd7b14a88caa46a4282a61ff058c6374b7cf721b7498829bb041, lastState=ContainerState(running=null, terminated=null, waiting=null, additionalProperties={}), name=spark-kubernetes-driver, ready=false, restartCount=0, state=ContainerState(running=null, terminated=ContainerStateTerminated(containerID=docker://b923c6ff02b93557c8c104c01a4eeb1c05f3d0c0123ec4e5895bfd6be398a03a, exitCode=1, finishedAt=Time(time=2018-12-12T06:26:24Z, additionalProperties={}), message=null, reason=Error, signal=null, startedAt=Time(time=2018-12-12T06:26:21Z, additionalProperties={}), additionalProperties={}), waiting=null, additionalProperties={}), additionalProperties={})] 
2018-12-12 06:26:24 INFO  LoggingPodStatusWatcherImpl:54 - Container final statuses: 


         Container name: spark-kubernetes-driver 
         Container image: stevenstetzler/spark:v1 
         Container state: Terminated 
         Exit code: 1 
2018-12-12 06:26:24 INFO  Client:54 - Application spark-pi finished. 
2018-12-12 06:26:24 INFO  ShutdownHookManager:54 - Shutdown hook called 
2018-12-12 06:26:24 INFO  ShutdownHookManager:54 - Deleting directory /tmp/spark-ca66ad4b-6526-4053-9ae3-f9ceb2e41cfd 

When I check the logs with 

kubectl logs spark-pi-1544595975520-driver 

I get the following: 

++ id -u 
+ myuid=0 
++ id -g 
+ mygid=0 
+ set +e 
++ getent passwd 0 
+ uidentry=root:x:0:0:root:/root:/bin/ash 
+ set -e 
+ '[' -z root:x:0:0:root:/root:/bin/ash ']' 
+ SPARK_K8S_CMD=driver 
+ case "$SPARK_K8S_CMD" in 
+ shift 1 
+ SPARK_CLASSPATH=':/opt/spark/jars/*' 
+ env 
+ grep SPARK_JAVA_OPT_ 
+ sed 's/[^=]*=\(.*\)/\1/g' 
+ sort -t_ -k4 -n 
+ readarray -t SPARK_EXECUTOR_JAVA_OPTS 
+ '[' -n '' ']' 
+ '[' -n '' ']' 
+ PYSPARK_ARGS= 
+ '[' -n '' ']' 
+ R_ARGS= 
+ '[' -n '' ']' 
+ '[' '' == 2 ']' 
+ '[' '' == 3 ']' 
+ case "$SPARK_K8S_CMD" in 
+ CMD=("$SPARK_HOME/bin/spark-submit" --conf "spark.driver.bindAddress=$SPARK_DRIVER_BIND_ADDRESS" --deploy-mode client "$@") 
+ exec /sbin/tini -s -- /opt/spark/bin/spark-submit --conf spark.driver.bindAddress=10.244.96.3 --deploy-mode client --properties-file /opt/spark/conf/spark.properties --class org.apache.spark.examples.SparkPi spark-internal 
2018-12-12 06:26:22 WARN  NativeCodeLoader:62 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 
2018-12-12 06:26:22 INFO  SparkContext:54 - Running Spark version 2.4.0 
2018-12-12 06:26:22 INFO  SparkContext:54 - Submitted application: Spark Pi 
2018-12-12 06:26:22 INFO  SecurityManager:54 - Changing view acls to: root 
2018-12-12 06:26:22 INFO  SecurityManager:54 - Changing modify acls to: root 
2018-12-12 06:26:22 INFO  SecurityManager:54 - Changing view acls groups to: 
2018-12-12 06:26:22 INFO  SecurityManager:54 - Changing modify acls groups to: 
2018-12-12 06:26:22 INFO  SecurityManager:54 - SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(root); groups with view permissions: Set(); users  with modify permissions: Set(root); groups with modify permissions: Set() 
2018-12-12 06:26:23 INFO  Utils:54 - Successfully started service 'sparkDriver' on port 7078. 
2018-12-12 06:26:23 INFO  SparkEnv:54 - Registering MapOutputTracker 
2018-12-12 06:26:23 INFO  SparkEnv:54 - Registering BlockManagerMaster 
2018-12-12 06:26:23 INFO  BlockManagerMasterEndpoint:54 - Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information 
2018-12-12 06:26:23 INFO  BlockManagerMasterEndpoint:54 - BlockManagerMasterEndpoint up 
2018-12-12 06:26:23 INFO  DiskBlockManager:54 - Created local directory at /var/data/spark-c306f3bc-cc7c-4090-a7f7-22fbc02b72bd/blockmgr-63358d1a-1e7e-493e-accd-7992da9b0adf 
2018-12-12 06:26:23 INFO  MemoryStore:54 - MemoryStore started with capacity 413.9 MB 
2018-12-12 06:26:23 INFO  SparkEnv:54 - Registering OutputCommitCoordinator 
2018-12-12 06:26:23 INFO  log:192 - Logging initialized @2059ms 
2018-12-12 06:26:23 INFO  Server:351 - jetty-9.3.z-SNAPSHOT, build timestamp: unknown, git hash: unknown 
2018-12-12 06:26:23 INFO  Server:419 - Started @2155ms 
2018-12-12 06:26:23 INFO  AbstractConnector:278 - Started ServerConnector@7180e701{HTTP/1.1,[http/1.1]}{0.0.0.0:4040 [0.0.0.0]} 
2018-12-12 06:26:23 INFO  Utils:54 - Successfully started service 'SparkUI' on port 4040. 
2018-12-12 06:26:23 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@9f6e406{/jobs,null,AVAILABLE,@Spark} 
2018-12-12 06:26:23 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@3a60c416{/jobs/json,null,AVAILABLE,@Spark} 
2018-12-12 06:26:23 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@60f2e0bd{/jobs/job,null,AVAILABLE,@Spark} 
2018-12-12 06:26:23 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@6b410923{/jobs/job/json,null,AVAILABLE,@Spark} 
2018-12-12 06:26:23 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@57bd2029{/stages,null,AVAILABLE,@Spark} 
2018-12-12 06:26:23 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@203dd56b{/stages/json,null,AVAILABLE,@Spark} 
2018-12-12 06:26:23 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@e8fadb0{/stages/stage,null,AVAILABLE,@Spark} 
2018-12-12 06:26:23 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@1d3e6d34{/stages/stage/json,null,AVAILABLE,@Spark} 
2018-12-12 06:26:23 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@6eafb10e{/stages/pool,null,AVAILABLE,@Spark} 
2018-12-12 06:26:23 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@26a94fa5{/stages/pool/json,null,AVAILABLE,@Spark} 
2018-12-12 06:26:23 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@464a4442{/storage,null,AVAILABLE,@Spark} 
2018-12-12 06:26:23 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@2873d672{/storage/json,null,AVAILABLE,@Spark} 
2018-12-12 06:26:23 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@3bc735b3{/storage/rdd,null,AVAILABLE,@Spark} 
2018-12-12 06:26:23 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@577f9109{/storage/rdd/json,null,AVAILABLE,@Spark} 
2018-12-12 06:26:23 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@4303b7f0{/environment,null,AVAILABLE,@Spark} 
2018-12-12 06:26:23 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@757529a4{/environment/json,null,AVAILABLE,@Spark} 
2018-12-12 06:26:23 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@779de014{/executors,null,AVAILABLE,@Spark} 
2018-12-12 06:26:23 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@5c41d037{/executors/json,null,AVAILABLE,@Spark} 
2018-12-12 06:26:23 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@2234078{/executors/threadDump,null,AVAILABLE,@Spark} 
2018-12-12 06:26:23 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@5ec77191{/executors/threadDump/json,null,AVAILABLE,@Spark} 
2018-12-12 06:26:23 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@4642b71d{/static,null,AVAILABLE,@Spark} 
2018-12-12 06:26:23 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@7ed9ae94{/,null,AVAILABLE,@Spark} 
2018-12-12 06:26:23 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@66908383{/api,null,AVAILABLE,@Spark} 
2018-12-12 06:26:23 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@534ca02b{/jobs/job/kill,null,AVAILABLE,@Spark} 
2018-12-12 06:26:23 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@29a23c3d{/stages/stage/kill,null,AVAILABLE,@Spark} 
2018-12-12 06:26:23 INFO  SparkUI:54 - Bound SparkUI to 0.0.0.0, and started at http://spark-pi-1544595975520-driver-svc.default.svc:4040 [spark-pi-1544595975520-driver-svc.default.svc]
2018-12-12 06:26:23 INFO  SparkContext:54 - Added JAR file:///opt/spark/examples/jars/spark-examples_2.11-2.4.0.jar at spark://spark-pi-1544595975520-driver-svc.default.svc:7078/jars/spark-examples_2.11-2.4.0.jar with timestamp 1544595983645 
2018-12-12 06:26:23 ERROR SparkContext:91 - Error initializing SparkContext. 
org.apache.spark.SparkException: External scheduler cannot be instantiated 
        at org.apache.spark.SparkContext$.org$apache$spark$SparkContext$$createTaskScheduler(SparkContext.scala:2794) 
        at org.apache.spark.SparkContext.<init>(SparkContext.scala:493) 
        at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2520) 
        at org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:935) 
        at org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:926) 
        at scala.Option.getOrElse(Option.scala:121) 
        at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:926) 
        at org.apache.spark.examples.SparkPi$.main(SparkPi.scala:31) 
        at org.apache.spark.examples.SparkPi.main(SparkPi.scala) 
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) 
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 
        at java.lang.reflect.Method.invoke(Method.java:498) 
        at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52) 
        at org.apache.spark.deploy.SparkSubmit.org [org.apache.spark.deploy.sparksubmit.org]$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:849) 
        at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:167) 
        at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:195) 
        at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86) 
        at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:924) 
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:933) 
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) 
Caused by: io.fabric8.kubernetes.client.KubernetesClientException: An error has occurred. 
        at io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:62) 
        at io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:53) 
        at io.fabric8.kubernetes.client.utils.HttpClientUtils.createHttpClient(HttpClientUtils.java:167) 
        at org.apache.spark.deploy.k8s.SparkKubernetesClientFactory$.createKubernetesClient(SparkKubernetesClientFactory.scala:84) 
        at org.apache.spark.scheduler.cluster.k8s.KubernetesClusterManager.createSchedulerBackend(KubernetesClusterManager.scala:64) 
        at org.apache.spark.SparkContext$.org$apache$spark$SparkContext$$createTaskScheduler(SparkContext.scala:2788) 
        ... 20 more 
Caused by: java.security.cert.CertificateException: Could not parse certificate: java.io.IOException: Empty input 
        at sun.security.provider.X509Factory.engineGenerateCertificate(X509Factory.java:110) 
        at java.security.cert.CertificateFactory.generateCertificate(CertificateFactory.java:339) 
        at io.fabric8.kubernetes.client.internal.CertUtils.createTrustStore(CertUtils.java:93) 
        at io.fabric8.kubernetes.client.internal.CertUtils.createTrustStore(CertUtils.java:71) 
        at io.fabric8.kubernetes.client.internal.SSLUtils.trustManagers(SSLUtils.java:114) 
        at io.fabric8.kubernetes.client.internal.SSLUtils.trustManagers(SSLUtils.java:93) 
        at io.fabric8.kubernetes.client.utils.HttpClientUtils.createHttpClient(HttpClientUtils.java:63) 
        ... 23 more 
Caused by: java.io.IOException: Empty input 
        at sun.security.provider.X509Factory.engineGenerateCertificate(X509Factory.java:106) 
        ... 29 more 
2018-12-12 06:26:23 INFO  AbstractConnector:318 - Stopped Spark@7180e701{HTTP/1.1,[http/1.1]}{0.0.0.0:4040 [0.0.0.0]} 
2018-12-12 06:26:23 INFO  SparkUI:54 - Stopped Spark web UI at http://spark-pi-1544595975520-driver-svc.default.svc:4040 [spark-pi-1544595975520-driver-svc.default.svc]
2018-12-12 06:26:23 INFO  MapOutputTrackerMasterEndpoint:54 - MapOutputTrackerMasterEndpoint stopped! 
2018-12-12 06:26:23 INFO  MemoryStore:54 - MemoryStore cleared 
2018-12-12 06:26:23 INFO  BlockManager:54 - BlockManager stopped 
2018-12-12 06:26:23 INFO  BlockManagerMaster:54 - BlockManagerMaster stopped 
2018-12-12 06:26:23 WARN  MetricsSystem:66 - Stopping a MetricsSystem that is not running 
2018-12-12 06:26:23 INFO  OutputCommitCoordinator$OutputCommitCoordinatorEndpoint:54 - OutputCommitCoordinator stopped! 
2018-12-12 06:26:23 INFO  SparkContext:54 - Successfully stopped SparkContext 
Exception in thread "main" org.apache.spark.SparkException: External scheduler cannot be instantiated 
        at org.apache.spark.SparkContext$.org$apache$spark$SparkContext$$createTaskScheduler(SparkContext.scala:2794) 
        at org.apache.spark.SparkContext.<init>(SparkContext.scala:493) 
        at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2520) 
        at org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:935) 
        at org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:926) 
        at scala.Option.getOrElse(Option.scala:121) 
        at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:926) 
        at org.apache.spark.examples.SparkPi$.main(SparkPi.scala:31) 
        at org.apache.spark.examples.SparkPi.main(SparkPi.scala) 
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) 
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 
        at java.lang.reflect.Method.invoke(Method.java:498) 
        at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52) 
        at org.apache.spark.deploy.SparkSubmit.org [org.apache.spark.deploy.sparksubmit.org]$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:849) 
        at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:167) 
        at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:195) 
        at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86) 
        at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:924) 
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:933) 
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) 
Caused by: io.fabric8.kubernetes.client.KubernetesClientException: An error has occurred. 
        at io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:62) 
        at io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:53) 
        at io.fabric8.kubernetes.client.utils.HttpClientUtils.createHttpClient(HttpClientUtils.java:167) 
        at org.apache.spark.deploy.k8s.SparkKubernetesClientFactory$.createKubernetesClient(SparkKubernetesClientFactory.scala:84) 
        at org.apache.spark.scheduler.cluster.k8s.KubernetesClusterManager.createSchedulerBackend(KubernetesClusterManager.scala:64) 
        at org.apache.spark.SparkContext$.org$apache$spark$SparkContext$$createTaskScheduler(SparkContext.scala:2788) 
        ... 20 more 
Caused by: java.security.cert.CertificateException: Could not parse certificate: java.io.IOException: Empty input 
        at sun.security.provider.X509Factory.engineGenerateCertificate(X509Factory.java:110) 
        at java.security.cert.CertificateFactory.generateCertificate(CertificateFactory.java:339) 
        at io.fabric8.kubernetes.client.internal.CertUtils.createTrustStore(CertUtils.java:93) 
        at io.fabric8.kubernetes.client.internal.CertUtils.createTrustStore(CertUtils.java:71) 
        at io.fabric8.kubernetes.client.internal.SSLUtils.trustManagers(SSLUtils.java:114) 
        at io.fabric8.kubernetes.client.internal.SSLUtils.trustManagers(SSLUtils.java:93) 
        at io.fabric8.kubernetes.client.utils.HttpClientUtils.createHttpClient(HttpClientUtils.java:63) 
        ... 23 more 
Caused by: java.io.IOException: Empty input 
        at sun.security.provider.X509Factory.engineGenerateCertificate(X509Factory.java:106) 
        ... 29 more 
2018-12-12 06:26:23 INFO  ShutdownHookManager:54 - Shutdown hook called 
2018-12-12 06:26:23 INFO  ShutdownHookManager:54 - Deleting directory /var/data/spark-c306f3bc-cc7c-4090-a7f7-22fbc02b72bd/spark-157bde7a-d0e1-4092-9e39-5ab2b727e3e7 
2018-12-12 06:26:23 INFO  ShutdownHookManager:54 - Deleting directory /tmp/spark-07910ccb-c212-4622-9c1b-3e8b772f3faa 

It looks to me like there is a communication / authentication issue during pod creation, and it looks like there might be a certificate missing that Spark needs to communicate with Kubernetes, which I infer from the "java.security.cert.CertificateException: Could not parse certificate: java.io.IOException: Empty input" error message. 

Any ideas on what I can do to resolve this? Thanks!