You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Steven Stetzler <st...@gmail.com> on 2018/12/13 21:48:51 UTC
Problem running Spark on Kubernetes: Certificate error
Hello,
I am following the tutorial here (
https://spark.apache.org/docs/latest/running-on-kubernetes.html) to get
spark running on a Kubernetes cluster. My Kubernetes cluster is hosted with
Digital Ocean's kubernetes cluster manager. I have change the KUBECONFIG
environment variable to point to my cluster access credentials, so both
Spark and kubectl can speak with the nodes.
I am running into an issue when trying to run the SparkPi example as
described in the Spark on Kubernetes tutorials. The command I am running
is:
./bin/spark-submit --master k8s://$CLUSTERIP --deploy-mode cluster --name
spark-pi --class org.apache.spark.examples.SparkPi --conf
spark.executor.instances=1 --conf
spark.kubernetes.container.image=$IMAGEURL --conf
spark.kubernetes.authenticate.driver.serviceAccountName=spark
local:///opt/spark/examples/jars/spark-examples_2.11-2.4.0.jar
where CLUSTERIP contains the ip of my cluster and IMAGEURL contains the URL
of the Spark docker image I am using (
https://hub.docker.com/r/stevenstetzler/spark/). This docker image was
built and pushed with the script included in the Spark 2.4 distribution. I
have created a service account for Spark to ensure that it has proper
permissions to create pods etc., which I checked using
kubectl auth can-i create pods --as=system:serviceaccount:default:spark
When I try to run the SparkPi example using the above command, I get the
following output:
2018-12-12 06:26:15 WARN Utils:66 - Your hostname, docker-test resolves to
a loopback address: 127.0.1.1; using 10.46.0.6 instead (on interface eth0)
2018-12-12 06:26:15 WARN Utils:66 - Set SPARK_LOCAL_IP if you need to bind
to another address
2018-12-12 06:26:19 INFO LoggingPodStatusWatcherImpl:54 - State changed,
new state:
pod name: spark-pi-1544595975520-driver
namespace: default
labels: spark-app-selector ->
spark-ec5eb54644d348e7a213f8178b8ef61f, spark-role -> driver
pod uid: d5d6bdc7-fdd6-11e8-b666-8e815d3815b2
creation time: 2018-12-12T06:26:18Z
service account name: spark
volumes: spark-local-dir-1, spark-conf-volume, spark-token-qf9dt
node name: N/A
start time: N/A
container images: N/A
phase: Pending
status: []
2018-12-12 06:26:19 INFO LoggingPodStatusWatcherImpl:54 - State changed,
new state:
pod name: spark-pi-1544595975520-driver
namespace: default
labels: spark-app-selector ->
spark-ec5eb54644d348e7a213f8178b8ef61f, spark-role -> driver
pod uid: d5d6bdc7-fdd6-11e8-b666-8e815d3815b2
creation time: 2018-12-12T06:26:18Z
service account name: spark
volumes: spark-local-dir-1, spark-conf-volume, spark-token-qf9dt
node name: flamboyant-darwin-3rhc
start time: N/A
container images: N/A
phase: Pending
status: []
2018-12-12 06:26:19 INFO LoggingPodStatusWatcherImpl:54 - State changed,
new state:
pod name: spark-pi-1544595975520-driver
namespace: default
labels: spark-app-selector ->
spark-ec5eb54644d348e7a213f8178b8ef61f, spark-role -> driver
pod uid: d5d6bdc7-fdd6-11e8-b666-8e815d3815b2
creation time: 2018-12-12T06:26:18Z
service account name: spark
volumes: spark-local-dir-1, spark-conf-volume, spark-token-qf9dt
node name: flamboyant-darwin-3rhc
start time: 2018-12-12T06:26:18Z
container images: docker.io/stevenstetzler/spark:v1
phase: Pending
status: [ContainerStatus(containerID=null, image=
docker.io/stevenstetzler/spark:v1, imageID=,
lastState=ContainerState(running=null, terminated=null, waiting=null,
additionalProperties={}), name=spark-kubernetes-driver, ready=false,
restartCount=0, state=ContainerState(running=null, terminated=null,
waiting=ContainerStateWaiting(message=null, reason=ContainerCreating,
additionalProperties={}), additionalProperties={}),
additionalProperties={})]
2018-12-12 06:26:19 INFO Client:54 - Waiting for application spark-pi to
finish...
2018-12-12 06:26:21 INFO LoggingPodStatusWatcherImpl:54 - State changed,
new state:
pod name: spark-pi-1544595975520-driver
namespace: default
labels: spark-app-selector ->
spark-ec5eb54644d348e7a213f8178b8ef61f, spark-role -> driver
pod uid: d5d6bdc7-fdd6-11e8-b666-8e815d3815b2
creation time: 2018-12-12T06:26:18Z
service account name: spark
volumes: spark-local-dir-1, spark-conf-volume, spark-token-qf9dt
node name: flamboyant-darwin-3rhc
start time: 2018-12-12T06:26:18Z
container images: stevenstetzler/spark:v1
phase: Running
status:
[ContainerStatus(containerID=docker://b923c6ff02b93557c8c104c01a4eeb1c05f3d0c0123ec4e5895bfd6be398a03a,
image=stevenstetzler/spark:v1,
imageID=docker-pullable://stevenstetzler/spark@sha256:dc4bce1e410ebd7b14a88caa46a4282a61ff058c6374b7cf721b7498829bb041,
lastState=ContainerState(running=null, terminated=null, waiting=null,
additionalProperties={}), name=spark-kubernetes-driver, ready=true,
restartCount=0,
state=ContainerState(running=ContainerStateRunning(startedAt=Time(time=2018-12-12T06:26:21Z,
additionalProperties={}), additionalProperties={}), terminated=null,
waiting=null, additionalProperties={}), additionalProperties={})]
2018-12-12 06:26:24 INFO LoggingPodStatusWatcherImpl:54 - State changed,
new state:
pod name: spark-pi-1544595975520-driver
namespace: default
labels: spark-app-selector ->
spark-ec5eb54644d348e7a213f8178b8ef61f, spark-role -> driver
pod uid: d5d6bdc7-fdd6-11e8-b666-8e815d3815b2
creation time: 2018-12-12T06:26:18Z
service account name: spark
volumes: spark-local-dir-1, spark-conf-volume, spark-token-qf9dt
node name: flamboyant-darwin-3rhc
start time: 2018-12-12T06:26:18Z
container images: stevenstetzler/spark:v1
phase: Failed
status:
[ContainerStatus(containerID=docker://b923c6ff02b93557c8c104c01a4eeb1c05f3d0c0123ec4e5895bfd6be398a03a,
image=stevenstetzler/spark:v1,
imageID=docker-pullable://stevenstetzler/spark@sha256:dc4bce1e410ebd7b14a88caa46a4282a61ff058c6374b7cf721b7498829bb041,
lastState=ContainerState(running=null, terminated=null, waiting=null,
additionalProperties={}), name=spark-kubernetes-driver, ready=false,
restartCount=0, state=ContainerState(running=null,
terminated=ContainerStateTerminated(containerID=docker://b923c6ff02b93557c8c104c01a4eeb1c05f3d0c0123ec4e5895bfd6be398a03a,
exitCode=1, finishedAt=Time(time=2018-12-12T06:26:24Z,
additionalProperties={}), message=null, reason=Error, signal=null,
startedAt=Time(time=2018-12-12T06:26:21Z, additionalProperties={}),
additionalProperties={}), waiting=null, additionalProperties={}),
additionalProperties={})]
2018-12-12 06:26:24 INFO LoggingPodStatusWatcherImpl:54 - Container final
statuses:
Container name: spark-kubernetes-driver
Container image: stevenstetzler/spark:v1
Container state: Terminated
Exit code: 1
2018-12-12 06:26:24 INFO Client:54 - Application spark-pi finished.
2018-12-12 06:26:24 INFO ShutdownHookManager:54 - Shutdown hook called
2018-12-12 06:26:24 INFO ShutdownHookManager:54 - Deleting directory
/tmp/spark-ca66ad4b-6526-4053-9ae3-f9ceb2e41cfd
When I check the logs with
kubectl logs spark-pi-1544595975520-driver
I get the following:
++ id -u
+ myuid=0
++ id -g
+ mygid=0
+ set +e
++ getent passwd 0
+ uidentry=root:x:0:0:root:/root:/bin/ash
+ set -e
+ '[' -z root:x:0:0:root:/root:/bin/ash ']'
+ SPARK_K8S_CMD=driver
+ case "$SPARK_K8S_CMD" in
+ shift 1
+ SPARK_CLASSPATH=':/opt/spark/jars/*'
+ env
+ grep SPARK_JAVA_OPT_
+ sed 's/[^=]*=\(.*\)/\1/g'
+ sort -t_ -k4 -n
+ readarray -t SPARK_EXECUTOR_JAVA_OPTS
+ '[' -n '' ']'
+ '[' -n '' ']'
+ PYSPARK_ARGS=
+ '[' -n '' ']'
+ R_ARGS=
+ '[' -n '' ']'
+ '[' '' == 2 ']'
+ '[' '' == 3 ']'
+ case "$SPARK_K8S_CMD" in
+ CMD=("$SPARK_HOME/bin/spark-submit" --conf
"spark.driver.bindAddress=$SPARK_DRIVER_BIND_ADDRESS" --deploy-mode client
"$@")
+ exec /sbin/tini -s -- /opt/spark/bin/spark-submit --conf
spark.driver.bindAddress=10.244.96.3 --deploy-mode client --properties-file
/opt/spark/conf/spark.properties --class org.apache.spark.examples.SparkPi
spark-internal
2018-12-12 06:26:22 WARN NativeCodeLoader:62 - Unable to load
native-hadoop library for your platform... using builtin-java classes where
applicable
2018-12-12 06:26:22 INFO SparkContext:54 - Running Spark version 2.4.0
2018-12-12 06:26:22 INFO SparkContext:54 - Submitted application: Spark Pi
2018-12-12 06:26:22 INFO SecurityManager:54 - Changing view acls to: root
2018-12-12 06:26:22 INFO SecurityManager:54 - Changing modify acls to:
root
2018-12-12 06:26:22 INFO SecurityManager:54 - Changing view acls groups
to:
2018-12-12 06:26:22 INFO SecurityManager:54 - Changing modify acls groups
to:
2018-12-12 06:26:22 INFO SecurityManager:54 - SecurityManager:
authentication disabled; ui acls disabled; users with view permissions:
Set(root); groups with view permissions: Set(); users with modify
permissions: Set(root); groups with modify permissions: Set()
2018-12-12 06:26:23 INFO Utils:54 - Successfully started service
'sparkDriver' on port 7078.
2018-12-12 06:26:23 INFO SparkEnv:54 - Registering MapOutputTracker
2018-12-12 06:26:23 INFO SparkEnv:54 - Registering BlockManagerMaster
2018-12-12 06:26:23 INFO BlockManagerMasterEndpoint:54 - Using
org.apache.spark.storage.DefaultTopologyMapper for getting topology
information
2018-12-12 06:26:23 INFO BlockManagerMasterEndpoint:54 -
BlockManagerMasterEndpoint up
2018-12-12 06:26:23 INFO DiskBlockManager:54 - Created local directory at
/var/data/spark-c306f3bc-cc7c-4090-a7f7-22fbc02b72bd/blockmgr-63358d1a-1e7e-493e-accd-7992da9b0adf
2018-12-12 06:26:23 INFO MemoryStore:54 - MemoryStore started with
capacity 413.9 MB
2018-12-12 06:26:23 INFO SparkEnv:54 - Registering OutputCommitCoordinator
2018-12-12 06:26:23 INFO log:192 - Logging initialized @2059ms
2018-12-12 06:26:23 INFO Server:351 - jetty-9.3.z-SNAPSHOT, build
timestamp: unknown, git hash: unknown
2018-12-12 06:26:23 INFO Server:419 - Started @2155ms
2018-12-12 06:26:23 INFO AbstractConnector:278 - Started
ServerConnector@7180e701{HTTP/1.1,[http/1.1]}{0.0.0.0:4040}
2018-12-12 06:26:23 INFO Utils:54 - Successfully started service 'SparkUI'
on port 4040.
2018-12-12 06:26:23 INFO ContextHandler:781 - Started
o.s.j.s.ServletContextHandler@9f6e406{/jobs,null,AVAILABLE,@Spark}
2018-12-12 06:26:23 INFO ContextHandler:781 - Started
o.s.j.s.ServletContextHandler@3a60c416{/jobs/json,null,AVAILABLE,@Spark}
2018-12-12 06:26:23 INFO ContextHandler:781 - Started
o.s.j.s.ServletContextHandler@60f2e0bd{/jobs/job,null,AVAILABLE,@Spark}
2018-12-12 06:26:23 INFO ContextHandler:781 - Started
o.s.j.s.ServletContextHandler@6b410923
{/jobs/job/json,null,AVAILABLE,@Spark}
2018-12-12 06:26:23 INFO ContextHandler:781 - Started
o.s.j.s.ServletContextHandler@57bd2029{/stages,null,AVAILABLE,@Spark}
2018-12-12 06:26:23 INFO ContextHandler:781 - Started
o.s.j.s.ServletContextHandler@203dd56b{/stages/json,null,AVAILABLE,@Spark}
2018-12-12 06:26:23 INFO ContextHandler:781 - Started
o.s.j.s.ServletContextHandler@e8fadb0{/stages/stage,null,AVAILABLE,@Spark}
2018-12-12 06:26:23 INFO ContextHandler:781 - Started
o.s.j.s.ServletContextHandler@1d3e6d34
{/stages/stage/json,null,AVAILABLE,@Spark}
2018-12-12 06:26:23 INFO ContextHandler:781 - Started
o.s.j.s.ServletContextHandler@6eafb10e{/stages/pool,null,AVAILABLE,@Spark}
2018-12-12 06:26:23 INFO ContextHandler:781 - Started
o.s.j.s.ServletContextHandler@26a94fa5
{/stages/pool/json,null,AVAILABLE,@Spark}
2018-12-12 06:26:23 INFO ContextHandler:781 - Started
o.s.j.s.ServletContextHandler@464a4442{/storage,null,AVAILABLE,@Spark}
2018-12-12 06:26:23 INFO ContextHandler:781 - Started
o.s.j.s.ServletContextHandler@2873d672{/storage/json,null,AVAILABLE,@Spark}
2018-12-12 06:26:23 INFO ContextHandler:781 - Started
o.s.j.s.ServletContextHandler@3bc735b3{/storage/rdd,null,AVAILABLE,@Spark}
2018-12-12 06:26:23 INFO ContextHandler:781 - Started
o.s.j.s.ServletContextHandler@577f9109
{/storage/rdd/json,null,AVAILABLE,@Spark}
2018-12-12 06:26:23 INFO ContextHandler:781 - Started
o.s.j.s.ServletContextHandler@4303b7f0{/environment,null,AVAILABLE,@Spark}
2018-12-12 06:26:23 INFO ContextHandler:781 - Started
o.s.j.s.ServletContextHandler@757529a4
{/environment/json,null,AVAILABLE,@Spark}
2018-12-12 06:26:23 INFO ContextHandler:781 - Started
o.s.j.s.ServletContextHandler@779de014{/executors,null,AVAILABLE,@Spark}
2018-12-12 06:26:23 INFO ContextHandler:781 - Started
o.s.j.s.ServletContextHandler@5c41d037
{/executors/json,null,AVAILABLE,@Spark}
2018-12-12 06:26:23 INFO ContextHandler:781 - Started
o.s.j.s.ServletContextHandler@2234078
{/executors/threadDump,null,AVAILABLE,@Spark}
2018-12-12 06:26:23 INFO ContextHandler:781 - Started
o.s.j.s.ServletContextHandler@5ec77191
{/executors/threadDump/json,null,AVAILABLE,@Spark}
2018-12-12 06:26:23 INFO ContextHandler:781 - Started
o.s.j.s.ServletContextHandler@4642b71d{/static,null,AVAILABLE,@Spark}
2018-12-12 06:26:23 INFO ContextHandler:781 - Started
o.s.j.s.ServletContextHandler@7ed9ae94{/,null,AVAILABLE,@Spark}
2018-12-12 06:26:23 INFO ContextHandler:781 - Started
o.s.j.s.ServletContextHandler@66908383{/api,null,AVAILABLE,@Spark}
2018-12-12 06:26:23 INFO ContextHandler:781 - Started
o.s.j.s.ServletContextHandler@534ca02b
{/jobs/job/kill,null,AVAILABLE,@Spark}
2018-12-12 06:26:23 INFO ContextHandler:781 - Started
o.s.j.s.ServletContextHandler@29a23c3d
{/stages/stage/kill,null,AVAILABLE,@Spark}
2018-12-12 06:26:23 INFO SparkUI:54 - Bound SparkUI to 0.0.0.0, and
started at http://spark-pi-1544595975520-driver-svc.default.svc:4040
2018-12-12 06:26:23 INFO SparkContext:54 - Added JAR
file:///opt/spark/examples/jars/spark-examples_2.11-2.4.0.jar at
spark://spark-pi-1544595975520-driver-svc.default.svc:7078/jars/spark-examples_2.11-2.4.0.jar
with timestamp 1544595983645
2018-12-12 06:26:23 ERROR SparkContext:91 - Error initializing
SparkContext.
org.apache.spark.SparkException: External scheduler cannot be instantiated
at
org.apache.spark.SparkContext$.org$apache$spark$SparkContext$$createTaskScheduler(SparkContext.scala:2794)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:493)
at
org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2520)
at
org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:935)
at
org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:926)
at scala.Option.getOrElse(Option.scala:121)
at
org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:926)
at org.apache.spark.examples.SparkPi$.main(SparkPi.scala:31)
at org.apache.spark.examples.SparkPi.main(SparkPi.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
at org.apache.spark.deploy.SparkSubmit.org
$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:849)
at
org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:167)
at
org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:195)
at
org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
at
org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:924)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:933)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: io.fabric8.kubernetes.client.KubernetesClientException: An error
has occurred.
at
io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:62)
at
io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:53)
at
io.fabric8.kubernetes.client.utils.HttpClientUtils.createHttpClient(HttpClientUtils.java:167)
at
org.apache.spark.deploy.k8s.SparkKubernetesClientFactory$.createKubernetesClient(SparkKubernetesClientFactory.scala:84)
at
org.apache.spark.scheduler.cluster.k8s.KubernetesClusterManager.createSchedulerBackend(KubernetesClusterManager.scala:64)
at
org.apache.spark.SparkContext$.org$apache$spark$SparkContext$$createTaskScheduler(SparkContext.scala:2788)
... 20 more
Caused by: java.security.cert.CertificateException: Could not parse
certificate: java.io.IOException: Empty input
at
sun.security.provider.X509Factory.engineGenerateCertificate(X509Factory.java:110)
at
java.security.cert.CertificateFactory.generateCertificate(CertificateFactory.java:339)
at
io.fabric8.kubernetes.client.internal.CertUtils.createTrustStore(CertUtils.java:93)
at
io.fabric8.kubernetes.client.internal.CertUtils.createTrustStore(CertUtils.java:71)
at
io.fabric8.kubernetes.client.internal.SSLUtils.trustManagers(SSLUtils.java:114)
at
io.fabric8.kubernetes.client.internal.SSLUtils.trustManagers(SSLUtils.java:93)
at
io.fabric8.kubernetes.client.utils.HttpClientUtils.createHttpClient(HttpClientUtils.java:63)
... 23 more
Caused by: java.io.IOException: Empty input
at
sun.security.provider.X509Factory.engineGenerateCertificate(X509Factory.java:106)
... 29 more
2018-12-12 06:26:23 INFO AbstractConnector:318 - Stopped Spark@7180e701
{HTTP/1.1,[http/1.1]}{0.0.0.0:4040}
2018-12-12 06:26:23 INFO SparkUI:54 - Stopped Spark web UI at
http://spark-pi-1544595975520-driver-svc.default.svc:4040
2018-12-12 06:26:23 INFO MapOutputTrackerMasterEndpoint:54 -
MapOutputTrackerMasterEndpoint stopped!
2018-12-12 06:26:23 INFO MemoryStore:54 - MemoryStore cleared
2018-12-12 06:26:23 INFO BlockManager:54 - BlockManager stopped
2018-12-12 06:26:23 INFO BlockManagerMaster:54 - BlockManagerMaster
stopped
2018-12-12 06:26:23 WARN MetricsSystem:66 - Stopping a MetricsSystem that
is not running
2018-12-12 06:26:23 INFO
OutputCommitCoordinator$OutputCommitCoordinatorEndpoint:54 -
OutputCommitCoordinator stopped!
2018-12-12 06:26:23 INFO SparkContext:54 - Successfully stopped
SparkContext
Exception in thread "main" org.apache.spark.SparkException: External
scheduler cannot be instantiated
at
org.apache.spark.SparkContext$.org$apache$spark$SparkContext$$createTaskScheduler(SparkContext.scala:2794)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:493)
at
org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2520)
at
org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:935)
at
org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:926)
at scala.Option.getOrElse(Option.scala:121)
at
org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:926)
at org.apache.spark.examples.SparkPi$.main(SparkPi.scala:31)
at org.apache.spark.examples.SparkPi.main(SparkPi.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
at org.apache.spark.deploy.SparkSubmit.org
$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:849)
at
org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:167)
at
org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:195)
at
org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
at
org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:924)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:933)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: io.fabric8.kubernetes.client.KubernetesClientException: An error
has occurred.
at
io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:62)
at
io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:53)
at
io.fabric8.kubernetes.client.utils.HttpClientUtils.createHttpClient(HttpClientUtils.java:167)
at
org.apache.spark.deploy.k8s.SparkKubernetesClientFactory$.createKubernetesClient(SparkKubernetesClientFactory.scala:84)
at
org.apache.spark.scheduler.cluster.k8s.KubernetesClusterManager.createSchedulerBackend(KubernetesClusterManager.scala:64)
at
org.apache.spark.SparkContext$.org$apache$spark$SparkContext$$createTaskScheduler(SparkContext.scala:2788)
... 20 more
Caused by: java.security.cert.CertificateException: Could not parse
certificate: java.io.IOException: Empty input
at
sun.security.provider.X509Factory.engineGenerateCertificate(X509Factory.java:110)
at
java.security.cert.CertificateFactory.generateCertificate(CertificateFactory.java:339)
at
io.fabric8.kubernetes.client.internal.CertUtils.createTrustStore(CertUtils.java:93)
at
io.fabric8.kubernetes.client.internal.CertUtils.createTrustStore(CertUtils.java:71)
at
io.fabric8.kubernetes.client.internal.SSLUtils.trustManagers(SSLUtils.java:114)
at
io.fabric8.kubernetes.client.internal.SSLUtils.trustManagers(SSLUtils.java:93)
at
io.fabric8.kubernetes.client.utils.HttpClientUtils.createHttpClient(HttpClientUtils.java:63)
... 23 more
Caused by: java.io.IOException: Empty input
at
sun.security.provider.X509Factory.engineGenerateCertificate(X509Factory.java:106)
... 29 more
2018-12-12 06:26:23 INFO ShutdownHookManager:54 - Shutdown hook called
2018-12-12 06:26:23 INFO ShutdownHookManager:54 - Deleting directory
/var/data/spark-c306f3bc-cc7c-4090-a7f7-22fbc02b72bd/spark-157bde7a-d0e1-4092-9e39-5ab2b727e3e7
2018-12-12 06:26:23 INFO ShutdownHookManager:54 - Deleting directory
/tmp/spark-07910ccb-c212-4622-9c1b-3e8b772f3faa
It looks to me like there is a communication / authentication issue during
pod creation, and it looks like there might be a certificate missing that
Spark needs to communicate with Kubernetes, which I infer from the
"java.security.cert.CertificateException: Could not parse certificate:
java.io.IOException: Empty input" error message.
Any ideas on what I can do to resolve this? Thanks!
Re: Problem running Spark on Kubernetes: Certificate error
Posted by Steven Stetzler <st...@gmail.com>.
Hi Matt,
Thank your for the help. This worked for me. For posterity: convert the
data in the certificate-authority-data field of your DigitalOcean
Kubernetes configuration file, which is downloaded from their site, from
base64 to PEM format and save it to a file ca.crt, then submit with --conf
spark.kubernetes.authenticate.driver.caCertFile=ca.crt.
As an extension to this question, I am also interested in using PySpark on
a JupyterHub deployment to interact with my cluster. The JupyterHub
deployment provides a Python (with PySpark) environment in a pod on a node
in the cluster, and I would like to create a SparkContext for computations
that uses the same cluster. I have attempted to replicate the above
solution there using:
from pyspark import *
> conf = SparkConf()
> conf.setMaster('k8s://<CLUSTERIP>')
> conf.set('spark.kubernetes.container.image', '<IMAGEURL>')
> conf.set('spark.kubernetes.authenticate.driver.caCertFile', 'ca.crt')
> sc = SparkContext(conf=conf)
which returns an identical error:
---------------------------------------------------------------------------
> Py4JJavaError Traceback (most recent call last)
> <ipython-input-4-bfc2d0364053> in <module>
> 22
> 23
> ---> 24 sc = SparkContext(conf=conf)
> /usr/local/spark/python/pyspark/context.py in __init__(self, master,
> appName, sparkHome, pyFiles, environment, batchSize, serializer, conf,
> gateway, jsc, profiler_cls)
> 116 try:
> 117 self._do_init(master, appName, sparkHome, pyFiles, environment,
> batchSize, serializer,
> --> 118 conf, jsc, profiler_cls) 119 except:
> 120 # If an error occurs, clean up in order to allow future SparkContext
> creation: /usr/local/spark/python/pyspark/context.py in _do_init(self,
> master, appName, sparkHome, pyFiles, environment, batchSize, serializer,
> conf, jsc, profiler_cls)
> 178
> 179 # Create the Java SparkContext through Py4J
> --> 180 self._jsc = jsc or self._initialize_context(self._conf._jconf)
> 181 # Reset the SparkConf to the one actually used by the SparkContext in
> JVM.
> 182 self._conf = SparkConf(_jconf=self._jsc.sc().conf())
> /usr/local/spark/python/pyspark/context.py in _initialize_context(self,
> jconf)
> 286 Initialize SparkContext in function to allow subclass specific
> initialization
> 287 """
> --> 288 return self._jvm.JavaSparkContext(jconf)
> 289
> 290 @classmethod
> /usr/local/spark/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py in
> __call__(self, *args)
> 1523 answer = self._gateway_client.send_command(command)
> 1524 return_value = get_return_value(
> -> 1525 answer, self._gateway_client, None, self._fqn) 1526
> 1527 for temp_arg in temp_args:
> /usr/local/spark/python/lib/py4j-0.10.7-src.zip/py4j/protocol.py in
> get_return_value(answer, gateway_client, target_id, name)
> 326 raise Py4JJavaError(
> 327 "An error occurred while calling {0}{1}{2}.\n".
> --> 328 format(target_id, ".", name), value) 329 else:
> 330 raise Py4JError( Py4JJavaError: An error occurred while calling
> None.org.apache.spark.api.java.JavaSparkContext.
> : org.apache.spark.SparkException: External scheduler cannot be
> instantiated
> at
> org.apache.spark.SparkContext$.org$apache$spark$SparkContext$$createTaskScheduler(SparkContext.scala:2794)
> at org.apache.spark.SparkContext.<init>(SparkContext.scala:493)
> at
> org.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:58)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:247)
> at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
> at py4j.Gateway.invoke(Gateway.java:238)
> at
> py4j.commands.ConstructorCommand.invokeConstructor(ConstructorCommand.java:80)
> at py4j.commands.ConstructorCommand.execute(ConstructorCommand.java:69)
> at py4j.GatewayConnection.run(GatewayConnection.java:238)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: io.fabric8.kubernetes.client.KubernetesClientException: An
> error has occurred.
> at
> io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:62)
> at
> io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:53)
> at
> io.fabric8.kubernetes.client.utils.HttpClientUtils.createHttpClient(HttpClientUtils.java:167)
> at
> org.apache.spark.deploy.k8s.SparkKubernetesClientFactory$.createKubernetesClient(SparkKubernetesClientFactory.scala:84)
> at
> org.apache.spark.scheduler.cluster.k8s.KubernetesClusterManager.createSchedulerBackend(KubernetesClusterManager.scala:64)
> at
> org.apache.spark.SparkContext$.org$apache$spark$SparkContext$$createTaskScheduler(SparkContext.scala:2788)
> ... 13 more
> Caused by: java.security.cert.CertificateException: Could not parse
> certificate: java.io.IOException: Empty input
> at
> sun.security.provider.X509Factory.engineGenerateCertificate(X509Factory.java:110)
> at
> java.security.cert.CertificateFactory.generateCertificate(CertificateFactory.java:339)
> at
> io.fabric8.kubernetes.client.internal.CertUtils.createTrustStore(CertUtils.java:93)
> at
> io.fabric8.kubernetes.client.internal.CertUtils.createTrustStore(CertUtils.java:71)
> at
> io.fabric8.kubernetes.client.internal.SSLUtils.trustManagers(SSLUtils.java:114)
> at
> io.fabric8.kubernetes.client.internal.SSLUtils.trustManagers(SSLUtils.java:93)
> at
> io.fabric8.kubernetes.client.utils.HttpClientUtils.createHttpClient(HttpClientUtils.java:63)
> ... 16 more
> Caused by: java.io.IOException: Empty input
> at
> sun.security.provider.X509Factory.engineGenerateCertificate(X509Factory.java:106)
> ... 22 more
On the other hand, I have no problem running
from pyspark import *
> sc = SparkContext()
and then performing a computation. However I believe that this will only be
able to use the resources provided to the Kubernetes pod that the Jupyter
environment is deployed on and not the whole cluster, so it's not really
what I'm looking for. I have tried running the same Python code from
outside of the cluster (on my own machine with PySpark installed). Note
that it doesn't matter what IP I use for the master, the same error occurs.
Running from outside of the cluster does seem to allow PySpark to
communicate with the cluster. However, there appear to be issues allocating
resources / creating workers. (But that seems like an issue for another
thread.)
Any idea what can be done here? Note that if it was really passing the file
and the file was not found, it would raise an error to indicate that. (I
have tried.) The error raised indicates instead that no file was ever
passed. I would assume that
conf.set('spark.kubernetes.authenticate.driver.caCertFile',
'ca.crt') would have identical behavior to --conf
spark.kubernetes.authenticate.driver.caCertFile=ca.crt in spark-submit.
Sorry if I am misunderstanding something basic in how PySpark and
spark-submit work.
Thanks,
Steven
On Thu, Dec 13, 2018 at 7:59 PM Matt Cheah <mc...@palantir.com> wrote:
> Hi Steven,
>
>
>
> What I think is happening is that your machine has a CA certificate that
> is used for communicating with your API server, particularly because you’re
> using Digital Ocean’s cluster manager. However, it’s unclear if your pod
> has the same CA certificate or if the pod needs that certificate file. You
> can use configurations to have your pod use a particular CA certificate
> file to communicate with the APi server. If you set spark.kubernetes.authenticate.driver.caCertFile
> to the path of your CA certificate on your local disk, spark-submit will
> create a secret that contains that certificate file and use that
> certificate to configure TLS for the driver pod’s communication with the
> API server.
>
>
>
> It's clear that your driver pod doesn’t have the right TLS certificate to
> communicate with the API server, so I would try to introspect the driver
> pod and see what certificate it’s using for that communication. If there’s
> a fix that needs to happen in Spark, feel free to indicate as such.
>
>
>
> -Matt Cheah
>
>
>
> *From: *Steven Stetzler <st...@gmail.com>
> *Date: *Thursday, December 13, 2018 at 1:49 PM
> *To: *"user@spark.apache.org" <us...@spark.apache.org>
> *Subject: *Problem running Spark on Kubernetes: Certificate error
>
>
>
> Hello,
>
> I am following the tutorial here (https://spark.apache.org/docs/latest/running-on-kubernetes.html
> [spark.apache.org]
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__spark.apache.org_docs_latest_running-2Don-2Dkubernetes.html&d=DwMFaQ&c=izlc9mHr637UR4lpLEZLFFS3Vn2UXBrZ4tFb6oOnmz8&r=hzwIMNQ9E99EMYGuqHI0kXhVbvX3nU3OSDadUnJxjAs&m=koBmBBRy1RJd_Z00W1RO9fi0a72z2vYhFoj-O0FN3RQ&s=9gQXKMnjixgZA7jGBSWFYdhjnBsgzdy51B5bZJ5znQ8&e=>)
> to get spark running on a Kubernetes cluster. My Kubernetes cluster is
> hosted with Digital Ocean's kubernetes cluster manager. I have change the
> KUBECONFIG environment variable to point to my cluster access credentials,
> so both Spark and kubectl can speak with the nodes.
>
> I am running into an issue when trying to run the SparkPi example as
> described in the Spark on Kubernetes tutorials. The command I am running
> is:
>
> ./bin/spark-submit --master k8s://$CLUSTERIP --deploy-mode cluster --name
> spark-pi --class org.apache.spark.examples.SparkPi --conf
> spark.executor.instances=1 --conf
> spark.kubernetes.container.image=$IMAGEURL --conf
> spark.kubernetes.authenticate.driver.serviceAccountName=spark
> local:///opt/spark/examples/jars/spark-examples_2.11-2.4.0.jar
>
> where CLUSTERIP contains the ip of my cluster and IMAGEURL contains the
> URL of the Spark docker image I am using (https://hub.docker.com/r/stevenstetzler/spark/
> [hub.docker.com]
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__hub.docker.com_r_stevenstetzler_spark_&d=DwMFaQ&c=izlc9mHr637UR4lpLEZLFFS3Vn2UXBrZ4tFb6oOnmz8&r=hzwIMNQ9E99EMYGuqHI0kXhVbvX3nU3OSDadUnJxjAs&m=koBmBBRy1RJd_Z00W1RO9fi0a72z2vYhFoj-O0FN3RQ&s=lgzK2iTO0oHaHnD6w81b-Cwl8YAAK5XyddxSBCi35j0&e=>).
> This docker image was built and pushed with the script included in the
> Spark 2.4 distribution. I have created a service account for Spark to
> ensure that it has proper permissions to create pods etc., which I checked
> using
>
> kubectl auth can-i create pods --as=system:serviceaccount:default:spark
>
> When I try to run the SparkPi example using the above command, I get the
> following output:
>
> 2018-12-12 06:26:15 WARN Utils:66 - Your hostname, docker-test resolves
> to a loopback address: 127.0.1.1; using 10.46.0.6 instead (on interface
> eth0)
> 2018-12-12 06:26:15 WARN Utils:66 - Set SPARK_LOCAL_IP if you need to
> bind to another address
> 2018-12-12 06:26:19 INFO LoggingPodStatusWatcherImpl:54 - State changed,
> new state:
> pod name: spark-pi-1544595975520-driver
> namespace: default
> labels: spark-app-selector ->
> spark-ec5eb54644d348e7a213f8178b8ef61f, spark-role -> driver
> pod uid: d5d6bdc7-fdd6-11e8-b666-8e815d3815b2
> creation time: 2018-12-12T06:26:18Z
> service account name: spark
> volumes: spark-local-dir-1, spark-conf-volume, spark-token-qf9dt
> node name: N/A
> start time: N/A
> container images: N/A
> phase: Pending
> status: []
> 2018-12-12 06:26:19 INFO LoggingPodStatusWatcherImpl:54 - State changed,
> new state:
> pod name: spark-pi-1544595975520-driver
> namespace: default
> labels: spark-app-selector ->
> spark-ec5eb54644d348e7a213f8178b8ef61f, spark-role -> driver
> pod uid: d5d6bdc7-fdd6-11e8-b666-8e815d3815b2
> creation time: 2018-12-12T06:26:18Z
> service account name: spark
> volumes: spark-local-dir-1, spark-conf-volume, spark-token-qf9dt
> node name: flamboyant-darwin-3rhc
> start time: N/A
> container images: N/A
> phase: Pending
> status: []
> 2018-12-12 06:26:19 INFO LoggingPodStatusWatcherImpl:54 - State changed,
> new state:
> pod name: spark-pi-1544595975520-driver
> namespace: default
> labels: spark-app-selector ->
> spark-ec5eb54644d348e7a213f8178b8ef61f, spark-role -> driver
> pod uid: d5d6bdc7-fdd6-11e8-b666-8e815d3815b2
> creation time: 2018-12-12T06:26:18Z
> service account name: spark
> volumes: spark-local-dir-1, spark-conf-volume, spark-token-qf9dt
> node name: flamboyant-darwin-3rhc
> start time: 2018-12-12T06:26:18Z
> container images: docker.io/stevenstetzler/spark:v1 [docker.io]
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__docker.io_stevenstetzler_spark-3Av1&d=DwMFaQ&c=izlc9mHr637UR4lpLEZLFFS3Vn2UXBrZ4tFb6oOnmz8&r=hzwIMNQ9E99EMYGuqHI0kXhVbvX3nU3OSDadUnJxjAs&m=koBmBBRy1RJd_Z00W1RO9fi0a72z2vYhFoj-O0FN3RQ&s=O4l8wZMrO52IV576ItzHzN_REr0NCrRNnZ-0ZCB30a0&e=>
>
> phase: Pending
> status: [ContainerStatus(containerID=null, image=docker.io/stevenstetzler/spark:v1
> [docker.io]
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__docker.io_stevenstetzler_spark-3Av1&d=DwMFaQ&c=izlc9mHr637UR4lpLEZLFFS3Vn2UXBrZ4tFb6oOnmz8&r=hzwIMNQ9E99EMYGuqHI0kXhVbvX3nU3OSDadUnJxjAs&m=koBmBBRy1RJd_Z00W1RO9fi0a72z2vYhFoj-O0FN3RQ&s=O4l8wZMrO52IV576ItzHzN_REr0NCrRNnZ-0ZCB30a0&e=>,
> imageID=, lastState=ContainerState(running=null, terminated=null,
> waiting=null, additionalProperties={}), name=spark-kubernetes-driver,
> ready=false, restartCount=0, state=ContainerState(running=null,
> terminated=null, waiting=ContainerStateWaiting(message=null,
> reason=ContainerCreating, additionalProperties={}),
> additionalProperties={}), additionalProperties={})]
> 2018-12-12 06:26:19 INFO Client:54 - Waiting for application spark-pi to
> finish...
> 2018-12-12 06:26:21 INFO LoggingPodStatusWatcherImpl:54 - State changed,
> new state:
> pod name: spark-pi-1544595975520-driver
> namespace: default
> labels: spark-app-selector ->
> spark-ec5eb54644d348e7a213f8178b8ef61f, spark-role -> driver
> pod uid: d5d6bdc7-fdd6-11e8-b666-8e815d3815b2
> creation time: 2018-12-12T06:26:18Z
> service account name: spark
> volumes: spark-local-dir-1, spark-conf-volume, spark-token-qf9dt
> node name: flamboyant-darwin-3rhc
> start time: 2018-12-12T06:26:18Z
> container images: stevenstetzler/spark:v1
> phase: Running
> status:
> [ContainerStatus(containerID=docker://b923c6ff02b93557c8c104c01a4eeb1c05f3d0c0123ec4e5895bfd6be398a03a,
> image=stevenstetzler/spark:v1,
> imageID=docker-pullable://stevenstetzler/spark@sha256:dc4bce1e410ebd7b14a88caa46a4282a61ff058c6374b7cf721b7498829bb041,
> lastState=ContainerState(running=null, terminated=null, waiting=null,
> additionalProperties={}), name=spark-kubernetes-driver, ready=true,
> restartCount=0,
> state=ContainerState(running=ContainerStateRunning(startedAt=Time(time=2018-12-12T06:26:21Z,
> additionalProperties={}), additionalProperties={}), terminated=null,
> waiting=null, additionalProperties={}), additionalProperties={})]
> 2018-12-12 06:26:24 INFO LoggingPodStatusWatcherImpl:54 - State changed,
> new state:
> pod name: spark-pi-1544595975520-driver
> namespace: default
> labels: spark-app-selector ->
> spark-ec5eb54644d348e7a213f8178b8ef61f, spark-role -> driver
> pod uid: d5d6bdc7-fdd6-11e8-b666-8e815d3815b2
> creation time: 2018-12-12T06:26:18Z
> service account name: spark
> volumes: spark-local-dir-1, spark-conf-volume, spark-token-qf9dt
> node name: flamboyant-darwin-3rhc
> start time: 2018-12-12T06:26:18Z
> container images: stevenstetzler/spark:v1
> phase: Failed
> status:
> [ContainerStatus(containerID=docker://b923c6ff02b93557c8c104c01a4eeb1c05f3d0c0123ec4e5895bfd6be398a03a,
> image=stevenstetzler/spark:v1,
> imageID=docker-pullable://stevenstetzler/spark@sha256:dc4bce1e410ebd7b14a88caa46a4282a61ff058c6374b7cf721b7498829bb041,
> lastState=ContainerState(running=null, terminated=null, waiting=null,
> additionalProperties={}), name=spark-kubernetes-driver, ready=false,
> restartCount=0, state=ContainerState(running=null,
> terminated=ContainerStateTerminated(containerID=docker://b923c6ff02b93557c8c104c01a4eeb1c05f3d0c0123ec4e5895bfd6be398a03a,
> exitCode=1, finishedAt=Time(time=2018-12-12T06:26:24Z,
> additionalProperties={}), message=null, reason=Error, signal=null,
> startedAt=Time(time=2018-12-12T06:26:21Z, additionalProperties={}),
> additionalProperties={}), waiting=null, additionalProperties={}),
> additionalProperties={})]
> 2018-12-12 06:26:24 INFO LoggingPodStatusWatcherImpl:54 - Container final
> statuses:
>
>
> Container name: spark-kubernetes-driver
> Container image: stevenstetzler/spark:v1
> Container state: Terminated
> Exit code: 1
> 2018-12-12 06:26:24 INFO Client:54 - Application spark-pi finished.
> 2018-12-12 06:26:24 INFO ShutdownHookManager:54 - Shutdown hook called
> 2018-12-12 06:26:24 INFO ShutdownHookManager:54 - Deleting directory
> /tmp/spark-ca66ad4b-6526-4053-9ae3-f9ceb2e41cfd
>
> When I check the logs with
>
> kubectl logs spark-pi-1544595975520-driver
>
> I get the following:
>
> ++ id -u
> + myuid=0
> ++ id -g
> + mygid=0
> + set +e
> ++ getent passwd 0
> + uidentry=root:x:0:0:root:/root:/bin/ash
> + set -e
> + '[' -z root:x:0:0:root:/root:/bin/ash ']'
> + SPARK_K8S_CMD=driver
> + case "$SPARK_K8S_CMD" in
> + shift 1
> + SPARK_CLASSPATH=':/opt/spark/jars/*'
> + env
> + grep SPARK_JAVA_OPT_
> + sed 's/[^=]*=\(.*\)/\1/g'
> + sort -t_ -k4 -n
> + readarray -t SPARK_EXECUTOR_JAVA_OPTS
> + '[' -n '' ']'
> + '[' -n '' ']'
> + PYSPARK_ARGS=
> + '[' -n '' ']'
> + R_ARGS=
> + '[' -n '' ']'
> + '[' '' == 2 ']'
> + '[' '' == 3 ']'
> + case "$SPARK_K8S_CMD" in
> + CMD=("$SPARK_HOME/bin/spark-submit" --conf
> "spark.driver.bindAddress=$SPARK_DRIVER_BIND_ADDRESS" --deploy-mode client
> "$@")
> + exec /sbin/tini -s -- /opt/spark/bin/spark-submit --conf
> spark.driver.bindAddress=10.244.96.3 --deploy-mode client --properties-file
> /opt/spark/conf/spark.properties --class org.apache.spark.examples.SparkPi
> spark-internal
> 2018-12-12 06:26:22 WARN NativeCodeLoader:62 - Unable to load
> native-hadoop library for your platform... using builtin-java classes where
> applicable
> 2018-12-12 06:26:22 INFO SparkContext:54 - Running Spark version 2.4.0
> 2018-12-12 06:26:22 INFO SparkContext:54 - Submitted application: Spark
> Pi
> 2018-12-12 06:26:22 INFO SecurityManager:54 - Changing view acls to: root
> 2018-12-12 06:26:22 INFO SecurityManager:54 - Changing modify acls to:
> root
> 2018-12-12 06:26:22 INFO SecurityManager:54 - Changing view acls groups
> to:
> 2018-12-12 06:26:22 INFO SecurityManager:54 - Changing modify acls groups
> to:
> 2018-12-12 06:26:22 INFO SecurityManager:54 - SecurityManager:
> authentication disabled; ui acls disabled; users with view permissions:
> Set(root); groups with view permissions: Set(); users with modify
> permissions: Set(root); groups with modify permissions: Set()
> 2018-12-12 06:26:23 INFO Utils:54 - Successfully started service
> 'sparkDriver' on port 7078.
> 2018-12-12 06:26:23 INFO SparkEnv:54 - Registering MapOutputTracker
> 2018-12-12 06:26:23 INFO SparkEnv:54 - Registering BlockManagerMaster
> 2018-12-12 06:26:23 INFO BlockManagerMasterEndpoint:54 - Using
> org.apache.spark.storage.DefaultTopologyMapper for getting topology
> information
> 2018-12-12 06:26:23 INFO BlockManagerMasterEndpoint:54 -
> BlockManagerMasterEndpoint up
> 2018-12-12 06:26:23 INFO DiskBlockManager:54 - Created local directory at
> /var/data/spark-c306f3bc-cc7c-4090-a7f7-22fbc02b72bd/blockmgr-63358d1a-1e7e-493e-accd-7992da9b0adf
> 2018-12-12 06:26:23 INFO MemoryStore:54 - MemoryStore started with
> capacity 413.9 MB
> 2018-12-12 06:26:23 INFO SparkEnv:54 - Registering
> OutputCommitCoordinator
> 2018-12-12 06:26:23 INFO log:192 - Logging initialized @2059ms
> 2018-12-12 06:26:23 INFO Server:351 - jetty-9.3.z-SNAPSHOT, build
> timestamp: unknown, git hash: unknown
> 2018-12-12 06:26:23 INFO Server:419 - Started @2155ms
> 2018-12-12 06:26:23 INFO AbstractConnector:278 - Started
> ServerConnector@7180e701{HTTP/1.1,[http/1.1]}{0.0.0.0:4040 [0.0.0.0]
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__0.0.0.0-3A4040&d=DwMFaQ&c=izlc9mHr637UR4lpLEZLFFS3Vn2UXBrZ4tFb6oOnmz8&r=hzwIMNQ9E99EMYGuqHI0kXhVbvX3nU3OSDadUnJxjAs&m=koBmBBRy1RJd_Z00W1RO9fi0a72z2vYhFoj-O0FN3RQ&s=NEpJ4w980MUskRPIbAZm04Qx67LafCpnymbJwyzat4Y&e=>
> }
> 2018-12-12 06:26:23 INFO Utils:54 - Successfully started service
> 'SparkUI' on port 4040.
> 2018-12-12 06:26:23 INFO ContextHandler:781 - Started
> o.s.j.s.ServletContextHandler@9f6e406{/jobs,null,AVAILABLE,@Spark}
> 2018-12-12 06:26:23 INFO ContextHandler:781 - Started
> o.s.j.s.ServletContextHandler@3a60c416{/jobs/json,null,AVAILABLE,@Spark}
> 2018-12-12 06:26:23 INFO ContextHandler:781 - Started
> o.s.j.s.ServletContextHandler@60f2e0bd{/jobs/job,null,AVAILABLE,@Spark}
> 2018-12-12 06:26:23 INFO ContextHandler:781 - Started
> o.s.j.s.ServletContextHandler@6b410923
> {/jobs/job/json,null,AVAILABLE,@Spark}
> 2018-12-12 06:26:23 INFO ContextHandler:781 - Started
> o.s.j.s.ServletContextHandler@57bd2029{/stages,null,AVAILABLE,@Spark}
> 2018-12-12 06:26:23 INFO ContextHandler:781 - Started
> o.s.j.s.ServletContextHandler@203dd56b
> {/stages/json,null,AVAILABLE,@Spark}
> 2018-12-12 06:26:23 INFO ContextHandler:781 - Started
> o.s.j.s.ServletContextHandler@e8fadb0
> {/stages/stage,null,AVAILABLE,@Spark}
> 2018-12-12 06:26:23 INFO ContextHandler:781 - Started
> o.s.j.s.ServletContextHandler@1d3e6d34
> {/stages/stage/json,null,AVAILABLE,@Spark}
> 2018-12-12 06:26:23 INFO ContextHandler:781 - Started
> o.s.j.s.ServletContextHandler@6eafb10e
> {/stages/pool,null,AVAILABLE,@Spark}
> 2018-12-12 06:26:23 INFO ContextHandler:781 - Started
> o.s.j.s.ServletContextHandler@26a94fa5
> {/stages/pool/json,null,AVAILABLE,@Spark}
> 2018-12-12 06:26:23 INFO ContextHandler:781 - Started
> o.s.j.s.ServletContextHandler@464a4442{/storage,null,AVAILABLE,@Spark}
> 2018-12-12 06:26:23 INFO ContextHandler:781 - Started
> o.s.j.s.ServletContextHandler@2873d672
> {/storage/json,null,AVAILABLE,@Spark}
> 2018-12-12 06:26:23 INFO ContextHandler:781 - Started
> o.s.j.s.ServletContextHandler@3bc735b3
> {/storage/rdd,null,AVAILABLE,@Spark}
> 2018-12-12 06:26:23 INFO ContextHandler:781 - Started
> o.s.j.s.ServletContextHandler@577f9109
> {/storage/rdd/json,null,AVAILABLE,@Spark}
> 2018-12-12 06:26:23 INFO ContextHandler:781 - Started
> o.s.j.s.ServletContextHandler@4303b7f0
> {/environment,null,AVAILABLE,@Spark}
> 2018-12-12 06:26:23 INFO ContextHandler:781 - Started
> o.s.j.s.ServletContextHandler@757529a4
> {/environment/json,null,AVAILABLE,@Spark}
> 2018-12-12 06:26:23 INFO ContextHandler:781 - Started
> o.s.j.s.ServletContextHandler@779de014{/executors,null,AVAILABLE,@Spark}
> 2018-12-12 06:26:23 INFO ContextHandler:781 - Started
> o.s.j.s.ServletContextHandler@5c41d037
> {/executors/json,null,AVAILABLE,@Spark}
> 2018-12-12 06:26:23 INFO ContextHandler:781 - Started
> o.s.j.s.ServletContextHandler@2234078
> {/executors/threadDump,null,AVAILABLE,@Spark}
> 2018-12-12 06:26:23 INFO ContextHandler:781 - Started
> o.s.j.s.ServletContextHandler@5ec77191
> {/executors/threadDump/json,null,AVAILABLE,@Spark}
> 2018-12-12 06:26:23 INFO ContextHandler:781 - Started
> o.s.j.s.ServletContextHandler@4642b71d{/static,null,AVAILABLE,@Spark}
> 2018-12-12 06:26:23 INFO ContextHandler:781 - Started
> o.s.j.s.ServletContextHandler@7ed9ae94{/,null,AVAILABLE,@Spark}
> 2018-12-12 06:26:23 INFO ContextHandler:781 - Started
> o.s.j.s.ServletContextHandler@66908383{/api,null,AVAILABLE,@Spark}
> 2018-12-12 06:26:23 INFO ContextHandler:781 - Started
> o.s.j.s.ServletContextHandler@534ca02b
> {/jobs/job/kill,null,AVAILABLE,@Spark}
> 2018-12-12 06:26:23 INFO ContextHandler:781 - Started
> o.s.j.s.ServletContextHandler@29a23c3d
> {/stages/stage/kill,null,AVAILABLE,@Spark}
> 2018-12-12 06:26:23 INFO SparkUI:54 - Bound SparkUI to 0.0.0.0, and
> started at http://spark-pi-1544595975520-driver-svc.default.svc:4040
> [spark-pi-1544595975520-driver-svc.default.svc]
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__spark-2Dpi-2D1544595975520-2Ddriver-2Dsvc.default.svc-3A4040_&d=DwMFaQ&c=izlc9mHr637UR4lpLEZLFFS3Vn2UXBrZ4tFb6oOnmz8&r=hzwIMNQ9E99EMYGuqHI0kXhVbvX3nU3OSDadUnJxjAs&m=koBmBBRy1RJd_Z00W1RO9fi0a72z2vYhFoj-O0FN3RQ&s=7_pmhkpOYkR7IRMQolPOXsfzMG9byW5WEPVrfs4UgCg&e=>
> 2018-12-12 06:26:23 INFO SparkContext:54 - Added JAR
> file:///opt/spark/examples/jars/spark-examples_2.11-2.4.0.jar at
> spark://spark-pi-1544595975520-driver-svc.default.svc:7078/jars/spark-examples_2.11-2.4.0.jar
> with timestamp 1544595983645
> 2018-12-12 06:26:23 ERROR SparkContext:91 - Error initializing
> SparkContext.
> org.apache.spark.SparkException: External scheduler cannot be instantiated
> at
> org.apache.spark.SparkContext$.org$apache$spark$SparkContext$$createTaskScheduler(SparkContext.scala:2794)
> at org.apache.spark.SparkContext.<init>(SparkContext.scala:493)
> at
> org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2520)
> at
> org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:935)
> at
> org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:926)
> at scala.Option.getOrElse(Option.scala:121)
> at
> org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:926)
> at org.apache.spark.examples.SparkPi$.main(SparkPi.scala:31)
> at org.apache.spark.examples.SparkPi.main(SparkPi.scala)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
> at org.apache.spark.deploy.SparkSubmit.org
> [org.apache.spark.deploy.sparksubmit.org]
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__org.apache.spark.deploy.SparkSubmit.org&d=DwMFaQ&c=izlc9mHr637UR4lpLEZLFFS3Vn2UXBrZ4tFb6oOnmz8&r=hzwIMNQ9E99EMYGuqHI0kXhVbvX3nU3OSDadUnJxjAs&m=koBmBBRy1RJd_Z00W1RO9fi0a72z2vYhFoj-O0FN3RQ&s=UFf1bgM3AcQSaqrUz94wpKROIXo4lHTVmIa6KAOv2vo&e=>
> $apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:849)
> at
> org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:167)
> at
> org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:195)
> at
> org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
> at
> org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:924)
> at
> org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:933)
> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
> Caused by: io.fabric8.kubernetes.client.KubernetesClientException: An
> error has occurred.
> at
> io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:62)
> at
> io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:53)
> at
> io.fabric8.kubernetes.client.utils.HttpClientUtils.createHttpClient(HttpClientUtils.java:167)
> at
> org.apache.spark.deploy.k8s.SparkKubernetesClientFactory$.createKubernetesClient(SparkKubernetesClientFactory.scala:84)
> at
> org.apache.spark.scheduler.cluster.k8s.KubernetesClusterManager.createSchedulerBackend(KubernetesClusterManager.scala:64)
> at
> org.apache.spark.SparkContext$.org$apache$spark$SparkContext$$createTaskScheduler(SparkContext.scala:2788)
> ... 20 more
> Caused by: java.security.cert.CertificateException: Could not parse
> certificate: java.io.IOException: Empty input
> at
> sun.security.provider.X509Factory.engineGenerateCertificate(X509Factory.java:110)
> at
> java.security.cert.CertificateFactory.generateCertificate(CertificateFactory.java:339)
> at
> io.fabric8.kubernetes.client.internal.CertUtils.createTrustStore(CertUtils.java:93)
> at
> io.fabric8.kubernetes.client.internal.CertUtils.createTrustStore(CertUtils.java:71)
> at
> io.fabric8.kubernetes.client.internal.SSLUtils.trustManagers(SSLUtils.java:114)
> at
> io.fabric8.kubernetes.client.internal.SSLUtils.trustManagers(SSLUtils.java:93)
> at
> io.fabric8.kubernetes.client.utils.HttpClientUtils.createHttpClient(HttpClientUtils.java:63)
> ... 23 more
> Caused by: java.io.IOException: Empty input
> at
> sun.security.provider.X509Factory.engineGenerateCertificate(X509Factory.java:106)
> ... 29 more
> 2018-12-12 06:26:23 INFO AbstractConnector:318 - Stopped Spark@7180e701
> {HTTP/1.1,[http/1.1]}{0.0.0.0:4040 [0.0.0.0]
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__0.0.0.0-3A4040&d=DwMFaQ&c=izlc9mHr637UR4lpLEZLFFS3Vn2UXBrZ4tFb6oOnmz8&r=hzwIMNQ9E99EMYGuqHI0kXhVbvX3nU3OSDadUnJxjAs&m=koBmBBRy1RJd_Z00W1RO9fi0a72z2vYhFoj-O0FN3RQ&s=NEpJ4w980MUskRPIbAZm04Qx67LafCpnymbJwyzat4Y&e=>
> }
> 2018-12-12 06:26:23 INFO SparkUI:54 - Stopped Spark web UI at http://spark-pi-1544595975520-driver-svc.default.svc:4040
> [spark-pi-1544595975520-driver-svc.default.svc]
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__spark-2Dpi-2D1544595975520-2Ddriver-2Dsvc.default.svc-3A4040_&d=DwMFaQ&c=izlc9mHr637UR4lpLEZLFFS3Vn2UXBrZ4tFb6oOnmz8&r=hzwIMNQ9E99EMYGuqHI0kXhVbvX3nU3OSDadUnJxjAs&m=koBmBBRy1RJd_Z00W1RO9fi0a72z2vYhFoj-O0FN3RQ&s=7_pmhkpOYkR7IRMQolPOXsfzMG9byW5WEPVrfs4UgCg&e=>
> 2018-12-12 06:26:23 INFO MapOutputTrackerMasterEndpoint:54 -
> MapOutputTrackerMasterEndpoint stopped!
> 2018-12-12 06:26:23 INFO MemoryStore:54 - MemoryStore cleared
> 2018-12-12 06:26:23 INFO BlockManager:54 - BlockManager stopped
> 2018-12-12 06:26:23 INFO BlockManagerMaster:54 - BlockManagerMaster
> stopped
> 2018-12-12 06:26:23 WARN MetricsSystem:66 - Stopping a MetricsSystem that
> is not running
> 2018-12-12 06:26:23 INFO
> OutputCommitCoordinator$OutputCommitCoordinatorEndpoint:54 -
> OutputCommitCoordinator stopped!
> 2018-12-12 06:26:23 INFO SparkContext:54 - Successfully stopped
> SparkContext
> Exception in thread "main" org.apache.spark.SparkException: External
> scheduler cannot be instantiated
> at
> org.apache.spark.SparkContext$.org$apache$spark$SparkContext$$createTaskScheduler(SparkContext.scala:2794)
> at org.apache.spark.SparkContext.<init>(SparkContext.scala:493)
> at
> org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2520)
> at
> org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:935)
> at
> org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:926)
> at scala.Option.getOrElse(Option.scala:121)
> at
> org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:926)
> at org.apache.spark.examples.SparkPi$.main(SparkPi.scala:31)
> at org.apache.spark.examples.SparkPi.main(SparkPi.scala)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
> at org.apache.spark.deploy.SparkSubmit.org
> [org.apache.spark.deploy.sparksubmit.org]
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__org.apache.spark.deploy.SparkSubmit.org&d=DwMFaQ&c=izlc9mHr637UR4lpLEZLFFS3Vn2UXBrZ4tFb6oOnmz8&r=hzwIMNQ9E99EMYGuqHI0kXhVbvX3nU3OSDadUnJxjAs&m=koBmBBRy1RJd_Z00W1RO9fi0a72z2vYhFoj-O0FN3RQ&s=UFf1bgM3AcQSaqrUz94wpKROIXo4lHTVmIa6KAOv2vo&e=>
> $apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:849)
> at
> org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:167)
> at
> org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:195)
> at
> org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
> at
> org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:924)
> at
> org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:933)
> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
> Caused by: io.fabric8.kubernetes.client.KubernetesClientException: An
> error has occurred.
> at
> io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:62)
> at
> io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:53)
> at
> io.fabric8.kubernetes.client.utils.HttpClientUtils.createHttpClient(HttpClientUtils.java:167)
> at
> org.apache.spark.deploy.k8s.SparkKubernetesClientFactory$.createKubernetesClient(SparkKubernetesClientFactory.scala:84)
> at
> org.apache.spark.scheduler.cluster.k8s.KubernetesClusterManager.createSchedulerBackend(KubernetesClusterManager.scala:64)
> at
> org.apache.spark.SparkContext$.org$apache$spark$SparkContext$$createTaskScheduler(SparkContext.scala:2788)
> ... 20 more
> Caused by: java.security.cert.CertificateException: Could not parse
> certificate: java.io.IOException: Empty input
> at
> sun.security.provider.X509Factory.engineGenerateCertificate(X509Factory.java:110)
> at
> java.security.cert.CertificateFactory.generateCertificate(CertificateFactory.java:339)
> at
> io.fabric8.kubernetes.client.internal.CertUtils.createTrustStore(CertUtils.java:93)
> at
> io.fabric8.kubernetes.client.internal.CertUtils.createTrustStore(CertUtils.java:71)
> at
> io.fabric8.kubernetes.client.internal.SSLUtils.trustManagers(SSLUtils.java:114)
> at
> io.fabric8.kubernetes.client.internal.SSLUtils.trustManagers(SSLUtils.java:93)
> at
> io.fabric8.kubernetes.client.utils.HttpClientUtils.createHttpClient(HttpClientUtils.java:63)
> ... 23 more
> Caused by: java.io.IOException: Empty input
> at
> sun.security.provider.X509Factory.engineGenerateCertificate(X509Factory.java:106)
> ... 29 more
> 2018-12-12 06:26:23 INFO ShutdownHookManager:54 - Shutdown hook called
> 2018-12-12 06:26:23 INFO ShutdownHookManager:54 - Deleting directory
> /var/data/spark-c306f3bc-cc7c-4090-a7f7-22fbc02b72bd/spark-157bde7a-d0e1-4092-9e39-5ab2b727e3e7
> 2018-12-12 06:26:23 INFO ShutdownHookManager:54 - Deleting directory
> /tmp/spark-07910ccb-c212-4622-9c1b-3e8b772f3faa
>
> It looks to me like there is a communication / authentication issue during
> pod creation, and it looks like there might be a certificate missing that
> Spark needs to communicate with Kubernetes, which I infer from the
> "java.security.cert.CertificateException: Could not parse certificate:
> java.io.IOException: Empty input" error message.
>
> Any ideas on what I can do to resolve this? Thanks!
>
Re: Problem running Spark on Kubernetes: Certificate error
Posted by Matt Cheah <mc...@palantir.com>.
Hi Steven,
What I think is happening is that your machine has a CA certificate that is used for communicating with your API server, particularly because you’re using Digital Ocean’s cluster manager. However, it’s unclear if your pod has the same CA certificate or if the pod needs that certificate file. You can use configurations to have your pod use a particular CA certificate file to communicate with the APi server. If you set spark.kubernetes.authenticate.driver.caCertFile to the path of your CA certificate on your local disk, spark-submit will create a secret that contains that certificate file and use that certificate to configure TLS for the driver pod’s communication with the API server.
It's clear that your driver pod doesn’t have the right TLS certificate to communicate with the API server, so I would try to introspect the driver pod and see what certificate it’s using for that communication. If there’s a fix that needs to happen in Spark, feel free to indicate as such.
-Matt Cheah
From: Steven Stetzler <st...@gmail.com>
Date: Thursday, December 13, 2018 at 1:49 PM
To: "user@spark.apache.org" <us...@spark.apache.org>
Subject: Problem running Spark on Kubernetes: Certificate error
Hello,
I am following the tutorial here (https://spark.apache.org/docs/latest/running-on-kubernetes.html [spark.apache.org]) to get spark running on a Kubernetes cluster. My Kubernetes cluster is hosted with Digital Ocean's kubernetes cluster manager. I have change the KUBECONFIG environment variable to point to my cluster access credentials, so both Spark and kubectl can speak with the nodes.
I am running into an issue when trying to run the SparkPi example as described in the Spark on Kubernetes tutorials. The command I am running is:
./bin/spark-submit --master k8s://$CLUSTERIP --deploy-mode cluster --name spark-pi --class org.apache.spark.examples.SparkPi --conf spark.executor.instances=1 --conf spark.kubernetes.container.image=$IMAGEURL --conf spark.kubernetes.authenticate.driver.serviceAccountName=spark local:///opt/spark/examples/jars/spark-examples_2.11-2.4.0.jar
where CLUSTERIP contains the ip of my cluster and IMAGEURL contains the URL of the Spark docker image I am using (https://hub.docker.com/r/stevenstetzler/spark/ [hub.docker.com]). This docker image was built and pushed with the script included in the Spark 2.4 distribution. I have created a service account for Spark to ensure that it has proper permissions to create pods etc., which I checked using
kubectl auth can-i create pods --as=system:serviceaccount:default:spark
When I try to run the SparkPi example using the above command, I get the following output:
2018-12-12 06:26:15 WARN Utils:66 - Your hostname, docker-test resolves to a loopback address: 127.0.1.1; using 10.46.0.6 instead (on interface eth0)
2018-12-12 06:26:15 WARN Utils:66 - Set SPARK_LOCAL_IP if you need to bind to another address
2018-12-12 06:26:19 INFO LoggingPodStatusWatcherImpl:54 - State changed, new state:
pod name: spark-pi-1544595975520-driver
namespace: default
labels: spark-app-selector -> spark-ec5eb54644d348e7a213f8178b8ef61f, spark-role -> driver
pod uid: d5d6bdc7-fdd6-11e8-b666-8e815d3815b2
creation time: 2018-12-12T06:26:18Z
service account name: spark
volumes: spark-local-dir-1, spark-conf-volume, spark-token-qf9dt
node name: N/A
start time: N/A
container images: N/A
phase: Pending
status: []
2018-12-12 06:26:19 INFO LoggingPodStatusWatcherImpl:54 - State changed, new state:
pod name: spark-pi-1544595975520-driver
namespace: default
labels: spark-app-selector -> spark-ec5eb54644d348e7a213f8178b8ef61f, spark-role -> driver
pod uid: d5d6bdc7-fdd6-11e8-b666-8e815d3815b2
creation time: 2018-12-12T06:26:18Z
service account name: spark
volumes: spark-local-dir-1, spark-conf-volume, spark-token-qf9dt
node name: flamboyant-darwin-3rhc
start time: N/A
container images: N/A
phase: Pending
status: []
2018-12-12 06:26:19 INFO LoggingPodStatusWatcherImpl:54 - State changed, new state:
pod name: spark-pi-1544595975520-driver
namespace: default
labels: spark-app-selector -> spark-ec5eb54644d348e7a213f8178b8ef61f, spark-role -> driver
pod uid: d5d6bdc7-fdd6-11e8-b666-8e815d3815b2
creation time: 2018-12-12T06:26:18Z
service account name: spark
volumes: spark-local-dir-1, spark-conf-volume, spark-token-qf9dt
node name: flamboyant-darwin-3rhc
start time: 2018-12-12T06:26:18Z
container images: docker.io/stevenstetzler/spark:v1 [docker.io]
phase: Pending
status: [ContainerStatus(containerID=null, image=docker.io/stevenstetzler/spark:v1 [docker.io], imageID=, lastState=ContainerState(running=null, terminated=null, waiting=null, additionalProperties={}), name=spark-kubernetes-driver, ready=false, restartCount=0, state=ContainerState(running=null, terminated=null, waiting=ContainerStateWaiting(message=null, reason=ContainerCreating, additionalProperties={}), additionalProperties={}), additionalProperties={})]
2018-12-12 06:26:19 INFO Client:54 - Waiting for application spark-pi to finish...
2018-12-12 06:26:21 INFO LoggingPodStatusWatcherImpl:54 - State changed, new state:
pod name: spark-pi-1544595975520-driver
namespace: default
labels: spark-app-selector -> spark-ec5eb54644d348e7a213f8178b8ef61f, spark-role -> driver
pod uid: d5d6bdc7-fdd6-11e8-b666-8e815d3815b2
creation time: 2018-12-12T06:26:18Z
service account name: spark
volumes: spark-local-dir-1, spark-conf-volume, spark-token-qf9dt
node name: flamboyant-darwin-3rhc
start time: 2018-12-12T06:26:18Z
container images: stevenstetzler/spark:v1
phase: Running
status: [ContainerStatus(containerID=docker://b923c6ff02b93557c8c104c01a4eeb1c05f3d0c0123ec4e5895bfd6be398a03a, image=stevenstetzler/spark:v1, imageID=docker-pullable://stevenstetzler/spark@sha256:dc4bce1e410ebd7b14a88caa46a4282a61ff058c6374b7cf721b7498829bb041, lastState=ContainerState(running=null, terminated=null, waiting=null, additionalProperties={}), name=spark-kubernetes-driver, ready=true, restartCount=0, state=ContainerState(running=ContainerStateRunning(startedAt=Time(time=2018-12-12T06:26:21Z, additionalProperties={}), additionalProperties={}), terminated=null, waiting=null, additionalProperties={}), additionalProperties={})]
2018-12-12 06:26:24 INFO LoggingPodStatusWatcherImpl:54 - State changed, new state:
pod name: spark-pi-1544595975520-driver
namespace: default
labels: spark-app-selector -> spark-ec5eb54644d348e7a213f8178b8ef61f, spark-role -> driver
pod uid: d5d6bdc7-fdd6-11e8-b666-8e815d3815b2
creation time: 2018-12-12T06:26:18Z
service account name: spark
volumes: spark-local-dir-1, spark-conf-volume, spark-token-qf9dt
node name: flamboyant-darwin-3rhc
start time: 2018-12-12T06:26:18Z
container images: stevenstetzler/spark:v1
phase: Failed
status: [ContainerStatus(containerID=docker://b923c6ff02b93557c8c104c01a4eeb1c05f3d0c0123ec4e5895bfd6be398a03a, image=stevenstetzler/spark:v1, imageID=docker-pullable://stevenstetzler/spark@sha256:dc4bce1e410ebd7b14a88caa46a4282a61ff058c6374b7cf721b7498829bb041, lastState=ContainerState(running=null, terminated=null, waiting=null, additionalProperties={}), name=spark-kubernetes-driver, ready=false, restartCount=0, state=ContainerState(running=null, terminated=ContainerStateTerminated(containerID=docker://b923c6ff02b93557c8c104c01a4eeb1c05f3d0c0123ec4e5895bfd6be398a03a, exitCode=1, finishedAt=Time(time=2018-12-12T06:26:24Z, additionalProperties={}), message=null, reason=Error, signal=null, startedAt=Time(time=2018-12-12T06:26:21Z, additionalProperties={}), additionalProperties={}), waiting=null, additionalProperties={}), additionalProperties={})]
2018-12-12 06:26:24 INFO LoggingPodStatusWatcherImpl:54 - Container final statuses:
Container name: spark-kubernetes-driver
Container image: stevenstetzler/spark:v1
Container state: Terminated
Exit code: 1
2018-12-12 06:26:24 INFO Client:54 - Application spark-pi finished.
2018-12-12 06:26:24 INFO ShutdownHookManager:54 - Shutdown hook called
2018-12-12 06:26:24 INFO ShutdownHookManager:54 - Deleting directory /tmp/spark-ca66ad4b-6526-4053-9ae3-f9ceb2e41cfd
When I check the logs with
kubectl logs spark-pi-1544595975520-driver
I get the following:
++ id -u
+ myuid=0
++ id -g
+ mygid=0
+ set +e
++ getent passwd 0
+ uidentry=root:x:0:0:root:/root:/bin/ash
+ set -e
+ '[' -z root:x:0:0:root:/root:/bin/ash ']'
+ SPARK_K8S_CMD=driver
+ case "$SPARK_K8S_CMD" in
+ shift 1
+ SPARK_CLASSPATH=':/opt/spark/jars/*'
+ env
+ grep SPARK_JAVA_OPT_
+ sed 's/[^=]*=\(.*\)/\1/g'
+ sort -t_ -k4 -n
+ readarray -t SPARK_EXECUTOR_JAVA_OPTS
+ '[' -n '' ']'
+ '[' -n '' ']'
+ PYSPARK_ARGS=
+ '[' -n '' ']'
+ R_ARGS=
+ '[' -n '' ']'
+ '[' '' == 2 ']'
+ '[' '' == 3 ']'
+ case "$SPARK_K8S_CMD" in
+ CMD=("$SPARK_HOME/bin/spark-submit" --conf "spark.driver.bindAddress=$SPARK_DRIVER_BIND_ADDRESS" --deploy-mode client "$@")
+ exec /sbin/tini -s -- /opt/spark/bin/spark-submit --conf spark.driver.bindAddress=10.244.96.3 --deploy-mode client --properties-file /opt/spark/conf/spark.properties --class org.apache.spark.examples.SparkPi spark-internal
2018-12-12 06:26:22 WARN NativeCodeLoader:62 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2018-12-12 06:26:22 INFO SparkContext:54 - Running Spark version 2.4.0
2018-12-12 06:26:22 INFO SparkContext:54 - Submitted application: Spark Pi
2018-12-12 06:26:22 INFO SecurityManager:54 - Changing view acls to: root
2018-12-12 06:26:22 INFO SecurityManager:54 - Changing modify acls to: root
2018-12-12 06:26:22 INFO SecurityManager:54 - Changing view acls groups to:
2018-12-12 06:26:22 INFO SecurityManager:54 - Changing modify acls groups to:
2018-12-12 06:26:22 INFO SecurityManager:54 - SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(root); groups with view permissions: Set(); users with modify permissions: Set(root); groups with modify permissions: Set()
2018-12-12 06:26:23 INFO Utils:54 - Successfully started service 'sparkDriver' on port 7078.
2018-12-12 06:26:23 INFO SparkEnv:54 - Registering MapOutputTracker
2018-12-12 06:26:23 INFO SparkEnv:54 - Registering BlockManagerMaster
2018-12-12 06:26:23 INFO BlockManagerMasterEndpoint:54 - Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information
2018-12-12 06:26:23 INFO BlockManagerMasterEndpoint:54 - BlockManagerMasterEndpoint up
2018-12-12 06:26:23 INFO DiskBlockManager:54 - Created local directory at /var/data/spark-c306f3bc-cc7c-4090-a7f7-22fbc02b72bd/blockmgr-63358d1a-1e7e-493e-accd-7992da9b0adf
2018-12-12 06:26:23 INFO MemoryStore:54 - MemoryStore started with capacity 413.9 MB
2018-12-12 06:26:23 INFO SparkEnv:54 - Registering OutputCommitCoordinator
2018-12-12 06:26:23 INFO log:192 - Logging initialized @2059ms
2018-12-12 06:26:23 INFO Server:351 - jetty-9.3.z-SNAPSHOT, build timestamp: unknown, git hash: unknown
2018-12-12 06:26:23 INFO Server:419 - Started @2155ms
2018-12-12 06:26:23 INFO AbstractConnector:278 - Started ServerConnector@7180e701{HTTP/1.1,[http/1.1]}{0.0.0.0:4040 [0.0.0.0]}
2018-12-12 06:26:23 INFO Utils:54 - Successfully started service 'SparkUI' on port 4040.
2018-12-12 06:26:23 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@9f6e406{/jobs,null,AVAILABLE,@Spark}
2018-12-12 06:26:23 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@3a60c416{/jobs/json,null,AVAILABLE,@Spark}
2018-12-12 06:26:23 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@60f2e0bd{/jobs/job,null,AVAILABLE,@Spark}
2018-12-12 06:26:23 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@6b410923{/jobs/job/json,null,AVAILABLE,@Spark}
2018-12-12 06:26:23 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@57bd2029{/stages,null,AVAILABLE,@Spark}
2018-12-12 06:26:23 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@203dd56b{/stages/json,null,AVAILABLE,@Spark}
2018-12-12 06:26:23 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@e8fadb0{/stages/stage,null,AVAILABLE,@Spark}
2018-12-12 06:26:23 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@1d3e6d34{/stages/stage/json,null,AVAILABLE,@Spark}
2018-12-12 06:26:23 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@6eafb10e{/stages/pool,null,AVAILABLE,@Spark}
2018-12-12 06:26:23 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@26a94fa5{/stages/pool/json,null,AVAILABLE,@Spark}
2018-12-12 06:26:23 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@464a4442{/storage,null,AVAILABLE,@Spark}
2018-12-12 06:26:23 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@2873d672{/storage/json,null,AVAILABLE,@Spark}
2018-12-12 06:26:23 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@3bc735b3{/storage/rdd,null,AVAILABLE,@Spark}
2018-12-12 06:26:23 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@577f9109{/storage/rdd/json,null,AVAILABLE,@Spark}
2018-12-12 06:26:23 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@4303b7f0{/environment,null,AVAILABLE,@Spark}
2018-12-12 06:26:23 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@757529a4{/environment/json,null,AVAILABLE,@Spark}
2018-12-12 06:26:23 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@779de014{/executors,null,AVAILABLE,@Spark}
2018-12-12 06:26:23 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@5c41d037{/executors/json,null,AVAILABLE,@Spark}
2018-12-12 06:26:23 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@2234078{/executors/threadDump,null,AVAILABLE,@Spark}
2018-12-12 06:26:23 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@5ec77191{/executors/threadDump/json,null,AVAILABLE,@Spark}
2018-12-12 06:26:23 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@4642b71d{/static,null,AVAILABLE,@Spark}
2018-12-12 06:26:23 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@7ed9ae94{/,null,AVAILABLE,@Spark}
2018-12-12 06:26:23 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@66908383{/api,null,AVAILABLE,@Spark}
2018-12-12 06:26:23 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@534ca02b{/jobs/job/kill,null,AVAILABLE,@Spark}
2018-12-12 06:26:23 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@29a23c3d{/stages/stage/kill,null,AVAILABLE,@Spark}
2018-12-12 06:26:23 INFO SparkUI:54 - Bound SparkUI to 0.0.0.0, and started at http://spark-pi-1544595975520-driver-svc.default.svc:4040 [spark-pi-1544595975520-driver-svc.default.svc]
2018-12-12 06:26:23 INFO SparkContext:54 - Added JAR file:///opt/spark/examples/jars/spark-examples_2.11-2.4.0.jar at spark://spark-pi-1544595975520-driver-svc.default.svc:7078/jars/spark-examples_2.11-2.4.0.jar with timestamp 1544595983645
2018-12-12 06:26:23 ERROR SparkContext:91 - Error initializing SparkContext.
org.apache.spark.SparkException: External scheduler cannot be instantiated
at org.apache.spark.SparkContext$.org$apache$spark$SparkContext$$createTaskScheduler(SparkContext.scala:2794)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:493)
at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2520)
at org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:935)
at org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:926)
at scala.Option.getOrElse(Option.scala:121)
at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:926)
at org.apache.spark.examples.SparkPi$.main(SparkPi.scala:31)
at org.apache.spark.examples.SparkPi.main(SparkPi.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
at org.apache.spark.deploy.SparkSubmit.org [org.apache.spark.deploy.sparksubmit.org]$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:849)
at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:167)
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:195)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:924)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:933)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: io.fabric8.kubernetes.client.KubernetesClientException: An error has occurred.
at io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:62)
at io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:53)
at io.fabric8.kubernetes.client.utils.HttpClientUtils.createHttpClient(HttpClientUtils.java:167)
at org.apache.spark.deploy.k8s.SparkKubernetesClientFactory$.createKubernetesClient(SparkKubernetesClientFactory.scala:84)
at org.apache.spark.scheduler.cluster.k8s.KubernetesClusterManager.createSchedulerBackend(KubernetesClusterManager.scala:64)
at org.apache.spark.SparkContext$.org$apache$spark$SparkContext$$createTaskScheduler(SparkContext.scala:2788)
... 20 more
Caused by: java.security.cert.CertificateException: Could not parse certificate: java.io.IOException: Empty input
at sun.security.provider.X509Factory.engineGenerateCertificate(X509Factory.java:110)
at java.security.cert.CertificateFactory.generateCertificate(CertificateFactory.java:339)
at io.fabric8.kubernetes.client.internal.CertUtils.createTrustStore(CertUtils.java:93)
at io.fabric8.kubernetes.client.internal.CertUtils.createTrustStore(CertUtils.java:71)
at io.fabric8.kubernetes.client.internal.SSLUtils.trustManagers(SSLUtils.java:114)
at io.fabric8.kubernetes.client.internal.SSLUtils.trustManagers(SSLUtils.java:93)
at io.fabric8.kubernetes.client.utils.HttpClientUtils.createHttpClient(HttpClientUtils.java:63)
... 23 more
Caused by: java.io.IOException: Empty input
at sun.security.provider.X509Factory.engineGenerateCertificate(X509Factory.java:106)
... 29 more
2018-12-12 06:26:23 INFO AbstractConnector:318 - Stopped Spark@7180e701{HTTP/1.1,[http/1.1]}{0.0.0.0:4040 [0.0.0.0]}
2018-12-12 06:26:23 INFO SparkUI:54 - Stopped Spark web UI at http://spark-pi-1544595975520-driver-svc.default.svc:4040 [spark-pi-1544595975520-driver-svc.default.svc]
2018-12-12 06:26:23 INFO MapOutputTrackerMasterEndpoint:54 - MapOutputTrackerMasterEndpoint stopped!
2018-12-12 06:26:23 INFO MemoryStore:54 - MemoryStore cleared
2018-12-12 06:26:23 INFO BlockManager:54 - BlockManager stopped
2018-12-12 06:26:23 INFO BlockManagerMaster:54 - BlockManagerMaster stopped
2018-12-12 06:26:23 WARN MetricsSystem:66 - Stopping a MetricsSystem that is not running
2018-12-12 06:26:23 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint:54 - OutputCommitCoordinator stopped!
2018-12-12 06:26:23 INFO SparkContext:54 - Successfully stopped SparkContext
Exception in thread "main" org.apache.spark.SparkException: External scheduler cannot be instantiated
at org.apache.spark.SparkContext$.org$apache$spark$SparkContext$$createTaskScheduler(SparkContext.scala:2794)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:493)
at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2520)
at org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:935)
at org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:926)
at scala.Option.getOrElse(Option.scala:121)
at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:926)
at org.apache.spark.examples.SparkPi$.main(SparkPi.scala:31)
at org.apache.spark.examples.SparkPi.main(SparkPi.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
at org.apache.spark.deploy.SparkSubmit.org [org.apache.spark.deploy.sparksubmit.org]$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:849)
at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:167)
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:195)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:924)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:933)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: io.fabric8.kubernetes.client.KubernetesClientException: An error has occurred.
at io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:62)
at io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:53)
at io.fabric8.kubernetes.client.utils.HttpClientUtils.createHttpClient(HttpClientUtils.java:167)
at org.apache.spark.deploy.k8s.SparkKubernetesClientFactory$.createKubernetesClient(SparkKubernetesClientFactory.scala:84)
at org.apache.spark.scheduler.cluster.k8s.KubernetesClusterManager.createSchedulerBackend(KubernetesClusterManager.scala:64)
at org.apache.spark.SparkContext$.org$apache$spark$SparkContext$$createTaskScheduler(SparkContext.scala:2788)
... 20 more
Caused by: java.security.cert.CertificateException: Could not parse certificate: java.io.IOException: Empty input
at sun.security.provider.X509Factory.engineGenerateCertificate(X509Factory.java:110)
at java.security.cert.CertificateFactory.generateCertificate(CertificateFactory.java:339)
at io.fabric8.kubernetes.client.internal.CertUtils.createTrustStore(CertUtils.java:93)
at io.fabric8.kubernetes.client.internal.CertUtils.createTrustStore(CertUtils.java:71)
at io.fabric8.kubernetes.client.internal.SSLUtils.trustManagers(SSLUtils.java:114)
at io.fabric8.kubernetes.client.internal.SSLUtils.trustManagers(SSLUtils.java:93)
at io.fabric8.kubernetes.client.utils.HttpClientUtils.createHttpClient(HttpClientUtils.java:63)
... 23 more
Caused by: java.io.IOException: Empty input
at sun.security.provider.X509Factory.engineGenerateCertificate(X509Factory.java:106)
... 29 more
2018-12-12 06:26:23 INFO ShutdownHookManager:54 - Shutdown hook called
2018-12-12 06:26:23 INFO ShutdownHookManager:54 - Deleting directory /var/data/spark-c306f3bc-cc7c-4090-a7f7-22fbc02b72bd/spark-157bde7a-d0e1-4092-9e39-5ab2b727e3e7
2018-12-12 06:26:23 INFO ShutdownHookManager:54 - Deleting directory /tmp/spark-07910ccb-c212-4622-9c1b-3e8b772f3faa
It looks to me like there is a communication / authentication issue during pod creation, and it looks like there might be a certificate missing that Spark needs to communicate with Kubernetes, which I infer from the "java.security.cert.CertificateException: Could not parse certificate: java.io.IOException: Empty input" error message.
Any ideas on what I can do to resolve this? Thanks!