Spark on Kubernetes Authentication error

[Nick Dawes <ni...@gmail.com>]

Hi there,

I'm trying to run Spark on EKS. Created an EKS cluster, added nodes and
then trying to submit a Spark job from an EC2 instance.

Ran following commands for access. kubectl create serviceaccount spark
kubectl create clusterrolebinding spark-role --clusterrole=admin
--serviceaccount=default:spark --namespace=default

spark-submit command used:

bin/spark-submit \ --master k8s://
https://XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.us-east-1.eks.amazonaws.com
<https://xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx.us-east-1.eks.amazonaws.com/> \
--deploy-mode cluster \ --name spark-pi \ --class
org.apache.spark.examples.SparkPi \ --conf spark.executor.instances=2 \
--conf spark.app.name=spark-pi \ --conf
spark.kubernetes.authenticate.driver.serviceAccountName=spark \ --conf
spark.kubernetes.container.image=k8sspark:latest \ --conf
spark.kubernetes.authenticate.submission.caCertFile=ca.pem \
local:////usr/spark-2.4.3-bin-hadoop2.7/examples/jars/spark-examples_2.11-2.4.3.jar
100000

log4j:WARN No appenders could be found for logger
(io.fabric8.kubernetes.client.Config). log4j:WARN Please initialize the
log4j system properly. log4j:WARN See
http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info. Using
Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
19/06/06 16:03:50 WARN WatchConnectionManager: Executor didn't terminate in
time after shutdown in close(), killing it in:
io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@5b43fbf6
Exception in thread "main"
io.fabric8.kubernetes.client.KubernetesClientException: Failure executing:
POST at:
https://XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.us-east-1.eks.amazonaws.com/api/v1/namespaces/default/pods
<https://xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx.us-east-1.eks.amazonaws.com/api/v1/namespaces/default/pods>.
Message: pods is forbidden: User "system:anonymous" cannot create resource
"pods" in API group "" in the namespace "default". Received status:
Status(apiVersion=v1, code=403, details=StatusDetails(causes=[],
group=null, kind=pods, name=null, retryAfterSeconds=null, uid=null,
additionalProperties={}), kind=Status, message=pods is forbidden: User
"system:anonymous" cannot create resource "pods" in API group "" in the
namespace "default", metadata=ListMeta(_continue=null,
resourceVersion=null, selfLink=null, additionalProperties={}),
reason=Forbidden, status=Failure, additionalProperties={}). at
io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:478)
at
io.fabric8.kubernetes.client.dsl.base.OperationSupport.assertResponseCode(OperationSupport.java:417)
at
io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:381)
at
io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:344)
at
io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleCreate(OperationSupport.java:227)
at
io.fabric8.kubernetes.client.dsl.base.BaseOperation.handleCreate(BaseOperation.java:787)
at
io.fabric8.kubernetes.client.dsl.base.BaseOperation.create(BaseOperation.java:357)
at
org.apache.spark.deploy.k8s.submit.Client$$anonfun$run$2.apply(KubernetesClientApplication.scala:141)
at
org.apache.spark.deploy.k8s.submit.Client$$anonfun$run$2.apply(KubernetesClientApplication.scala:140)
at org.apache.spark.util.Utils$.tryWithResource(Utils.scala:2543) at
org.apache.spark.deploy.k8s.submit.Client.run(KubernetesClientApplication.scala:140)
at
org.apache.spark.deploy.k8s.submit.KubernetesClientApplication$$anonfun$run$5.apply(KubernetesClientApplication.scala:250)
at
org.apache.spark.deploy.k8s.submit.KubernetesClientApplication$$anonfun$run$5.apply(KubernetesClientApplication.scala:241)
at org.apache.spark.util.Utils$.tryWithResource(Utils.scala:2543) at
org.apache.spark.deploy.k8s.submit.KubernetesClientApplication.run(KubernetesClientApplication.scala:241)
at
org.apache.spark.deploy.k8s.submit.KubernetesClientApplication.start(KubernetesClientApplication.scala:204)
at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:849)
at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:167)
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:195) at
org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86) at
org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:924)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:933) at
org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) 19/06/06
16:03:50 INFO ShutdownHookManager: Shutdown hook called 19/06/06 16:03:50
INFO ShutdownHookManager: Deleting directory
/tmp/spark-0060fe01-33eb-4cb4-b96b-d5be687016bc

Tried creating different clusterrole with admin privilege. But it did not
work.

Any idea how to fix this one? Thanks.


- Nick