[k8s] Fail to expose custom port on executor container specified in my executor pod template

[James Yu <ja...@ispot.tv>]

Hi Team,


I have no luck in trying to expose port 5005 (for remote debugging purpose) on my executor container using the following pod template and spark configuration

s3a://mybucket/pod-template-executor-debug.yaml
--------------------------------------------------------------------
apiVersion: v1
kind: Pod
spec:
  containers:
  - name: spark-kubernetes-executor
    ports:
    - containerPort: 5005
      name: debug
      protocol: TCP
--------------------------------------------------------------------
--config spark.kubernetes.executor.podTemplateFile=s3a://mybucket/pod-template-executor-debug.yaml
--------------------------------------------------------------------

The resultant executor container only exposes the default 7079/TCP port, but not the 5005/TCP that I wanted it to expose.

It works just fine for the driver container with the similar settings where I can see all ports are exposed (5005/TCP, 7078/TCP, 7079/TCP, 4040/TCP) as expected.

Did I miss anything, or is this a known bug where executor pod template is not respected in terms of the port expose?

Thanks in advance for your help.

James