[jira] [Created] (SPARK-41496) Spark doesn't support multiple documents in one pod template file

["Yanzhe Xu (Jira)" <ji...@apache.org>]

Yanzhe Xu created SPARK-41496:
---------------------------------

             Summary: Spark doesn't support multiple documents in one pod template file
                 Key: SPARK-41496
                 URL: https://issues.apache.org/jira/browse/SPARK-41496
             Project: Spark
          Issue Type: Bug
          Components: Kubernetes
    Affects Versions: 3.2.2
            Reporter: Yanzhe Xu


When defining multiple documents in the pod template file:

```yaml
apiVersion: v1
kind: Pod
...# the first document to create a PVC
...
---
apiVersion: v1
kind: Pod
...
... # the second document to create the Spark pod.
```

 

Then set the podTemplate file in spark-submit command:

```
--conf spark.kubernetes.executor.podTemplateFile=s3a://***/executor_pod_template.yaml \
```

 

Then Spark throws:
22/12/09 17:17:41 ERROR KubernetesUtils: Encountered exception while attempting to load initial pod spec from file
expected a single document in the stream
 in 'reader', line 1, column 1:
    apiVersion: v1
    ^
but found another document
 in 'reader', line 27, column 1:
    ---
    ^

        at org.yaml.snakeyaml.composer.Composer.getSingleNode(Composer.java:125)
        at org.yaml.snakeyaml.constructor.BaseConstructor.getSingleData(BaseConstructor.java:150)
        at org.yaml.snakeyaml.Yaml.loadFromReader(Yaml.java:490)
        at org.yaml.snakeyaml.Yaml.load(Yaml.java:429)
        at io.fabric8.kubernetes.client.utils.Serialization.unmarshalYaml(Serialization.java:304)
        at io.fabric8.kubernetes.client.utils.Serialization.unmarshal(Serialization.java:245)
        at io.fabric8.kubernetes.client.utils.Serialization.unmarshal(Serialization.java:201)
        at io.fabric8.kubernetes.client.utils.Serialization.unmarshal(Serialization.java:189)
        at io.fabric8.kubernetes.client.dsl.base.OperationSupport.unmarshal(OperationSupport.java:656)
        at io.fabric8.kubernetes.client.dsl.base.BaseOperation.load(BaseOperation.java:315)
        at io.fabric8.kubernetes.client.dsl.base.BaseOperation.load(BaseOperation.java:330)
        at io.fabric8.kubernetes.client.dsl.base.BaseOperation.load(BaseOperation.java:86)
        at org.apache.spark.deploy.k8s.KubernetesUtils$.loadPodFromTemplate(KubernetesUtils.scala:105)
 

The entrypoint code: [https://github.com/apache/spark/blame/master/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/KubernetesUtils.scala#L105] 

 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org