You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Yanzhe Xu (Jira)" <ji...@apache.org> on 2022/12/12 09:20:00 UTC
[jira] [Updated] (SPARK-41496) Spark doesn't support multiple documents in one pod template file
[ https://issues.apache.org/jira/browse/SPARK-41496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Yanzhe Xu updated SPARK-41496:
------------------------------
Description:
When defining multiple documents in the pod template file:
```yaml
apiVersion: v1
kind: Pod
...# the first document to create a PVC
...
--- # use tripple-dash to split multiple documents in one yaml file
apiVersion: v1
kind: Pod
...
... # the second document to create the Spark pod.
```
Then set the podTemplate file in spark-submit command:
```
--conf spark.kubernetes.executor.podTemplateFile=s3a://***/executor_pod_template.yaml \
```
Then Spark throws:
22/12/09 17:17:41 ERROR KubernetesUtils: Encountered exception while attempting to load initial pod spec from file
expected a single document in the stream
in 'reader', line 1, column 1:
apiVersion: v1
^
but found another document
in 'reader', line 27, column 1:
—
^
at org.yaml.snakeyaml.composer.Composer.getSingleNode(Composer.java:125)
at org.yaml.snakeyaml.constructor.BaseConstructor.getSingleData(BaseConstructor.java:150)
at org.yaml.snakeyaml.Yaml.loadFromReader(Yaml.java:490)
at org.yaml.snakeyaml.Yaml.load(Yaml.java:429)
at io.fabric8.kubernetes.client.utils.Serialization.unmarshalYaml(Serialization.java:304)
at io.fabric8.kubernetes.client.utils.Serialization.unmarshal(Serialization.java:245)
at io.fabric8.kubernetes.client.utils.Serialization.unmarshal(Serialization.java:201)
at io.fabric8.kubernetes.client.utils.Serialization.unmarshal(Serialization.java:189)
at io.fabric8.kubernetes.client.dsl.base.OperationSupport.unmarshal(OperationSupport.java:656)
at io.fabric8.kubernetes.client.dsl.base.BaseOperation.load(BaseOperation.java:315)
at io.fabric8.kubernetes.client.dsl.base.BaseOperation.load(BaseOperation.java:330)
at io.fabric8.kubernetes.client.dsl.base.BaseOperation.load(BaseOperation.java:86)
at org.apache.spark.deploy.k8s.KubernetesUtils$.loadPodFromTemplate(KubernetesUtils.scala:105)
The entrypoint code: [https://github.com/apache/spark/blame/master/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/KubernetesUtils.scala#L105]
was:
When defining multiple documents in the pod template file:
```yaml
apiVersion: v1
kind: Pod
...# the first document to create a PVC
...
---
apiVersion: v1
kind: Pod
...
... # the second document to create the Spark pod.
```
Then set the podTemplate file in spark-submit command:
```
--conf spark.kubernetes.executor.podTemplateFile=s3a://***/executor_pod_template.yaml \
```
Then Spark throws:
22/12/09 17:17:41 ERROR KubernetesUtils: Encountered exception while attempting to load initial pod spec from file
expected a single document in the stream
in 'reader', line 1, column 1:
apiVersion: v1
^
but found another document
in 'reader', line 27, column 1:
---
^
at org.yaml.snakeyaml.composer.Composer.getSingleNode(Composer.java:125)
at org.yaml.snakeyaml.constructor.BaseConstructor.getSingleData(BaseConstructor.java:150)
at org.yaml.snakeyaml.Yaml.loadFromReader(Yaml.java:490)
at org.yaml.snakeyaml.Yaml.load(Yaml.java:429)
at io.fabric8.kubernetes.client.utils.Serialization.unmarshalYaml(Serialization.java:304)
at io.fabric8.kubernetes.client.utils.Serialization.unmarshal(Serialization.java:245)
at io.fabric8.kubernetes.client.utils.Serialization.unmarshal(Serialization.java:201)
at io.fabric8.kubernetes.client.utils.Serialization.unmarshal(Serialization.java:189)
at io.fabric8.kubernetes.client.dsl.base.OperationSupport.unmarshal(OperationSupport.java:656)
at io.fabric8.kubernetes.client.dsl.base.BaseOperation.load(BaseOperation.java:315)
at io.fabric8.kubernetes.client.dsl.base.BaseOperation.load(BaseOperation.java:330)
at io.fabric8.kubernetes.client.dsl.base.BaseOperation.load(BaseOperation.java:86)
at org.apache.spark.deploy.k8s.KubernetesUtils$.loadPodFromTemplate(KubernetesUtils.scala:105)
The entrypoint code: [https://github.com/apache/spark/blame/master/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/KubernetesUtils.scala#L105]
> Spark doesn't support multiple documents in one pod template file
> -----------------------------------------------------------------
>
> Key: SPARK-41496
> URL: https://issues.apache.org/jira/browse/SPARK-41496
> Project: Spark
> Issue Type: Bug
> Components: Kubernetes
> Affects Versions: 3.2.2
> Reporter: Yanzhe Xu
> Priority: Major
>
> When defining multiple documents in the pod template file:
> ```yaml
> apiVersion: v1
> kind: Pod
> ...# the first document to create a PVC
> ...
> --- # use tripple-dash to split multiple documents in one yaml file
> apiVersion: v1
> kind: Pod
> ...
> ... # the second document to create the Spark pod.
> ```
>
> Then set the podTemplate file in spark-submit command:
> ```
> --conf spark.kubernetes.executor.podTemplateFile=s3a://***/executor_pod_template.yaml \
> ```
>
> Then Spark throws:
> 22/12/09 17:17:41 ERROR KubernetesUtils: Encountered exception while attempting to load initial pod spec from file
> expected a single document in the stream
> in 'reader', line 1, column 1:
> apiVersion: v1
> ^
> but found another document
> in 'reader', line 27, column 1:
> —
> ^
> at org.yaml.snakeyaml.composer.Composer.getSingleNode(Composer.java:125)
> at org.yaml.snakeyaml.constructor.BaseConstructor.getSingleData(BaseConstructor.java:150)
> at org.yaml.snakeyaml.Yaml.loadFromReader(Yaml.java:490)
> at org.yaml.snakeyaml.Yaml.load(Yaml.java:429)
> at io.fabric8.kubernetes.client.utils.Serialization.unmarshalYaml(Serialization.java:304)
> at io.fabric8.kubernetes.client.utils.Serialization.unmarshal(Serialization.java:245)
> at io.fabric8.kubernetes.client.utils.Serialization.unmarshal(Serialization.java:201)
> at io.fabric8.kubernetes.client.utils.Serialization.unmarshal(Serialization.java:189)
> at io.fabric8.kubernetes.client.dsl.base.OperationSupport.unmarshal(OperationSupport.java:656)
> at io.fabric8.kubernetes.client.dsl.base.BaseOperation.load(BaseOperation.java:315)
> at io.fabric8.kubernetes.client.dsl.base.BaseOperation.load(BaseOperation.java:330)
> at io.fabric8.kubernetes.client.dsl.base.BaseOperation.load(BaseOperation.java:86)
> at org.apache.spark.deploy.k8s.KubernetesUtils$.loadPodFromTemplate(KubernetesUtils.scala:105)
>
> The entrypoint code: [https://github.com/apache/spark/blame/master/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/KubernetesUtils.scala#L105]
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org