You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Liu Runzhong (Jira)" <ji...@apache.org> on 2019/11/06 06:33:00 UTC
[jira] [Updated] (SPARK-29770) Allow setting spark.app.id when
spark-submit for Spark on Kubernetes
[ https://issues.apache.org/jira/browse/SPARK-29770?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Liu Runzhong updated SPARK-29770:
---------------------------------
Description:
when the user provides `spark.app.id` by `spark-submit`, it's actually doing nothing to change the `spark.app.id`, as `spark.app.id` can only be set by `kubernetesAppId` every time, which makes the users feel confused.
[https://github.com/apache/spark/blob/master/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/submit/KubernetesClientApplication.scala#L196]
Knowing that `spark.app.id` would be labeled to Driver/Executor pods and other resources and the strict limitation of the label values, but I think it would be more flexible to users to decide how to generate the `spark.app.id` by themselves.
was:
when the user provides `spark.app.id` by `spark-submit`, it's actually doing nothing to change the `spark.app.id`, as `spark.app.id` can only be set by `kubernetesAppId` every time, which makes the users feel confused.
[https://github.com/apache/spark/blob/master/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/submit/KubernetesClientApplication.scala#L196]
Knowing that `spark.app.id` would be labeled to Driver/Executor pods and other resources and the strict limitation of the label values, but I think it would be more flexible to users to decide how to generate the `spark.app.id` by themselves.
> Allow setting spark.app.id when spark-submit for Spark on Kubernetes
> --------------------------------------------------------------------
>
> Key: SPARK-29770
> URL: https://issues.apache.org/jira/browse/SPARK-29770
> Project: Spark
> Issue Type: Improvement
> Components: Kubernetes
> Affects Versions: 3.0.0
> Reporter: Liu Runzhong
> Priority: Minor
> Labels: easyfix
>
> when the user provides `spark.app.id` by `spark-submit`, it's actually doing nothing to change the `spark.app.id`, as `spark.app.id` can only be set by `kubernetesAppId` every time, which makes the users feel confused.
> [https://github.com/apache/spark/blob/master/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/submit/KubernetesClientApplication.scala#L196]
> Knowing that `spark.app.id` would be labeled to Driver/Executor pods and other resources and the strict limitation of the label values, but I think it would be more flexible to users to decide how to generate the `spark.app.id` by themselves.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org