You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Dongjoon Hyun (Jira)" <ji...@apache.org> on 2026/03/31 17:48:00 UTC
[jira] [Updated] (SPARK-56274) Simplify `SparkClusterSubmissionWorker.getResourceSpec`
[ https://issues.apache.org/jira/browse/SPARK-56274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Dongjoon Hyun updated SPARK-56274:
----------------------------------
Summary: Simplify `SparkClusterSubmissionWorker.getResourceSpec` (was: Fix redundant SparkConf set() call in SparkClusterSubmissionWorker)
> Simplify `SparkClusterSubmissionWorker.getResourceSpec`
> -------------------------------------------------------
>
> Key: SPARK-56274
> URL: https://issues.apache.org/jira/browse/SPARK-56274
> Project: Spark
> Issue Type: Improvement
> Components: Kubernetes
> Affects Versions: kubernetes-operator-0.9.0
> Reporter: Jiwon Park
> Assignee: Jiwon Park
> Priority: Minor
> Labels: pull-request-available
>
> In {{SparkClusterSubmissionWorker.getResourceSpec()}}, each entry in {{sparkConf}} is written to {{effectiveSparkConf}} twice per iteration:
> {code:java}
> for (Map.Entry<String, String> entry : confFromSpec.entrySet()) {
> effectiveSparkConf.set(entry.getKey(), entry.getValue()); // first set
> String value = entry.getValue();
> if ("spark.kubernetes.container.image".equals(entry.getKey())) {
> value = value.replace("{{SPARK_VERSION}}", sparkVersion);
> }
> effectiveSparkConf.set(entry.getKey(), value); // second set
> }
> {code}
> For the {{spark.kubernetes.container.image}} key, the first {{set()}} stores the raw (unsubstituted) value before the {{{{SPARK_VERSION}}}} placeholder is replaced, and the second {{set()}} immediately overwrites it with the correct value. For all other keys, both calls write the identical value. The first {{set()}} is therefore always redundant.
> *Fix:* Remove the first {{set()}} call and restructure the loop to call {{set()}} exactly once per entry with the correctly substituted value.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org