You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "John Zhuge (Jira)" <ji...@apache.org> on 2025/11/12 02:50:00 UTC

[jira] [Created] (SPARK-54313) Add --extra-properties-file option for configuration layering in spark-submit

John Zhuge created SPARK-54313:
----------------------------------

             Summary: Add --extra-properties-file option for configuration layering in spark-submit
                 Key: SPARK-54313
                 URL: https://issues.apache.org/jira/browse/SPARK-54313
             Project: Spark
          Issue Type: New Feature
          Components: Spark Core
    Affects Versions: 4.0.1
            Reporter: John Zhuge


This proposal adds support for specifying multiple properties files in spark-submit through a repeatable {{--extra-properties-file}} option. This enables configuration layering and composition, similar to SPARK-34345 but a bit more cleaner.

Large-scale Spark deployments often need to compose configurations from multiple sources, for example:

  * Base/shared defaults - Common settings across all applications
  * Environment-specific configs - Dev/staging/prod differences
  * Cluster-specific settings - Different clusters, regions, or Kubernetes namespaces
  * Application-specific overrides - Per-job tuning

Currently, we must either merge properties files before submission or rely solely on {{--conf}} flags (command line can get very long).

Proposed command-line syntax:

{code:bash}
  spark-submit \
    --properties-file /path/to/base.conf \
    --load-spark-defaults \
    --extra-properties-file /path/to/override1.conf \
    --extra-properties-file /path/to/override2.conf \
    --conf spark.executor.memory=4g \
    --class com.example.MyApp \
    myapp.jar
{code}

Property precedence (highest to lowest):

  # {{--conf}} flags (command-line overrides)
  # {{--extra-properties-file}} files (in order specified, later overrides earlier)
  # {{--properties-file}} (base file)
  # {{conf/spark-defaults.conf}} (when {{\-\-properties-file}} is not specified or if {{--load-spark-defaults}} is used)

Key behaviors:
  - {{--extra-properties-file}} can be specified multiple times
  - Files are loaded left-to-right; later values override earlier ones
  - Missing files cause validation errors
  - Works with all deployment modes (client, cluster, local)
  - Compatible with {{--load-spark-defaults}} flag

 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org