[jira] [Updated] (SPARK-54313) Add --extra-properties-file option for configuration layering in spark-submit

["ASF GitHub Bot (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-54313?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

ASF GitHub Bot updated SPARK-54313:
-----------------------------------
    Labels: pull-request-available  (was: )

> Add --extra-properties-file option for configuration layering in spark-submit
> -----------------------------------------------------------------------------
>
>                 Key: SPARK-54313
>                 URL: https://issues.apache.org/jira/browse/SPARK-54313
>             Project: Spark
>          Issue Type: New Feature
>          Components: Spark Core
>    Affects Versions: 4.0.1
>            Reporter: John Zhuge
>            Priority: Minor
>              Labels: pull-request-available
>
> This proposal adds support for specifying multiple properties files in spark-submit through a repeatable {{--extra-properties-file}} option. This enables configuration layering and composition, similar to SPARK-34345 but a bit more cleaner.
> Large-scale Spark deployments often need to compose configurations from multiple sources, for example:
>   * Base/shared defaults - Common settings across all applications
>   * Environment-specific configs - Dev/staging/prod differences
>   * Cluster-specific settings - Different clusters, regions, or Kubernetes namespaces
>   * Application-specific overrides - Per-job tuning
> Currently, we must either merge properties files before submission or rely solely on {{--conf}} flags (command line can get very long).
> Proposed command-line syntax:
> {code:bash}
>   spark-submit \
>     --properties-file /path/to/base.conf \
>     --load-spark-defaults \
>     --extra-properties-file /path/to/override1.conf \
>     --extra-properties-file /path/to/override2.conf \
>     --conf spark.executor.memory=4g \
>     --class com.example.MyApp \
>     myapp.jar
> {code}
> Property precedence (highest to lowest):
>   # {{--conf}} flags (command-line overrides)
>   # {{--extra-properties-file}} files (in order specified, later overrides earlier)
>   # {{--properties-file}} (base file)
>   # {{conf/spark-defaults.conf}} (when {{\-\-properties-file}} is not specified or if {{--load-spark-defaults}} is used)
> Key behaviors:
>   - {{--extra-properties-file}} can be specified multiple times
>   - Files are loaded left-to-right; later values override earlier ones
>   - Missing files cause validation errors
>   - Works with all deployment modes (client, cluster, local)
>   - Compatible with {{--load-spark-defaults}} flag
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org