You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "jzhuge (via GitHub)" <gi...@apache.org> on 2025/11/12 08:16:12 UTC
[PR] [SPARK-54313][CORE] Add --extra-properties-file option for configuration layering [spark]
jzhuge opened a new pull request, #53012:
URL: https://github.com/apache/spark/pull/53012
### What changes were proposed in this pull request?
This PR adds support for multiple properties files via the `--extra-properties-file` option in spark-submit, enabling configuration layering and composition.
**Changes:**
- Added `extraPropertiesFiles: Seq[String]` field to `SparkSubmitArguments`
- Added `EXTRA_PROPERTIES_FILE` constant to `SparkSubmitOptionParser`
- Modified `AbstractCommandBuilder` to load and merge extra properties files
- Updated `SparkSubmitCommandBuilder` to parse and rebuild args with extra files
- Added 4 unit tests in `SparkSubmitSuite`
### Why are the changes needed?
Large-scale Spark deployments often need to compose configurations from multiple sources:
- Base/shared defaults - Common settings reused across applications
- Environment-specific configs - Different settings for dev/staging/prod
- Cluster-specific settings - YARN queues, resource limits per cluster
- Application-specific overrides - Per-app tuning parameters
### Does this PR introduce any user-facing changes?
Yes. Adds new `--extra-properties-file` option to spark-submit.
**Usage:**
```bash
spark-submit \
--properties-file base.conf \
--extra-properties-file cluster.conf \
--extra-properties-file app.conf \
--class com.example.App app.jar
```
**Property Precedence** (highest to lowest):
1. `--conf` flags (command-line overrides)
2. `--extra-properties-file` files (later files override earlier ones)
3. `--properties-file` (base file)
4. `conf/spark-defaults.conf` (with `--load-spark-defaults`)
**Key Features:**
- Repeatable flag - can be specified multiple times
- Later files override earlier files (left-to-right precedence)
- No breaking changes - new flag, existing behavior unchanged
- Shell-friendly - no comma escaping or string manipulation needed
### How was this patch tested?
Added 4 unit tests in `SparkSubmitSuite`.
### Was this patch authored or co-authored using generative AI tooling?
Unit tests and descriptions were generated by Claude Code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54313][CORE] Add --extra-properties-file option for configuration layering [spark]
Posted by "sarutak (via GitHub)" <gi...@apache.org>.
sarutak closed pull request #53012: [SPARK-54313][CORE] Add --extra-properties-file option for configuration layering
URL: https://github.com/apache/spark/pull/53012
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54313][CORE] Add --extra-properties-file option for configuration layering [spark]
Posted by "sarutak (via GitHub)" <gi...@apache.org>.
sarutak commented on PR #53012:
URL: https://github.com/apache/spark/pull/53012#issuecomment-3713184575
Merged to `master`.
Thank you @jzhuge and @holdenk !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54313][CORE] Add --extra-properties-file option for configuration layering [spark]
Posted by "jzhuge (via GitHub)" <gi...@apache.org>.
jzhuge commented on PR #53012:
URL: https://github.com/apache/spark/pull/53012#issuecomment-3524566955
All checks passed. Ready for review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54313][CORE] Add --extra-properties-file option for configuration layering [spark]
Posted by "jzhuge (via GitHub)" <gi...@apache.org>.
jzhuge commented on PR #53012:
URL: https://github.com/apache/spark/pull/53012#issuecomment-3713485830
Thank you @sarutak and @holdenk !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54313][CORE] Add --extra-properties-file option for configuration layering [spark]
Posted by "sarutak (via GitHub)" <gi...@apache.org>.
sarutak commented on PR #53012:
URL: https://github.com/apache/spark/pull/53012#issuecomment-3709039437
Hi @jzhuge , this change looks good to me too. Could you rebase this change to `master`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org