You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "ASF GitHub Bot (Jira)" <ji...@apache.org> on 2025/05/08 13:57:00 UTC
[jira] [Updated] (SPARK-52041) Add better support for integrating with external cluster manager #50770
[ https://issues.apache.org/jira/browse/SPARK-52041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
ASF GitHub Bot updated SPARK-52041:
-----------------------------------
Labels: pull-request-available (was: )
> Add better support for integrating with external cluster manager #50770
> -----------------------------------------------------------------------
>
> Key: SPARK-52041
> URL: https://issues.apache.org/jira/browse/SPARK-52041
> Project: Spark
> Issue Type: Improvement
> Components: Spark Submit
> Affects Versions: 4.1.0
> Reporter: Dejan Zele Pejchev
> Priority: Minor
> Labels: pull-request-available
>
> h3. What changes were proposed?
> * Remove previous checks in {{SparkSubmit}} that restricted use of external managers to specific master URLs.
> * Add new configuration field {{{}spark.submit.external.class{}}}.
> h3. Why are the changes needed?
> SparkSubmit currently only supports built-in cluster managers (YARN, Kubernetes, Standalone), making it difficult to integrate custom or third-party schedulers without forking the codebase.
> By removing hard-coded restrictions, this patch enables easier integration with various cluster managers.
> Now the following is possible (assuming the implementation is loaded):
> bin/spark-submit \
> --master armada://localhost:30002 \
> --deploy-mode cluster
> Does this change introduce _any_ user-facing change?
> Yes, users can easily integrate with their custom cluster managers.
> h3. How was this patch tested?
> Added unit tests and tested with [armadaproject/armada-spark|https://github.com/armadaproject/armada-spark] custom integration and with manual testing while developing the Armada integration.
> h3. Was this patch authored or co-authored using generative AI tooling?
> No.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org