You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Benson Muite (Jira)" <ji...@apache.org> on 2021/05/12 08:15:00 UTC
[jira] [Commented] (SPARK-35317) Job submission in high performance
computing environments
[ https://issues.apache.org/jira/browse/SPARK-35317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17343104#comment-17343104 ]
Benson Muite commented on SPARK-35317:
--------------------------------------
The plugin [https://github.com/apache/spark/pull/32136] does not address the issue. The challenge is that resource schedulers in a High Performance Computing (HPC) environment such as [Slurm|https://slurm.schedmd.com/overview.html] and [OpenPBS|[https://openpbs.org/]] do not typically expose IP addresses to users. When SPARK is used in such an environment, the SPARK scheduler is typically deployed on top of resources provisioned by the HPC scheduler. The aim is to avoid an extra configuration step for someone who may not have an IT background that wants to use SPARK in an HPC environment so that SPARK can be deployed more easily.
> Job submission in high performance computing environments
> ---------------------------------------------------------
>
> Key: SPARK-35317
> URL: https://issues.apache.org/jira/browse/SPARK-35317
> Project: Spark
> Issue Type: Improvement
> Components: Spark Submit
> Affects Versions: 3.1.2
> Reporter: Benson Muite
> Priority: Minor
>
> Spark is often used in high performance computing environments. The default launcher does not directly support schedulers used such as [slurm|https://slurm.schedmd.com/overview.html] and [pbs|https://openpbs.org/]. It would be good to support these directly. The repositories [https://github.com/ekasitk/spark-on-hpc] and [https://github.com/rokroskar/sparkhpc] contain most of the material necessary for this, but it may be good to incorporate this in Spark directly so that Java and Scala are also well supported.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org