[jira] [Created] (SPARK-37572) Flexible ways of launching executors

["Dagang Wei (Jira)" <ji...@apache.org>]

Dagang Wei created SPARK-37572:
----------------------------------

             Summary: Flexible ways of launching executors
                 Key: SPARK-37572
                 URL: https://issues.apache.org/jira/browse/SPARK-37572
             Project: Spark
          Issue Type: New Feature
          Components: Deploy
    Affects Versions: 3.2.0
            Reporter: Dagang Wei


Currently Spark launches executor processes by constructing and running a command [1], for example:

 

```

/usr/lib/jvm/adoptopenjdk-8-hotspot-amd64/jre/bin/java -cp /opt/spark-3.2.0-bin-hadoop3.2/conf/:/opt/spark-3.2.0-bin-hadoop3.2/jars/* -Xmx1024M -Dspark.driver.port=35729 org.apache.spark.executor.CoarseGrainedExecutorBackend --driver-url spark://CoarseGrainedScheduler@dagang.svl.corp.google.com:35729 --executor-id 0 --hostname 100.116.124.193 --cores 6 --app-id app-20211207131146-0002 --worker-url spark://Worker@100.116.124.193:45287

```

But there are use cases which require more flexible ways of launching executors. In particular, our use case is that we run Spark in standalone mode, while Spark workers are running in VMs, they need to launch Spark executors in Docker containers. The reason is that we want to allow Spark app developers to provide custom container images to customize the job runtime environment (typically Java and Python dependencies).

 

After investigating in the source code, we found that the concept of Spark Command Runner might be a good solution. Basically, we want to introduce an optional Spark command runner in Spark, so that instead of running the command to launch executor directly, it passes the command to the runner, which the runner then runs the command with its own strategy which could be running in Docker, or by default running the command directly.

 

The runner should be customizable through an env variable `SPARK_COMMAND_RUNNER`, which by default could be a simple script like:

 

```

#!/bin/bash

exec "$@"

```

 

or in the case of Docker container:

 

```

#!/bin/bash

docker run ... -- "$@"

```

[1]: https://github.com/apache/spark/blob/v3.2.0/core/src/main/scala/org/apache/spark/deploy/worker/CommandUtils.scala#L52



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org