You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Holden Karau (Jira)" <ji...@apache.org> on 2025/07/24 18:58:00 UTC
[jira] [Commented] (SPARK-46912) Spark-submit in cluster mode with standalone cluster uses wrong JAVA_HOME path
[ https://issues.apache.org/jira/browse/SPARK-46912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18009677#comment-18009677 ]
Holden Karau commented on SPARK-46912:
--------------------------------------
Doing a bit of thinking: I'd generally expect all of the executors to have the same JAVA_HOME but I could see an argument for the case where the driver program is a bit different because it's say running in a notebook.
> Spark-submit in cluster mode with standalone cluster uses wrong JAVA_HOME path
> ------------------------------------------------------------------------------
>
> Key: SPARK-46912
> URL: https://issues.apache.org/jira/browse/SPARK-46912
> Project: Spark
> Issue Type: Bug
> Components: Spark Core, Spark Submit
> Affects Versions: 3.5.0
> Reporter: Danh Pham
> Priority: Major
> Labels: pull-request-available
>
> When run spark submit to a standalone cluster using cluster mode, the worker machine will use the JAVA_HOME value from remote machine instead of from worker machine.
> To reproduce:
> * Create a standalone cluster using docker compose, set JAVA_HOME in each worker different from local machine.
> * Run spark-submit, deploy-mode cluster
> * Monitor the log from worker, the driver will print out: DriverRunner: Launch Command: "<value from local JAVA_HOME>" "-cp" ...
> Reason:
> When Master create a new driver in receiveAndReply method, it uses the environment variables from submitter to build the driver description command. After that, when launch the driver, a new local (of worker) is built but it still use environment variable from driver description (which came from submitter). The result is the building java command will use the submitter java home path instead of worker path.
> Suggestion:
> Replace JAVA_HOME and SPARK_HOME in buildLocalCommand method ofÂ
> org.apache.spark.deploy.worker.CommandUtils by worker value
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org