[jira] [Created] (SPARK-39358) Have the arguments passed via spark.executor.extraJavaOptions placed before the classpath when composing the executors commandline

["Alberto Bortolan (Jira)" <ji...@apache.org>]

Alberto Bortolan created SPARK-39358:
----------------------------------------

             Summary: Have the arguments passed via spark.executor.extraJavaOptions placed before the classpath when composing the executors commandline
                 Key: SPARK-39358
                 URL: https://issues.apache.org/jira/browse/SPARK-39358
             Project: Spark
          Issue Type: Wish
          Components: Spark Core
    Affects Versions: 3.2.1, 2.4.8
            Reporter: Alberto Bortolan


When submitting a job it's possible to pass java options to be applied to the executor processes via the configuration parameter {{spark.executor.extraJavaOptions}} as in 
{noformat}
--conf 'spark.driver.extraJavaOptions=-Dmycompany.app.name=some_name'{noformat}
The command line for the executors is composed as
{noformat}
java <$SPARK_CONF_DIF/java.opts options>  -cp <classpath> -Xmx <extra_java_options> ...{noformat}
Since the classpath can be particularly long, it would be helpful for the purpose of visibility  when systems are monitored with tools that show the first part of the command line of a process (e.g nmon), to get the extra java options moved before classpath, as in
{noformat}
java <$SPARK_CONF_DIR/java.opts options> <extra_java_options> -cp <classpath> -Xmx  ...{noformat}
The type of extra options that would be passed in this way would be application and submission dependent and so it would not be appropriate or even possible to place them inside {{{}$SPARK_CONF_DIR/java-opts{}}}.

---

For reference, the code that builds the command line is [buildSparkSubmitCommand|https://github.com/apache/spark/blob/4f25b3f71238a00508a356591553f2dfa89f8290/launcher/src/main/java/org/apache/spark/launcher/SparkSubmitCommandBuilder.java#L263], where, one of the first actions is calling  [buildJavaCommand|https://github.com/apache/spark/blob/4f25b3f71238a00508a356591553f2dfa89f8290/launcher/src/main/java/org/apache/spark/launcher/AbstractCommandBuilder.java#L92] that prepare the first part of the string from the "java" command to "-cp classpath" included.



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org