[jira] [Created] (SPARK-28525) Allow Launcher to be applied Java options

["Kousuke Saruta (JIRA)" <ji...@apache.org>]

Kousuke Saruta created SPARK-28525:
--------------------------------------

             Summary: Allow Launcher to be applied Java options
                 Key: SPARK-28525
                 URL: https://issues.apache.org/jira/browse/SPARK-28525
             Project: Spark
          Issue Type: Improvement
          Components: Deploy
    Affects Versions: 3.0.0
            Reporter: Kousuke Saruta
            Assignee: Kousuke Saruta


Launcher is implemented as a Java application and sometimes I'd like to apply Java options.
One situation I have met is the time I try to attach debugger to Launcher.

Launcher is launched from bin/spark-class but there is no room to apply Java options.

{code}
build_command() {
  "$RUNNER" -Xmx128m -cp "$LAUNCH_CLASSPATH" org.apache.spark.launcher.Main "$@"
  printf "%d\0" $?
}
{code}

Considering that it's not so many times to apply Java options to Launcher,  one compromise would just modify spark-class by user like as follows.

{code}
build_command() {
  "$RUNNER" -Xmx128m $SPARK_LAUNCHER_OPTS -cp "$LAUNCH_CLASSPATH" org.apache.spark.launcher.Main "$@"
  printf "%d\0" $?
}
{code}

But it doesn't work when any text related to Java options is output to standard output because whole output is used as command-string for spark-shell and spark-submit in current implementation.

One example is jdwp. When apply agentlib option to use jdwp for debug, we will get output like as follows.

{code}
Listening for transport dt_socket at address: 9876
{code}

The output shown above is not a command-string so spark-submit and spark-shell will fail.

To enable Java options for Launcher, we need treat command-string and others.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org