[jira] [Created] (HIVE-16484) Investigate SparkLauncher for HoS as alternative to bin/spark-submit

["Sahil Takiar (JIRA)" <ji...@apache.org>]

Sahil Takiar created HIVE-16484:
-----------------------------------

             Summary: Investigate SparkLauncher for HoS as alternative to bin/spark-submit
                 Key: HIVE-16484
                 URL: https://issues.apache.org/jira/browse/HIVE-16484
             Project: Hive
          Issue Type: Bug
          Components: Spark
            Reporter: Sahil Takiar
            Assignee: Sahil Takiar


The {{SparkClientImpl#startDriver}} currently looks for the {{SPARK_HOME}} directory and invokes the {{bin/spark-submit}} script, which spawns a separate process to run the Spark application.

{{SparkLauncher}} was added in SPARK-4924 and is a programatic way to launch Spark applications.

I see a few advantages:

* No need to spawn a separate process to launch a HoS --> lower startup time
* Simplifies the code in {{SparkClientImpl}} --> easier to debug
* {{SparkLauncher#startApplication}} returns a {{SparkAppHandle}} which contains some useful utilities for querying the state of the Spark job
** It also allows the launcher to specify a list of job listeners



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)