[jira] [Created] (SPARK-4474) Improve handling of jars that cannot be included in the uber jar

["Jim Lim (JIRA)" <ji...@apache.org>]

Jim Lim created SPARK-4474:
------------------------------

             Summary: Improve handling of jars that cannot be included in the uber jar
                 Key: SPARK-4474
                 URL: https://issues.apache.org/jira/browse/SPARK-4474
             Project: Spark
          Issue Type: Improvement
          Components: YARN
    Affects Versions: 1.2.0
            Reporter: Jim Lim
            Priority: Minor


Please refer to this [pull request](https://github.com/apache/spark/pull/3238) for more details.

Some jars, such as the datanucleus jars, are distributed with spark but cannot be included in the uber (aka assembly) jar. This caused some classpath issues, such as [SPARK-2624]. A workaround was done in the above pull request, allowing the user to specify the location of the datanucleus jars in `spark.yarn.datanucleus.dir`.

Some things to consider:

- renaming `spark.yarn.datanucleus.dir` to be more generic to deal with such jars in the future
- figure out how to include datanucleus jars in the uber jar



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org