You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Jim Lim (JIRA)" <ji...@apache.org> on 2014/11/18 17:43:34 UTC
[jira] [Updated] (SPARK-4474) Improve handling of jars that cannot
be included in the uber jar
[ https://issues.apache.org/jira/browse/SPARK-4474?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jim Lim updated SPARK-4474:
---------------------------
Description:
Please refer to this [pull request|https://github.com/apache/spark/pull/3238] for more details.
Some jars, such as the datanucleus jars, are distributed with spark but cannot be included in the uber (aka assembly) jar. This caused some classpath issues, such as [SPARK-2624]. A workaround was done in the above pull request, allowing the user to specify the location of the datanucleus jars in {{spark.yarn.datanucleus.dir}}.
Some things to consider:
- rename {{spark.yarn.datanucleus.dir}} to be more generic to deal with such jars in the future
- figure out how to include datanucleus jars in the uber jar
was:
Please refer to this [pull request|https://github.com/apache/spark/pull/3238] for more details.
Some jars, such as the datanucleus jars, are distributed with spark but cannot be included in the uber (aka assembly) jar. This caused some classpath issues, such as [SPARK-2624]. A workaround was done in the above pull request, allowing the user to specify the location of the datanucleus jars in {{spark.yarn.datanucleus.dir}}.
Some things to consider:
- renaming {{spark.yarn.datanucleus.dir}} to be more generic to deal with such jars in the future
- figure out how to include datanucleus jars in the uber jar
> Improve handling of jars that cannot be included in the uber jar
> ----------------------------------------------------------------
>
> Key: SPARK-4474
> URL: https://issues.apache.org/jira/browse/SPARK-4474
> Project: Spark
> Issue Type: Improvement
> Components: YARN
> Affects Versions: 1.2.0
> Reporter: Jim Lim
> Priority: Minor
>
> Please refer to this [pull request|https://github.com/apache/spark/pull/3238] for more details.
> Some jars, such as the datanucleus jars, are distributed with spark but cannot be included in the uber (aka assembly) jar. This caused some classpath issues, such as [SPARK-2624]. A workaround was done in the above pull request, allowing the user to specify the location of the datanucleus jars in {{spark.yarn.datanucleus.dir}}.
> Some things to consider:
> - rename {{spark.yarn.datanucleus.dir}} to be more generic to deal with such jars in the future
> - figure out how to include datanucleus jars in the uber jar
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org