You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@oozie.apache.org by "Tejwinder (JIRA)" <ji...@apache.org> on 2017/02/10 17:57:41 UTC
[jira] [Commented] (OOZIE-2606) Set spark.yarn.jars to fix Spark
2.0 with Oozie
[ https://issues.apache.org/jira/browse/OOZIE-2606?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15861602#comment-15861602 ]
Tejwinder commented on OOZIE-2606:
----------------------------------
Hi All
We have oozie 4.2 installed on our system ,
can we apply the above 4 patches directly on it and "spark.yarn.jars " flag will be supported , or there are other dependencies ?
> Set spark.yarn.jars to fix Spark 2.0 with Oozie
> -----------------------------------------------
>
> Key: OOZIE-2606
> URL: https://issues.apache.org/jira/browse/OOZIE-2606
> Project: Oozie
> Issue Type: Bug
> Components: core
> Affects Versions: 4.2.0
> Reporter: Jonathan Kelly
> Assignee: Satish Subhashrao Saley
> Labels: spark, spark2.0.0
> Fix For: 4.3.0
>
> Attachments: OOZIE-2606-2.patch, OOZIE-2606-3.patch, OOZIE-2606-4.patch, OOZIE-2606.patch
>
>
> Oozie adds all of the jars in the Oozie Spark sharelib to the DistributedCache such that all jars will be present in the current working directory of the YARN container (as well as in the container classpath). However, this is not quite enough to make Spark 2.0 work, since Spark 2.0 by default looks for the jars in assembly/target/scala-2.11/jars [1] (as if it is a locally built distribution for development) and will not find them in the current working directory.
> To fix this, we can set spark.yarn.jars to *.jar so that it finds the jars in the current working directory rather than looking in the wrong place. [2]
> [1] https://github.com/apache/spark/blob/v2.0.0-rc2/launcher/src/main/java/org/apache/spark/launcher/CommandBuilderUtils.java#L357
> [2] https://github.com/apache/spark/blob/v2.0.0-rc2/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala#L476
> Note: This property will be ignored by Spark 1.x.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)