You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@reef.apache.org by "Markus Weimer (JIRA)" <ji...@apache.org> on 2017/04/26 18:01:04 UTC
[jira] [Commented] (REEF-1787) Unmanaged REEF on Spark packages
global.jar resources incorrectly
[ https://issues.apache.org/jira/browse/REEF-1787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15985273#comment-15985273 ]
Markus Weimer commented on REEF-1787:
-------------------------------------
Interesting. This looks as if we erroneously follow the symlinks and use the original path in the JAR we create on the driver. There should be an option to use the path to the symlink instead in the ZIP file library we use.
> Unmanaged REEF on Spark packages global.jar resources incorrectly
> -----------------------------------------------------------------
>
> Key: REEF-1787
> URL: https://issues.apache.org/jira/browse/REEF-1787
> Project: REEF
> Issue Type: Bug
> Components: REEF Driver, REEF Runtime YARN
> Environment: YARN 2.7.3+
> Reporter: Sergiy Matusevych
> Assignee: Sergiy Matusevych
> Labels: jar, resource, spark, yarn
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> When REEF application is being launched in Unmanaged AM mode from Spark driver on YARN, it packages the global resources incorrectly. As a result, REEF Evaluators cannot find {{REEFLauncher}} class and fail.
> All global resources required by REEF application are being packaged into the {{global.jar}} file on HDFS; however, when packaged from Spark, the resources include full path to the files, e.g.
> {code}
> $ jar tvf global.jar
> 15082 Wed Apr 19 19:03:50 PDT 2017 mnt/data/0/local/nm/usercache/hadoop/filecache/18/reef-driver-on-spark_2.11-1.01.jar
> 16963704 Wed Apr 19 19:03:50 PDT 2017 mnt/data/0/local/nm/usercache/hadoop/filecache/15/reef-examples-0.16.0-SNAPSHOT-shaded.jar
> {code}
> whereas the same {{global.jar}} produced by REEF client does _not_ include such paths.
> We have to investigate whether absence or presence of such paths in the {{global.jar}} causes the Evaluator start failure.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)