You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@reef.apache.org by Reed Umbrasas <ry...@microsoft.com.INVALID> on 2018/01/18 19:59:19 UTC
How does REEFLauncher get copied over to the cluster?
Hi,
Can someone please point me to where REEF framework classes (i.e. REEFLauncher) get copied over to the remote cluster?
I found that the job submission handler does package up a JAR file (https://github.com/apache/reef/blob/master/lang/java/reef-runtime-hdinsight/src/main/java/org/apache/reef/runtime/hdinsight/client/HDInsightJobSubmissionHandler.java#L99) and copy it over to the cluster; however, it only seems to contain the user classes (i.e. HelloDriver). So how do the classes provided by REEF make it into the remote cluster from the client?
Thanks,
Reed
RE: How does REEFLauncher get copied over to the cluster?
Posted by Reed Umbrasas <ry...@microsoft.com.INVALID>.
Thank you! I spent two days running the examples from IntelliJ and wondering why the zip didn’t have any of the REEF classes. It turns out things work as expected when invoking my program from the command line via the "uber" jar.
Reed
-----Original Message-----
From: Sergiy Matusevych [mailto:sergiy.matusevych@gmail.com]
Sent: Thursday, January 18, 2018 7:39 PM
To: dev@reef.apache.org
Subject: Re: How does REEFLauncher get copied over to the cluster?
Hi Reed,
You are right that REEF jars have to be part of the job configuration on the client. In our HelloREEF example, we specify only one jar - the one that contains the user application classes. We usually assume that if the jar contains HelloDriver, all other necessary files are also there (including REEF files). That works *only* if our application build process produces an uber-jar that contains all dependencies. usually, it's a file named ./target/*-shaded.jar
If you don't have an uber-jar, you have to add all required dependencies by hand to the app configuration - using the same
.set(DriverConfiguration.GLOBAL_LIBRARIES, "...")
call.
Hope that helps!
Cheers,
Sergiy.
On Thu, Jan 18, 2018 at 11:59 AM, Reed Umbrasas < ryumbra@microsoft.com.invalid> wrote:
> Hi,
>
> Can someone please point me to where REEF framework classes (i.e.
> REEFLauncher) get copied over to the remote cluster?
>
> I found that the job submission handler does package up a JAR file (
> https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithu
> b.com%2Fapache%2Freef%2Fblob%2Fmaster%2Flang%2Fjava%2F&data=02%7C01%7C
> ryumbra%40microsoft.com%7C35d6d93cf5d342b787da08d55eee36f9%7C72f988bf8
> 6f141af91ab2d7cd011db47%7C1%7C0%7C636519299576100801&sdata=vq4vlrZdcRj
> c7FwZIUOCJSjDNNOx0TzEnqZq5nrvZRI%3D&reserved=0
> reef-runtime-hdinsight/src/main/java/org/apache/reef/
> runtime/hdinsight/client/HDInsightJobSubmissionHandler.java#L99) and
> copy it over to the cluster; however, it only seems to contain the
> user classes (i.e. HelloDriver). So how do the classes provided by
> REEF make it into the remote cluster from the client?
>
> Thanks,
> Reed
>
Re: How does REEFLauncher get copied over to the cluster?
Posted by Sergiy Matusevych <se...@gmail.com>.
Hi Reed,
You are right that REEF jars have to be part of the job configuration on
the client. In our HelloREEF example, we specify only one jar - the one
that contains the user application classes. We usually assume that if the
jar contains HelloDriver, all other necessary files are also there
(including REEF files). That works *only* if our application build process
produces an uber-jar that contains all dependencies. usually, it's a file
named ./target/*-shaded.jar
If you don't have an uber-jar, you have to add all required dependencies by
hand to the app configuration - using the same
.set(DriverConfiguration.GLOBAL_LIBRARIES, "...")
call.
Hope that helps!
Cheers,
Sergiy.
On Thu, Jan 18, 2018 at 11:59 AM, Reed Umbrasas <
ryumbra@microsoft.com.invalid> wrote:
> Hi,
>
> Can someone please point me to where REEF framework classes (i.e.
> REEFLauncher) get copied over to the remote cluster?
>
> I found that the job submission handler does package up a JAR file (
> https://github.com/apache/reef/blob/master/lang/java/
> reef-runtime-hdinsight/src/main/java/org/apache/reef/
> runtime/hdinsight/client/HDInsightJobSubmissionHandler.java#L99) and copy
> it over to the cluster; however, it only seems to contain the user classes
> (i.e. HelloDriver). So how do the classes provided by REEF make it into the
> remote cluster from the client?
>
> Thanks,
> Reed
>