You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@reef.apache.org by Chenxi Zhao <Ch...@microsoft.com.INVALID> on 2018/03/02 23:03:23 UTC

Questions on REEF .NET Driver execution

Hi,

I am looking into REEF .NET Driver execution of YARN and have questions on the expected driver run environment. Could someone help clarify it?

In REEF .NET, do we expect the remote cluster, e.g. YARN, to be Windows environment only? I may miss this part but how do we take care the UNIX cluster environment from REEF .NET, e.g. submitting Jobs to Linux YARN cluster?

As far as I can find -


  1.  There is only WindowsYarnJobCommandProvider.cs (https://github.com/apache/reef/blob/master/lang/cs/Org.Apache.REEF.Client/YARN/WindowsYarnJobCommandProvider.cs),

implementing IYarnJobCommandProvider.cs used in YARN client. This is the only Yarn command builder in the code. And this command is using "Org.Apache.REEF.Bridge.exe" to launch the driver.



  1.  In the example HelloReef.cs (https://github.com/apache/reef/blob/master/lang/cs/Org.Apache.REEF.Examples.HelloREEF/HelloREEF.cs#L63), they are adding the Dlls files in global folder, driver configuration file and reef-bridge-client-0.17.0-SNAPSHOT-shaded.jar in local folders, and Org.Apache.REEF.Bridge.exe" and its configuration file, by (https://github.com/apache/reef/blob/master/lang/cs/Org.Apache.REEF.Client/Common/DriverFolderPreparationHelper.cs#L83).
In general, is HelloReef.cs expecting remote cluster to be Windows?

Thanks!

Chenxi




Re: Questions on REEF .NET Driver execution

Posted by Markus Weimer <ma...@weimo.de>.
On Fri, Mar 2, 2018 at 3:03 PM, Chenxi Zhao <
Chenxi.Zhao@microsoft.com.invalid> wrote:

> In REEF .NET, do we expect the remote cluster, e.g. YARN, to be Windows
> environment only? I may miss this part but how do we take care the UNIX
> cluster environment from REEF .NET, e.g. submitting Jobs to Linux YARN
> cluster?
>

Today, REEF.NET is not available on Linux. Doug is working on fixing that :)

Markus

RE: Questions on REEF .NET Driver execution

Posted by Chenxi Zhao <Ch...@microsoft.com.INVALID>.
That is right. It won't need to change any exsiting behavior of other runtime implementation.

Chenxi

Get Outlook for Android<https://aka.ms/ghei36>

________________________________
From: Julia Wang (QIUHE) <Qi...@microsoft.com.INVALID>
Sent: Thursday, March 8, 2018 5:12:28 PM
To: dev@reef.apache.org
Subject: RE: Questions on REEF .NET Driver execution

Hi Chenxi,

So if I understand correctly, parallel to Local and Yarn, you will have an implementation for AzureBatch at both client and Java runtime side, and the way to submit job is still through
Org.Apache.REEF.Bridge.exe-> AzureBootStrapREEFLauncher-> REEFLauncher
Right?

--Julia

-----Original Message-----
From: Chenxi Zhao [mailto:Chenxi.Zhao@microsoft.com.INVALID]
Sent: Thursday, March 8, 2018 4:59 PM
To: dev@reef.apache.org
Subject: RE: Questions on REEF .NET Driver execution

My scenario is to submit job to AzureBatch. not YARN. But YARN is actually a good example for me to work on AzureBatch implementations. That is why I would like to understand YARN implementation well.

In my case, AzureBatchClient on .NET is submitting the command line invoking an "AzureBootStrapREEFLauncher", that calls REEFLauncher at last - just like YARN.

Chenxi



-----Original Message-----
From: Julia Wang (QIUHE) <Qi...@microsoft.com.INVALID>
Sent: Thursday, March 8, 2018 4:10 PM
To: dev@reef.apache.org
Subject: RE: Questions on REEF .NET Driver execution

Yes, the command line we submitted to YARN job is

reef\Org.Apache.REEF.Bridge.exe with class path, and -Dproc_reef org.apache.reef.bridge.client.YarnBootstrapREEFLauncher yarn-job-parameters.json reef/local/yarn-app-parameters.json

YarnBootstrapREEFLauncher is the entry class in Java for .Net launcher. Eventually it calls REEFLauncher in Java.

What is your scenario? Do you use REEF client? How do you submit Job to YARN?

-- Julia

-----Original Message-----
From: Chenxi Zhao [mailto:Chenxi.Zhao@microsoft.com.INVALID]
Sent: Thursday, March 8, 2018 2:43 PM
To: dev@reef.apache.org
Subject: RE: Questions on REEF .NET Driver execution

Appreciate everyone's help! I finally got through the problem.

The reason is that in order to let java bridge class to able to call JNI to Org.Apache.REEF.Bridge, it has to go through "Org.Apache.REEF.Bridge.exe". So the command line would be like "Org.Apache.REEF.Bridge.exe java -Xmx512m -XX:PermSize=128m -XX:MaxPermSize=128m -classpath 'reef/local/*;reef/global/*;' -Dproc_reef org.apache.reef.bridge.client.AzureBatchBootstrapREEFLauncher reef/job-submission-params.json" (Here uses my ongoing AzureBatch implementation).

I found a secondary issue is that, to implement a bridge launcher for a runtime in lang/java/reef-bridge-client, the class name has to be a substring of "REEFLauncher". It is enforced in https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Freef%2Fblob%2Fmaster%2Flang%2Fcs%2FOrg.Apache.REEF.Bridge%2FDriverLauncher.cpp%23L92&data=04%7C01%7CQiuhe.Wang%40microsoft.com%7C8c3d950caa4b4923613108d58545f509%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636561457870462102%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwifQ%3D%3D%7C-2&sdata=4uYz2TGqBkMu6NXxOoIZbtxi7sMdVnFrdV%2BulArzTxg%3D&reserved=0, which is very unfriendly code. I think this may be worth to open a ticket. Shall we?

Thanks,
Chenxi




-----Original Message-----
From: Julia Wang (QIUHE) <Qi...@microsoft.com.INVALID>
Sent: Thursday, March 8, 2018 1:20 PM
To: dev@reef.apache.org
Subject: RE: Questions on REEF .NET Driver execution

.Net hello REEF runs on environment that needs both .Net and Java installed. If runs on YARN, it also needs Hadoop in the environment set up. Not sure what additional settings are required for AzureBatch.

For dlls and jar files used by REEF driver, as long as you put all REEF dlls in the current client folder, REEF.Client wraps all needed jar files and extract them on the fly. Additional application files can be set through JobRequestBuilder and also put in the client working directory, REEF will pick them up.

--Julia

-----Original Message-----
From: Scott Inglis [mailto:msinglft@gmail.com]
Sent: Thursday, March 8, 2018 12:05 PM
To: dev@reef.apache.org
Subject: Re: Questions on REEF .NET Driver execution

Chenxi and I also talked offline -- and I wanted to reflect those discussions to the dev alias.

So with his changes, he is using the HelloREEF example to test the Azure batch runtime. If he changes HelloREEF back to local runtime, it works. So its only when running with AzureBatch that this happens. Because this works with the local runtime, I think we can rule out any missing dll dependencies (as this change does add Azure SDK dlls to the project from what I understand).

So the other possibility is the current working folder and how the paths are handled -- maybe these are different for the runtime? Like local does something different than what AzureBatch does with setting up the environment.

Chenxi was going to look into this area.


On Thu, Mar 8, 2018 at 11:34 AM, Markus Weimer <ma...@weimo.de> wrote:

> On Wed, Mar 7, 2018 at 11:08 PM, Chenxi Zhao <
> Chenxi.Zhao@microsoft.com.invalid> wrote:
>
> > I notice that in .NET, Client is submitting to the java bridge classes.
> > E.g. YarnBootstrapREEFLauncher and LocalClient. In these launchers,
> > they are binding Driver configuration like ON_DRIVER_STARTED to .NET
> > side
> driver
> > code through JNI call
> > https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgit
> > hub.com%2Fapache%2F&data=04%7C01%7CQiuhe.Wang%40microsoft.com%7Cf916
> > 44d4841c428bd51908d5852fdebd%7C72f988bf86f141af91ab2d7cd011db47%7C1%
> > 7C0%7C636561363004417602%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDA
> > iLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwifQ%3D%3D%7C-2&sdata=srwjDmNK4%2FxMt
> > fFPQejocbLhhVnu2KeFqkr%2BujssrX0%3D&reserved=0
> > reef/blob/master/lang/java/reef-bridge-java/src/main/
> java/org/apache/reef/
> > javabridge/NativeInterop.java#L38).
> >
> > Could some one help me understand how java bridge finds the dll that
> > contains those natives calls in local/Yarn runtime?
> >
>
> IIRC, we are asking YARN to launch `Driver.exe` which in turn depends
> on that DLL. That exe launches both the JVM and the CLR in-process and
> wires it all up. Doug would know better :)
>
> Markus
>

RE: Questions on REEF .NET Driver execution

Posted by "Julia Wang (QIUHE)" <Qi...@microsoft.com.INVALID>.
Hi Chenxi,

So if I understand correctly, parallel to Local and Yarn, you will have an implementation for AzureBatch at both client and Java runtime side, and the way to submit job is still through 
Org.Apache.REEF.Bridge.exe-> AzureBootStrapREEFLauncher-> REEFLauncher
Right?

--Julia

-----Original Message-----
From: Chenxi Zhao [mailto:Chenxi.Zhao@microsoft.com.INVALID] 
Sent: Thursday, March 8, 2018 4:59 PM
To: dev@reef.apache.org
Subject: RE: Questions on REEF .NET Driver execution

My scenario is to submit job to AzureBatch. not YARN. But YARN is actually a good example for me to work on AzureBatch implementations. That is why I would like to understand YARN implementation well.

In my case, AzureBatchClient on .NET is submitting the command line invoking an "AzureBootStrapREEFLauncher", that calls REEFLauncher at last - just like YARN.

Chenxi



-----Original Message-----
From: Julia Wang (QIUHE) <Qi...@microsoft.com.INVALID>
Sent: Thursday, March 8, 2018 4:10 PM
To: dev@reef.apache.org
Subject: RE: Questions on REEF .NET Driver execution

Yes, the command line we submitted to YARN job is 

reef\Org.Apache.REEF.Bridge.exe with class path, and -Dproc_reef org.apache.reef.bridge.client.YarnBootstrapREEFLauncher yarn-job-parameters.json reef/local/yarn-app-parameters.json

YarnBootstrapREEFLauncher is the entry class in Java for .Net launcher. Eventually it calls REEFLauncher in Java. 

What is your scenario? Do you use REEF client? How do you submit Job to YARN? 

-- Julia

-----Original Message-----
From: Chenxi Zhao [mailto:Chenxi.Zhao@microsoft.com.INVALID]
Sent: Thursday, March 8, 2018 2:43 PM
To: dev@reef.apache.org
Subject: RE: Questions on REEF .NET Driver execution

Appreciate everyone's help! I finally got through the problem.

The reason is that in order to let java bridge class to able to call JNI to Org.Apache.REEF.Bridge, it has to go through "Org.Apache.REEF.Bridge.exe". So the command line would be like "Org.Apache.REEF.Bridge.exe java -Xmx512m -XX:PermSize=128m -XX:MaxPermSize=128m -classpath 'reef/local/*;reef/global/*;' -Dproc_reef org.apache.reef.bridge.client.AzureBatchBootstrapREEFLauncher reef/job-submission-params.json" (Here uses my ongoing AzureBatch implementation).

I found a secondary issue is that, to implement a bridge launcher for a runtime in lang/java/reef-bridge-client, the class name has to be a substring of "REEFLauncher". It is enforced in https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Freef%2Fblob%2Fmaster%2Flang%2Fcs%2FOrg.Apache.REEF.Bridge%2FDriverLauncher.cpp%23L92&data=04%7C01%7CQiuhe.Wang%40microsoft.com%7C8c3d950caa4b4923613108d58545f509%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636561457870462102%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwifQ%3D%3D%7C-2&sdata=4uYz2TGqBkMu6NXxOoIZbtxi7sMdVnFrdV%2BulArzTxg%3D&reserved=0, which is very unfriendly code. I think this may be worth to open a ticket. Shall we?

Thanks,
Chenxi




-----Original Message-----
From: Julia Wang (QIUHE) <Qi...@microsoft.com.INVALID>
Sent: Thursday, March 8, 2018 1:20 PM
To: dev@reef.apache.org
Subject: RE: Questions on REEF .NET Driver execution

.Net hello REEF runs on environment that needs both .Net and Java installed. If runs on YARN, it also needs Hadoop in the environment set up. Not sure what additional settings are required for AzureBatch. 

For dlls and jar files used by REEF driver, as long as you put all REEF dlls in the current client folder, REEF.Client wraps all needed jar files and extract them on the fly. Additional application files can be set through JobRequestBuilder and also put in the client working directory, REEF will pick them up. 

--Julia

-----Original Message-----
From: Scott Inglis [mailto:msinglft@gmail.com]
Sent: Thursday, March 8, 2018 12:05 PM
To: dev@reef.apache.org
Subject: Re: Questions on REEF .NET Driver execution

Chenxi and I also talked offline -- and I wanted to reflect those discussions to the dev alias.

So with his changes, he is using the HelloREEF example to test the Azure batch runtime. If he changes HelloREEF back to local runtime, it works. So its only when running with AzureBatch that this happens. Because this works with the local runtime, I think we can rule out any missing dll dependencies (as this change does add Azure SDK dlls to the project from what I understand).

So the other possibility is the current working folder and how the paths are handled -- maybe these are different for the runtime? Like local does something different than what AzureBatch does with setting up the environment.

Chenxi was going to look into this area.


On Thu, Mar 8, 2018 at 11:34 AM, Markus Weimer <ma...@weimo.de> wrote:

> On Wed, Mar 7, 2018 at 11:08 PM, Chenxi Zhao < 
> Chenxi.Zhao@microsoft.com.invalid> wrote:
>
> > I notice that in .NET, Client is submitting to the java bridge classes.
> > E.g. YarnBootstrapREEFLauncher and LocalClient. In these launchers, 
> > they are binding Driver configuration like ON_DRIVER_STARTED to .NET 
> > side
> driver
> > code through JNI call
> > https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgit
> > hub.com%2Fapache%2F&data=04%7C01%7CQiuhe.Wang%40microsoft.com%7Cf916
> > 44d4841c428bd51908d5852fdebd%7C72f988bf86f141af91ab2d7cd011db47%7C1%
> > 7C0%7C636561363004417602%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDA
> > iLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwifQ%3D%3D%7C-2&sdata=srwjDmNK4%2FxMt
> > fFPQejocbLhhVnu2KeFqkr%2BujssrX0%3D&reserved=0
> > reef/blob/master/lang/java/reef-bridge-java/src/main/
> java/org/apache/reef/
> > javabridge/NativeInterop.java#L38).
> >
> > Could some one help me understand how java bridge finds the dll that 
> > contains those natives calls in local/Yarn runtime?
> >
>
> IIRC, we are asking YARN to launch `Driver.exe` which in turn depends 
> on that DLL. That exe launches both the JVM and the CLR in-process and 
> wires it all up. Doug would know better :)
>
> Markus
>

RE: Questions on REEF .NET Driver execution

Posted by Chenxi Zhao <Ch...@microsoft.com.INVALID>.
My scenario is to submit job to AzureBatch. not YARN. But YARN is actually a good example for me to work on AzureBatch implementations. That is why I would like to understand YARN implementation well.

In my case, AzureBatchClient on .NET is submitting the command line invoking an "AzureBootStrapREEFLauncher", that calls REEFLauncher at last - just like YARN.

Chenxi



-----Original Message-----
From: Julia Wang (QIUHE) <Qi...@microsoft.com.INVALID> 
Sent: Thursday, March 8, 2018 4:10 PM
To: dev@reef.apache.org
Subject: RE: Questions on REEF .NET Driver execution

Yes, the command line we submitted to YARN job is 

reef\Org.Apache.REEF.Bridge.exe with class path, and -Dproc_reef org.apache.reef.bridge.client.YarnBootstrapREEFLauncher yarn-job-parameters.json reef/local/yarn-app-parameters.json

YarnBootstrapREEFLauncher is the entry class in Java for .Net launcher. Eventually it calls REEFLauncher in Java. 

What is your scenario? Do you use REEF client? How do you submit Job to YARN? 

-- Julia

-----Original Message-----
From: Chenxi Zhao [mailto:Chenxi.Zhao@microsoft.com.INVALID]
Sent: Thursday, March 8, 2018 2:43 PM
To: dev@reef.apache.org
Subject: RE: Questions on REEF .NET Driver execution

Appreciate everyone's help! I finally got through the problem.

The reason is that in order to let java bridge class to able to call JNI to Org.Apache.REEF.Bridge, it has to go through "Org.Apache.REEF.Bridge.exe". So the command line would be like "Org.Apache.REEF.Bridge.exe java -Xmx512m -XX:PermSize=128m -XX:MaxPermSize=128m -classpath 'reef/local/*;reef/global/*;' -Dproc_reef org.apache.reef.bridge.client.AzureBatchBootstrapREEFLauncher reef/job-submission-params.json" (Here uses my ongoing AzureBatch implementation).

I found a secondary issue is that, to implement a bridge launcher for a runtime in lang/java/reef-bridge-client, the class name has to be a substring of "REEFLauncher". It is enforced in https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Freef%2Fblob%2Fmaster%2Flang%2Fcs%2FOrg.Apache.REEF.Bridge%2FDriverLauncher.cpp%23L92&data=04%7C01%7CQiuhe.Wang%40microsoft.com%7C8c3d950caa4b4923613108d58545f509%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636561457870462102%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwifQ%3D%3D%7C-2&sdata=4uYz2TGqBkMu6NXxOoIZbtxi7sMdVnFrdV%2BulArzTxg%3D&reserved=0, which is very unfriendly code. I think this may be worth to open a ticket. Shall we?

Thanks,
Chenxi




-----Original Message-----
From: Julia Wang (QIUHE) <Qi...@microsoft.com.INVALID>
Sent: Thursday, March 8, 2018 1:20 PM
To: dev@reef.apache.org
Subject: RE: Questions on REEF .NET Driver execution

.Net hello REEF runs on environment that needs both .Net and Java installed. If runs on YARN, it also needs Hadoop in the environment set up. Not sure what additional settings are required for AzureBatch. 

For dlls and jar files used by REEF driver, as long as you put all REEF dlls in the current client folder, REEF.Client wraps all needed jar files and extract them on the fly. Additional application files can be set through JobRequestBuilder and also put in the client working directory, REEF will pick them up. 

--Julia

-----Original Message-----
From: Scott Inglis [mailto:msinglft@gmail.com]
Sent: Thursday, March 8, 2018 12:05 PM
To: dev@reef.apache.org
Subject: Re: Questions on REEF .NET Driver execution

Chenxi and I also talked offline -- and I wanted to reflect those discussions to the dev alias.

So with his changes, he is using the HelloREEF example to test the Azure batch runtime. If he changes HelloREEF back to local runtime, it works. So its only when running with AzureBatch that this happens. Because this works with the local runtime, I think we can rule out any missing dll dependencies (as this change does add Azure SDK dlls to the project from what I understand).

So the other possibility is the current working folder and how the paths are handled -- maybe these are different for the runtime? Like local does something different than what AzureBatch does with setting up the environment.

Chenxi was going to look into this area.


On Thu, Mar 8, 2018 at 11:34 AM, Markus Weimer <ma...@weimo.de> wrote:

> On Wed, Mar 7, 2018 at 11:08 PM, Chenxi Zhao < 
> Chenxi.Zhao@microsoft.com.invalid> wrote:
>
> > I notice that in .NET, Client is submitting to the java bridge classes.
> > E.g. YarnBootstrapREEFLauncher and LocalClient. In these launchers, 
> > they are binding Driver configuration like ON_DRIVER_STARTED to .NET 
> > side
> driver
> > code through JNI call
> > https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgit
> > hub.com%2Fapache%2F&data=04%7C01%7CQiuhe.Wang%40microsoft.com%7Cf916
> > 44d4841c428bd51908d5852fdebd%7C72f988bf86f141af91ab2d7cd011db47%7C1%
> > 7C0%7C636561363004417602%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDA
> > iLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwifQ%3D%3D%7C-2&sdata=srwjDmNK4%2FxMt
> > fFPQejocbLhhVnu2KeFqkr%2BujssrX0%3D&reserved=0
> > reef/blob/master/lang/java/reef-bridge-java/src/main/
> java/org/apache/reef/
> > javabridge/NativeInterop.java#L38).
> >
> > Could some one help me understand how java bridge finds the dll that 
> > contains those natives calls in local/Yarn runtime?
> >
>
> IIRC, we are asking YARN to launch `Driver.exe` which in turn depends 
> on that DLL. That exe launches both the JVM and the CLR in-process and 
> wires it all up. Doug would know better :)
>
> Markus
>

RE: Questions on REEF .NET Driver execution

Posted by "Julia Wang (QIUHE)" <Qi...@microsoft.com.INVALID>.
Yes, the command line we submitted to YARN job is 

reef\Org.Apache.REEF.Bridge.exe with class path, and 
-Dproc_reef org.apache.reef.bridge.client.YarnBootstrapREEFLauncher yarn-job-parameters.json reef/local/yarn-app-parameters.json

YarnBootstrapREEFLauncher is the entry class in Java for .Net launcher. Eventually it calls REEFLauncher in Java. 

What is your scenario? Do you use REEF client? How do you submit Job to YARN? 

-- Julia

-----Original Message-----
From: Chenxi Zhao [mailto:Chenxi.Zhao@microsoft.com.INVALID] 
Sent: Thursday, March 8, 2018 2:43 PM
To: dev@reef.apache.org
Subject: RE: Questions on REEF .NET Driver execution

Appreciate everyone's help! I finally got through the problem.

The reason is that in order to let java bridge class to able to call JNI to Org.Apache.REEF.Bridge, it has to go through "Org.Apache.REEF.Bridge.exe". So the command line would be like "Org.Apache.REEF.Bridge.exe java -Xmx512m -XX:PermSize=128m -XX:MaxPermSize=128m -classpath 'reef/local/*;reef/global/*;' -Dproc_reef org.apache.reef.bridge.client.AzureBatchBootstrapREEFLauncher reef/job-submission-params.json" (Here uses my ongoing AzureBatch implementation).

I found a secondary issue is that, to implement a bridge launcher for a runtime in lang/java/reef-bridge-client, the class name has to be a substring of "REEFLauncher". It is enforced in https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Freef%2Fblob%2Fmaster%2Flang%2Fcs%2FOrg.Apache.REEF.Bridge%2FDriverLauncher.cpp%23L92&data=04%7C01%7CQiuhe.Wang%40microsoft.com%7C8c3d950caa4b4923613108d58545f509%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636561457870462102%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwifQ%3D%3D%7C-2&sdata=4uYz2TGqBkMu6NXxOoIZbtxi7sMdVnFrdV%2BulArzTxg%3D&reserved=0, which is very unfriendly code. I think this may be worth to open a ticket. Shall we?

Thanks,
Chenxi




-----Original Message-----
From: Julia Wang (QIUHE) <Qi...@microsoft.com.INVALID>
Sent: Thursday, March 8, 2018 1:20 PM
To: dev@reef.apache.org
Subject: RE: Questions on REEF .NET Driver execution

.Net hello REEF runs on environment that needs both .Net and Java installed. If runs on YARN, it also needs Hadoop in the environment set up. Not sure what additional settings are required for AzureBatch. 

For dlls and jar files used by REEF driver, as long as you put all REEF dlls in the current client folder, REEF.Client wraps all needed jar files and extract them on the fly. Additional application files can be set through JobRequestBuilder and also put in the client working directory, REEF will pick them up. 

--Julia

-----Original Message-----
From: Scott Inglis [mailto:msinglft@gmail.com]
Sent: Thursday, March 8, 2018 12:05 PM
To: dev@reef.apache.org
Subject: Re: Questions on REEF .NET Driver execution

Chenxi and I also talked offline -- and I wanted to reflect those discussions to the dev alias.

So with his changes, he is using the HelloREEF example to test the Azure batch runtime. If he changes HelloREEF back to local runtime, it works. So its only when running with AzureBatch that this happens. Because this works with the local runtime, I think we can rule out any missing dll dependencies (as this change does add Azure SDK dlls to the project from what I understand).

So the other possibility is the current working folder and how the paths are handled -- maybe these are different for the runtime? Like local does something different than what AzureBatch does with setting up the environment.

Chenxi was going to look into this area.


On Thu, Mar 8, 2018 at 11:34 AM, Markus Weimer <ma...@weimo.de> wrote:

> On Wed, Mar 7, 2018 at 11:08 PM, Chenxi Zhao < 
> Chenxi.Zhao@microsoft.com.invalid> wrote:
>
> > I notice that in .NET, Client is submitting to the java bridge classes.
> > E.g. YarnBootstrapREEFLauncher and LocalClient. In these launchers, 
> > they are binding Driver configuration like ON_DRIVER_STARTED to .NET 
> > side
> driver
> > code through JNI call
> > https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgit
> > hub.com%2Fapache%2F&data=04%7C01%7CQiuhe.Wang%40microsoft.com%7Cf916
> > 44d4841c428bd51908d5852fdebd%7C72f988bf86f141af91ab2d7cd011db47%7C1%
> > 7C0%7C636561363004417602%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDA
> > iLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwifQ%3D%3D%7C-2&sdata=srwjDmNK4%2FxMt
> > fFPQejocbLhhVnu2KeFqkr%2BujssrX0%3D&reserved=0
> > reef/blob/master/lang/java/reef-bridge-java/src/main/
> java/org/apache/reef/
> > javabridge/NativeInterop.java#L38).
> >
> > Could some one help me understand how java bridge finds the dll that 
> > contains those natives calls in local/Yarn runtime?
> >
>
> IIRC, we are asking YARN to launch `Driver.exe` which in turn depends 
> on that DLL. That exe launches both the JVM and the CLR in-process and 
> wires it all up. Doug would know better :)
>
> Markus
>

RE: Questions on REEF .NET Driver execution

Posted by Chenxi Zhao <Ch...@microsoft.com.INVALID>.
Appreciate everyone's help! I finally got through the problem.

The reason is that in order to let java bridge class to able to call JNI to Org.Apache.REEF.Bridge, it has to go through "Org.Apache.REEF.Bridge.exe". So the command line would be like "Org.Apache.REEF.Bridge.exe java -Xmx512m -XX:PermSize=128m -XX:MaxPermSize=128m -classpath 'reef/local/*;reef/global/*;' -Dproc_reef org.apache.reef.bridge.client.AzureBatchBootstrapREEFLauncher reef/job-submission-params.json" (Here uses my ongoing AzureBatch implementation).

I found a secondary issue is that, to implement a bridge launcher for a runtime in lang/java/reef-bridge-client, the class name has to be a substring of "REEFLauncher". It is enforced in https://github.com/apache/reef/blob/master/lang/cs/Org.Apache.REEF.Bridge/DriverLauncher.cpp#L92, which is very unfriendly code. I think this may be worth to open a ticket. Shall we?

Thanks,
Chenxi




-----Original Message-----
From: Julia Wang (QIUHE) <Qi...@microsoft.com.INVALID> 
Sent: Thursday, March 8, 2018 1:20 PM
To: dev@reef.apache.org
Subject: RE: Questions on REEF .NET Driver execution

.Net hello REEF runs on environment that needs both .Net and Java installed. If runs on YARN, it also needs Hadoop in the environment set up. Not sure what additional settings are required for AzureBatch. 

For dlls and jar files used by REEF driver, as long as you put all REEF dlls in the current client folder, REEF.Client wraps all needed jar files and extract them on the fly. Additional application files can be set through JobRequestBuilder and also put in the client working directory, REEF will pick them up. 

--Julia

-----Original Message-----
From: Scott Inglis [mailto:msinglft@gmail.com]
Sent: Thursday, March 8, 2018 12:05 PM
To: dev@reef.apache.org
Subject: Re: Questions on REEF .NET Driver execution

Chenxi and I also talked offline -- and I wanted to reflect those discussions to the dev alias.

So with his changes, he is using the HelloREEF example to test the Azure batch runtime. If he changes HelloREEF back to local runtime, it works. So its only when running with AzureBatch that this happens. Because this works with the local runtime, I think we can rule out any missing dll dependencies (as this change does add Azure SDK dlls to the project from what I understand).

So the other possibility is the current working folder and how the paths are handled -- maybe these are different for the runtime? Like local does something different than what AzureBatch does with setting up the environment.

Chenxi was going to look into this area.


On Thu, Mar 8, 2018 at 11:34 AM, Markus Weimer <ma...@weimo.de> wrote:

> On Wed, Mar 7, 2018 at 11:08 PM, Chenxi Zhao < 
> Chenxi.Zhao@microsoft.com.invalid> wrote:
>
> > I notice that in .NET, Client is submitting to the java bridge classes.
> > E.g. YarnBootstrapREEFLauncher and LocalClient. In these launchers, 
> > they are binding Driver configuration like ON_DRIVER_STARTED to .NET 
> > side
> driver
> > code through JNI call
> > https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgit
> > hub.com%2Fapache%2F&data=04%7C01%7CQiuhe.Wang%40microsoft.com%7Cf916
> > 44d4841c428bd51908d5852fdebd%7C72f988bf86f141af91ab2d7cd011db47%7C1%
> > 7C0%7C636561363004417602%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDA
> > iLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwifQ%3D%3D%7C-2&sdata=srwjDmNK4%2FxMt
> > fFPQejocbLhhVnu2KeFqkr%2BujssrX0%3D&reserved=0
> > reef/blob/master/lang/java/reef-bridge-java/src/main/
> java/org/apache/reef/
> > javabridge/NativeInterop.java#L38).
> >
> > Could some one help me understand how java bridge finds the dll that 
> > contains those natives calls in local/Yarn runtime?
> >
>
> IIRC, we are asking YARN to launch `Driver.exe` which in turn depends 
> on that DLL. That exe launches both the JVM and the CLR in-process and 
> wires it all up. Doug would know better :)
>
> Markus
>

RE: Questions on REEF .NET Driver execution

Posted by "Julia Wang (QIUHE)" <Qi...@microsoft.com.INVALID>.
.Net hello REEF runs on environment that needs both .Net and Java installed. If runs on YARN, it also needs Hadoop in the environment set up. Not sure what additional settings are required for AzureBatch. 

For dlls and jar files used by REEF driver, as long as you put all REEF dlls in the current client folder, REEF.Client wraps all needed jar files and extract them on the fly. Additional application files can be set through JobRequestBuilder and also put in the client working directory, REEF will pick them up. 

--Julia

-----Original Message-----
From: Scott Inglis [mailto:msinglft@gmail.com] 
Sent: Thursday, March 8, 2018 12:05 PM
To: dev@reef.apache.org
Subject: Re: Questions on REEF .NET Driver execution

Chenxi and I also talked offline -- and I wanted to reflect those discussions to the dev alias.

So with his changes, he is using the HelloREEF example to test the Azure batch runtime. If he changes HelloREEF back to local runtime, it works. So its only when running with AzureBatch that this happens. Because this works with the local runtime, I think we can rule out any missing dll dependencies (as this change does add Azure SDK dlls to the project from what I understand).

So the other possibility is the current working folder and how the paths are handled -- maybe these are different for the runtime? Like local does something different than what AzureBatch does with setting up the environment.

Chenxi was going to look into this area.


On Thu, Mar 8, 2018 at 11:34 AM, Markus Weimer <ma...@weimo.de> wrote:

> On Wed, Mar 7, 2018 at 11:08 PM, Chenxi Zhao < 
> Chenxi.Zhao@microsoft.com.invalid> wrote:
>
> > I notice that in .NET, Client is submitting to the java bridge classes.
> > E.g. YarnBootstrapREEFLauncher and LocalClient. In these launchers, 
> > they are binding Driver configuration like ON_DRIVER_STARTED to .NET 
> > side
> driver
> > code through JNI call 
> > https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgit
> > hub.com%2Fapache%2F&data=04%7C01%7CQiuhe.Wang%40microsoft.com%7Cf916
> > 44d4841c428bd51908d5852fdebd%7C72f988bf86f141af91ab2d7cd011db47%7C1%
> > 7C0%7C636561363004417602%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDA
> > iLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwifQ%3D%3D%7C-2&sdata=srwjDmNK4%2FxMt
> > fFPQejocbLhhVnu2KeFqkr%2BujssrX0%3D&reserved=0
> > reef/blob/master/lang/java/reef-bridge-java/src/main/
> java/org/apache/reef/
> > javabridge/NativeInterop.java#L38).
> >
> > Could some one help me understand how java bridge finds the dll that 
> > contains those natives calls in local/Yarn runtime?
> >
>
> IIRC, we are asking YARN to launch `Driver.exe` which in turn depends 
> on that DLL. That exe launches both the JVM and the CLR in-process and 
> wires it all up. Doug would know better :)
>
> Markus
>

Re: Questions on REEF .NET Driver execution

Posted by Scott Inglis <ms...@gmail.com>.
Chenxi and I also talked offline -- and I wanted to reflect those
discussions to the dev alias.

So with his changes, he is using the HelloREEF example to test the Azure
batch runtime. If he changes HelloREEF back to local runtime, it works. So
its only when running with AzureBatch that this happens. Because this works
with the local runtime, I think we can rule out any missing dll
dependencies (as this change does add Azure SDK dlls to the project from
what I understand).

So the other possibility is the current working folder and how the paths
are handled -- maybe these are different for the runtime? Like local does
something different than what AzureBatch does with setting up the
environment.

Chenxi was going to look into this area.


On Thu, Mar 8, 2018 at 11:34 AM, Markus Weimer <ma...@weimo.de> wrote:

> On Wed, Mar 7, 2018 at 11:08 PM, Chenxi Zhao <
> Chenxi.Zhao@microsoft.com.invalid> wrote:
>
> > I notice that in .NET, Client is submitting to the java bridge classes.
> > E.g. YarnBootstrapREEFLauncher and LocalClient. In these launchers, they
> > are binding Driver configuration like ON_DRIVER_STARTED to .NET side
> driver
> > code through JNI call https://github.com/apache/
> > reef/blob/master/lang/java/reef-bridge-java/src/main/
> java/org/apache/reef/
> > javabridge/NativeInterop.java#L38).
> >
> > Could some one help me understand how java bridge finds the dll that
> > contains those natives calls in local/Yarn runtime?
> >
>
> IIRC, we are asking YARN to launch `Driver.exe` which in turn depends on
> that DLL. That exe launches both the JVM and the CLR in-process and wires
> it all up. Doug would know better :)
>
> Markus
>

Re: Questions on REEF .NET Driver execution

Posted by Markus Weimer <ma...@weimo.de>.
On Wed, Mar 7, 2018 at 11:08 PM, Chenxi Zhao <
Chenxi.Zhao@microsoft.com.invalid> wrote:

> I notice that in .NET, Client is submitting to the java bridge classes.
> E.g. YarnBootstrapREEFLauncher and LocalClient. In these launchers, they
> are binding Driver configuration like ON_DRIVER_STARTED to .NET side driver
> code through JNI call https://github.com/apache/
> reef/blob/master/lang/java/reef-bridge-java/src/main/java/org/apache/reef/
> javabridge/NativeInterop.java#L38).
>
> Could some one help me understand how java bridge finds the dll that
> contains those natives calls in local/Yarn runtime?
>

IIRC, we are asking YARN to launch `Driver.exe` which in turn depends on
that DLL. That exe launches both the JVM and the CLR in-process and wires
it all up. Doug would know better :)

Markus

RE: Questions on REEF .NET Driver execution

Posted by Chenxi Zhao <Ch...@microsoft.com.INVALID>.
Hi,

I have been working AzureBatch .NET implementation and am a bit struggling understanding REEF .NET and have a follow up question. Greatly Appreciate if someone could help commend on below questions:

Follow up -


I notice that in .NET, Client is submitting to the java bridge classes. E.g. YarnBootstrapREEFLauncher and LocalClient. In these launchers, they are binding Driver configuration like ON_DRIVER_STARTED to .NET side driver code through JNI call https://github.com/apache/reef/blob/master/lang/java/reef-bridge-java/src/main/java/org/apache/reef/javabridge/NativeInterop.java#L38).

Could some one help me understand how java bridge finds the dll that contains those natives calls in local/Yarn runtime?

I am trying to implement the similar approach in AzureBatch.NET but I have the following exception: java.lang.UnsatisfiedLinkError: org.apache.reef.javabridge.NativeInterop.clrSystemSetupBridgeHandlerManager(Ljava/lang/String;Lorg/apache/reef/javabridge/BridgeHandlerManager;Lorg/apache/reef/javabridge/EvaluatorRequestorBridge;)V

I did some searchs on the error : it means the dll is not included either in local or global folder. I found that the implementation of the native calls is in Org.Apache.REEF.Bridge - javaClrBridge.cpp. However the project seems to be compiled to exe but not dlls.

Thanks!
Chenxi










From: Chenxi Zhao
Sent: Friday, March 2, 2018 3:03 PM
To: dev@reef.apache.org
Subject: Questions on REEF .NET Driver execution

Hi,

I am looking into REEF .NET Driver execution of YARN and have questions on the expected driver run environment. Could someone help clarify it?

In REEF .NET, do we expect the remote cluster, e.g. YARN, to be Windows environment only? I may miss this part but how do we take care the UNIX cluster environment from REEF .NET, e.g. submitting Jobs to Linux YARN cluster?

As far as I can find -


  1.  There is only WindowsYarnJobCommandProvider.cs (https://github.com/apache/reef/blob/master/lang/cs/Org.Apache.REEF.Client/YARN/WindowsYarnJobCommandProvider.cs),

implementing IYarnJobCommandProvider.cs used in YARN client. This is the only Yarn command builder in the code. And this command is using "Org.Apache.REEF.Bridge.exe" to launch the driver.



  1.  In the example HelloReef.cs (https://github.com/apache/reef/blob/master/lang/cs/Org.Apache.REEF.Examples.HelloREEF/HelloREEF.cs#L63), they are adding the Dlls files in global folder, driver configuration file and reef-bridge-client-0.17.0-SNAPSHOT-shaded.jar in local folders, and Org.Apache.REEF.Bridge.exe" and its configuration file, by (https://github.com/apache/reef/blob/master/lang/cs/Org.Apache.REEF.Client/Common/DriverFolderPreparationHelper.cs#L83).
In general, is HelloReef.cs expecting remote cluster to be Windows?

Thanks!

Chenxi