JIRA-2003

[tc...@gmail.com]

I have created a Jira (REEF-2003) that proposes the removal of Java on the
C# client for the revised bridge. After some investigation, I discovered
that the current Java-CLR bridge launches the initial job via Java, after
which an HTTP connection is established with the (running) C# driver for
control and status messaging i.e., C# does not use the Java client to kill
jobs or inquire about their status. The short question is whether we should
continue with this design (requiring Java to be installed on the client) or
find a way to avoid Java altogether on the client to alleviate the
burden/dependency of Java. Below, are my thoughts on this subject. 

 

In order to make a Java-less client for the new bridge (which presently acts
like the old bridge for submitting jobs i.e., it launches a java command
with the desired runtime parameters that are used to select/configure the
desired runtime and submit the job). This Jira will involve replicating such
job submission logic related to the runtime (YARN, Mesos, Azure Batch,
Local, etc.) in C#, and picking a static DriverConfiguration that will be
serialized in a driver.conf (this too has some per-job specific parameters
e.g., job id, that will somehow need to be revised or pulled out into the
runtime configuration); the static driver (defined in driver.conf) simply
binds to all handlers and publishes events to gRPC listening driver clients
(see REEF-2002 for more details). 

 

To make matters worse, the Java side will still need to be configured with a
runtime (e.g., YARN) so that it binds to the proper implementations for
requesting resources, launching evaluators, etc. Hence, our static driver
configuration might involve one for each supported runtime i.e.,
driver_yarn.conf, driver_azbatch.conf, driver_mesos.conf, driver_local.conf,
etc. 

 

Ultimately, I share the desire to remove Java from the .NET developer
static, but fear that doing so, when Java is still in the stack on the
driver, will result in an implementation nightmare. 

 

There has been some background discussions on creating a pure .NET Core
REEF. Naturally, this will take time, but would clearly yield a Java-less
everything for .NET developers. 

 

My questions to the REEF community is basically how best to proceed. Should
we go for a Java-less client that is able to submit an appropriate static
Java driver? Should we postpone this desideratum for a pure .NET Core REEF? 

 

Your feedback is most appreciated!

-Tyson

 

Re: JIRA-2003

[Byung-Gon Chun <bg...@gmail.com>]

I like tackling the problem in steps as Markus suggested.

-Gon

On Wed, Apr 18, 2018 at 6:10 AM, Markus Weimer <ma...@weimo.de> wrote:

> On Tue, Apr 17, 2018 at 10:20 AM, <tc...@gmail.com> wrote:
> > I have created a Jira (REEF-2003) that proposes the removal of Java on
> the C# client for the revised bridge. After some investigation, I
> discovered that the current Java-CLR bridge launches the initial job via
> Java, after which an HTTP connection is established with the (running) C#
> driver for control and status messaging i.e., C# does not use the Java
> client to kill jobs or inquire about their status. The short question is
> whether we should continue with this design (requiring Java to be installed
> on the client) or find a way to avoid Java altogether on the client to
> alleviate the burden/dependency of Java.
>
> Can we think of this as phases? For example:
>
> Phase I is to use Java for job submission, but the entire API is .NET.
> This is basically what it is today.
>
> Phase II replaces Java when dealing with runtimes that aren't Java
> affinitized. What I mean by that is not Mesos or YARN. Those
> environments are historically very Java-heavy and difficult to make
> work with other language environments. Also, people targeting them
> will more likely be comfortable with a Java dependencies. The runtimes
> which would have no Java dependency in this phase would be local and
> Azure Batch.
>
> Phase III replaces Java entirely on the client, for all runtimes. This
> might coincide with a Java-free driver, but doesn't have to.
>
> WDYT?
>
> Markus
>



-- 
Byung-Gon Chun

Re: JIRA-2003

[Markus Weimer <ma...@weimo.de>]

On Tue, Apr 17, 2018 at 10:20 AM, <tc...@gmail.com> wrote:
> I have created a Jira (REEF-2003) that proposes the removal of Java on the C# client for the revised bridge. After some investigation, I discovered that the current Java-CLR bridge launches the initial job via Java, after which an HTTP connection is established with the (running) C# driver for control and status messaging i.e., C# does not use the Java client to kill jobs or inquire about their status. The short question is whether we should continue with this design (requiring Java to be installed on the client) or find a way to avoid Java altogether on the client to alleviate the burden/dependency of Java.

Can we think of this as phases? For example:

Phase I is to use Java for job submission, but the entire API is .NET.
This is basically what it is today.

Phase II replaces Java when dealing with runtimes that aren't Java
affinitized. What I mean by that is not Mesos or YARN. Those
environments are historically very Java-heavy and difficult to make
work with other language environments. Also, people targeting them
will more likely be comfortable with a Java dependencies. The runtimes
which would have no Java dependency in this phase would be local and
Azure Batch.

Phase III replaces Java entirely on the client, for all runtimes. This
might coincide with a Java-free driver, but doesn't have to.

WDYT?

Markus