You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@reef.apache.org by Andrew Chung <af...@gmail.com> on 2015/11/19 01:17:10 UTC
REEF driver info endpoint in reef-webserver for Java driver restarts
Hi All,
For https://issues.apache.org/jira/browse/REEF-906, I would like to move
DriverInfo, EvaluatorInfo, EvaluatorList, and others into reef-common,
since I believe they can serve more purpose beyond reef-webserver. This is
also needed in order to support Driver Restarts on Java because Java
Evaluators only depend on reef-common but not on reef-webserver. There are
essentially two solutions here:
1. Split DriverInfo.avsc into DriverInfo.avsc and ReefServicesInfo.avsc,
and move DriverInfo, EvaluatorInfo, and EvaluatorList into reef-common. We
can then provide 2 separate endpoints for the Driver, one reporting the
driver RID, and the other one reporting the services running on the Driver
(i.e. a "services" endpoint). The Evaluator will thus know how to contact
the webserver driver endpoint and deserialize DriverInfo if restarts are
enabled, and won't need to introduce a dependency on reef-webserver.
2. Have Evaluator depend on reef-webserver.
However, to support 1, there are also a few approaches:
1. Change the "driver" endpoint to only return Driver RID and start time
directly, and provide a new "services" endpoint. For this solution, we assume
that no one currently is using this endpoint, and there will be no period
of deprecation.
2. Don't touch the "driver" endpoint. Add a "driverinfo" endpoint and
"services" endpoint. However, this introduces redundancy, and also uses a
name that isn't that great for the endpoint, in my opinion . We can choose
to deprecate the "driver" endpoint (though I'm not sure how to do this),
and remove it in a later iteration.
3. Don't touch the "driver" endpoint, and upgrade the version from v1 to v2
for the new "driver" and "services" endpoint. For this solution, we also
add redundancy, and I don't particularly think that it's worth it to
increment the version just for this minor change.
What do you think? Personally, I would like to go with 1-1, if no one is
really using the "driver" endpoint right now.
Thanks,
Andrew
Re: REEF driver info endpoint in reef-webserver for Java driver restarts
Posted by Andrew Chung <af...@gmail.com>.
If no one opposes, I will proceed after the (incubator) standard 48 hours
and fix up the JIRAs accordingly.
Thanks,
Andrew
On Wed, Nov 18, 2015 at 6:41 PM, Markus Weimer <ma...@weimo.de> wrote:
> On 2015-11-18 16:17, Andrew Chung wrote:
> > 2. Have Evaluator depend on reef-webserver.
>
> I'd be OK with this. The webserver is rather core to our manageability
> story and I don't see a reason not to use it. Also, we currently build
> one JAR for Driver and Evaluator anyhow, and therefore not having this
> dependency won't have benefits in JAR size.
>
> Markus
>
Re: REEF driver info endpoint in reef-webserver for Java driver restarts
Posted by Andrew Chung <af...@gmail.com>.
Going back to the original issue, is anyone opposed to changing the schema
for the "driver" endpoint (option 1-1)? Or should I go with options 1-2 or
1-3?
Thanks,
Andrew
On Mon, Nov 23, 2015 at 11:36 AM, Julia Wang (QIUHE) <
Qiuhe.Wang@microsoft.com> wrote:
> > What's the usual best practice for forming DLLs and EXEs? Similarly,
> what's the commonly accepted relationship between assemblies and NuGets?
>
> In .Net, usually we put one dll in one NuGet. That is to make minimum
> dependency unit. We can set up dependent NuGet, for example,
> Org.Apache.REEF.All NuGet depend all other REEF NuGets.
>
> I guess for .Net Evaluator, putting most code inside Common was to follow
> Java pattern. While Evaluator.exe must be in a independent project,
> therefore it is separated. I will look at it to see if it can be better
> grouped.
>
> Julia
>
> -----Original Message-----
> From: Markus Weimer [mailto:markus@weimo.de]
> Sent: Monday, November 23, 2015 10:14 AM
> To: dev@reef.incubator.apache.org
> Subject: Re: REEF driver info endpoint in reef-webserver for Java driver
> restarts
>
> On 2015-11-23 10:04, Julia Wang (QIUHE) wrote:
> > In Java, Driver, Evaluator and Task are all in reef-common, in .Net,
> > Evaluator code is in multiple places. What is the principle?
>
> I'm confused about the principle of forming / splitting assemblies in .NET
> as well. But I assume the Evaluator has to be an .exe and therefore has to
> be its own assembly? That of course does not solve the question of whether
> that exe is a thin launcher for code that is in a DLL.
>
> What's the usual best practice for forming DLLs and EXEs? Similarly,
> what's the commonly accepted relationship between assemblies and NuGets?
>
> In Java land, we generally strive to keep dependency chains apart to the
> extend possible. The main reason to split a module into two is if that
> creates smaller dependency trees for some users. As the REEF Evaluator and
> Driver share all of their dependencies, there is no value in splitting them
> into two modules, especially as they share much of the code.
>
> Markus
>
RE: REEF driver info endpoint in reef-webserver for Java driver
restarts
Posted by "Julia Wang (QIUHE)" <Qi...@microsoft.com>.
> What's the usual best practice for forming DLLs and EXEs? Similarly, what's the commonly accepted relationship between assemblies and NuGets?
In .Net, usually we put one dll in one NuGet. That is to make minimum dependency unit. We can set up dependent NuGet, for example, Org.Apache.REEF.All NuGet depend all other REEF NuGets.
I guess for .Net Evaluator, putting most code inside Common was to follow Java pattern. While Evaluator.exe must be in a independent project, therefore it is separated. I will look at it to see if it can be better grouped.
Julia
-----Original Message-----
From: Markus Weimer [mailto:markus@weimo.de]
Sent: Monday, November 23, 2015 10:14 AM
To: dev@reef.incubator.apache.org
Subject: Re: REEF driver info endpoint in reef-webserver for Java driver restarts
On 2015-11-23 10:04, Julia Wang (QIUHE) wrote:
> In Java, Driver, Evaluator and Task are all in reef-common, in .Net,
> Evaluator code is in multiple places. What is the principle?
I'm confused about the principle of forming / splitting assemblies in .NET as well. But I assume the Evaluator has to be an .exe and therefore has to be its own assembly? That of course does not solve the question of whether that exe is a thin launcher for code that is in a DLL.
What's the usual best practice for forming DLLs and EXEs? Similarly, what's the commonly accepted relationship between assemblies and NuGets?
In Java land, we generally strive to keep dependency chains apart to the extend possible. The main reason to split a module into two is if that creates smaller dependency trees for some users. As the REEF Evaluator and Driver share all of their dependencies, there is no value in splitting them into two modules, especially as they share much of the code.
Markus
Re: REEF driver info endpoint in reef-webserver for Java driver
restarts
Posted by Markus Weimer <ma...@weimo.de>.
On 2015-11-23 10:04, Julia Wang (QIUHE) wrote:
> In Java, Driver, Evaluator and Task are all in reef-common, in .Net,
> Evaluator code is in multiple places. What is the principle?
I'm confused about the principle of forming / splitting assemblies in
.NET as well. But I assume the Evaluator has to be an .exe and therefore
has to be its own assembly? That of course does not solve the question
of whether that exe is a thin launcher for code that is in a DLL.
What's the usual best practice for forming DLLs and EXEs? Similarly,
what's the commonly accepted relationship between assemblies and NuGets?
In Java land, we generally strive to keep dependency chains apart to the
extend possible. The main reason to split a module into two is if that
creates smaller dependency trees for some users. As the REEF Evaluator
and Driver share all of their dependencies, there is no value in
splitting them into two modules, especially as they share much of the code.
Markus
RE: REEF driver info endpoint in reef-webserver for Java driver
restarts
Posted by "Julia Wang (QIUHE)" <Qi...@microsoft.com>.
In Java, Driver, Evaluator and Task are all in reef-common, in .Net, Evaluator code is in multiple places. What is the principle?
-----Original Message-----
From: Andrew Chung [mailto:afchung90@gmail.com]
Sent: Friday, November 20, 2015 1:53 PM
To: dev@reef.incubator.apache.org
Subject: Re: REEF driver info endpoint in reef-webserver for Java driver restarts
Hmm, I actually just discovered that the Java REEF Evaluator implementation is actually in reef-common. Should we merge reef-common with reef-webserver? Is this still what we want?
Thanks,
Andrew
On Fri, Nov 20, 2015 at 1:50 PM, Andrew Chung <af...@gmail.com> wrote:
> Given that 2 days have passed, I will move on to have Evaluator depend
> on reef-webserver. Thanks!
>
> On Thu, Nov 19, 2015 at 11:03 AM, Julia Wang (QIUHE) <
> Qiuhe.Wang@microsoft.com> wrote:
>
>> Agree. Reef-webserver is at pretty lower dependent chain, just above
>> reef-common. It should be able to be referenced by evaluator.
>>
>> -----Original Message-----
>> From: Markus Weimer [mailto:markus@weimo.de]
>> Sent: Wednesday, November 18, 2015 6:42 PM
>> To: dev@reef.incubator.apache.org
>> Subject: Re: REEF driver info endpoint in reef-webserver for Java
>> driver restarts
>>
>> On 2015-11-18 16:17, Andrew Chung wrote:
>> > 2. Have Evaluator depend on reef-webserver.
>>
>> I'd be OK with this. The webserver is rather core to our
>> manageability story and I don't see a reason not to use it. Also, we
>> currently build one JAR for Driver and Evaluator anyhow, and
>> therefore not having this dependency won't have benefits in JAR size.
>>
>> Markus
>>
>
>
Re: REEF driver info endpoint in reef-webserver for Java driver restarts
Posted by Byung-Gon Chun <bg...@gmail.com>.
Probably we don't want to merge reef-common with reef-webserver.
On Sat, Nov 21, 2015 at 6:53 AM, Andrew Chung <af...@gmail.com> wrote:
> Hmm, I actually just discovered that the Java REEF Evaluator implementation
> is actually in reef-common. Should we merge reef-common with
> reef-webserver? Is this still what we want?
>
> Thanks,
> Andrew
>
> On Fri, Nov 20, 2015 at 1:50 PM, Andrew Chung <af...@gmail.com> wrote:
>
> > Given that 2 days have passed, I will move on to have Evaluator depend on
> > reef-webserver. Thanks!
> >
> > On Thu, Nov 19, 2015 at 11:03 AM, Julia Wang (QIUHE) <
> > Qiuhe.Wang@microsoft.com> wrote:
> >
> >> Agree. Reef-webserver is at pretty lower dependent chain, just above
> >> reef-common. It should be able to be referenced by evaluator.
> >>
> >> -----Original Message-----
> >> From: Markus Weimer [mailto:markus@weimo.de]
> >> Sent: Wednesday, November 18, 2015 6:42 PM
> >> To: dev@reef.incubator.apache.org
> >> Subject: Re: REEF driver info endpoint in reef-webserver for Java driver
> >> restarts
> >>
> >> On 2015-11-18 16:17, Andrew Chung wrote:
> >> > 2. Have Evaluator depend on reef-webserver.
> >>
> >> I'd be OK with this. The webserver is rather core to our manageability
> >> story and I don't see a reason not to use it. Also, we currently build
> one
> >> JAR for Driver and Evaluator anyhow, and therefore not having this
> >> dependency won't have benefits in JAR size.
> >>
> >> Markus
> >>
> >
> >
>
--
Byung-Gon Chun
Re: REEF driver info endpoint in reef-webserver for Java driver restarts
Posted by Andrew Chung <af...@gmail.com>.
Hmm, I actually just discovered that the Java REEF Evaluator implementation
is actually in reef-common. Should we merge reef-common with
reef-webserver? Is this still what we want?
Thanks,
Andrew
On Fri, Nov 20, 2015 at 1:50 PM, Andrew Chung <af...@gmail.com> wrote:
> Given that 2 days have passed, I will move on to have Evaluator depend on
> reef-webserver. Thanks!
>
> On Thu, Nov 19, 2015 at 11:03 AM, Julia Wang (QIUHE) <
> Qiuhe.Wang@microsoft.com> wrote:
>
>> Agree. Reef-webserver is at pretty lower dependent chain, just above
>> reef-common. It should be able to be referenced by evaluator.
>>
>> -----Original Message-----
>> From: Markus Weimer [mailto:markus@weimo.de]
>> Sent: Wednesday, November 18, 2015 6:42 PM
>> To: dev@reef.incubator.apache.org
>> Subject: Re: REEF driver info endpoint in reef-webserver for Java driver
>> restarts
>>
>> On 2015-11-18 16:17, Andrew Chung wrote:
>> > 2. Have Evaluator depend on reef-webserver.
>>
>> I'd be OK with this. The webserver is rather core to our manageability
>> story and I don't see a reason not to use it. Also, we currently build one
>> JAR for Driver and Evaluator anyhow, and therefore not having this
>> dependency won't have benefits in JAR size.
>>
>> Markus
>>
>
>
Re: REEF driver info endpoint in reef-webserver for Java driver restarts
Posted by Andrew Chung <af...@gmail.com>.
Given that 2 days have passed, I will move on to have Evaluator depend on
reef-webserver. Thanks!
On Thu, Nov 19, 2015 at 11:03 AM, Julia Wang (QIUHE) <
Qiuhe.Wang@microsoft.com> wrote:
> Agree. Reef-webserver is at pretty lower dependent chain, just above
> reef-common. It should be able to be referenced by evaluator.
>
> -----Original Message-----
> From: Markus Weimer [mailto:markus@weimo.de]
> Sent: Wednesday, November 18, 2015 6:42 PM
> To: dev@reef.incubator.apache.org
> Subject: Re: REEF driver info endpoint in reef-webserver for Java driver
> restarts
>
> On 2015-11-18 16:17, Andrew Chung wrote:
> > 2. Have Evaluator depend on reef-webserver.
>
> I'd be OK with this. The webserver is rather core to our manageability
> story and I don't see a reason not to use it. Also, we currently build one
> JAR for Driver and Evaluator anyhow, and therefore not having this
> dependency won't have benefits in JAR size.
>
> Markus
>
RE: REEF driver info endpoint in reef-webserver for Java driver
restarts
Posted by "Julia Wang (QIUHE)" <Qi...@microsoft.com>.
Agree. Reef-webserver is at pretty lower dependent chain, just above reef-common. It should be able to be referenced by evaluator.
-----Original Message-----
From: Markus Weimer [mailto:markus@weimo.de]
Sent: Wednesday, November 18, 2015 6:42 PM
To: dev@reef.incubator.apache.org
Subject: Re: REEF driver info endpoint in reef-webserver for Java driver restarts
On 2015-11-18 16:17, Andrew Chung wrote:
> 2. Have Evaluator depend on reef-webserver.
I'd be OK with this. The webserver is rather core to our manageability story and I don't see a reason not to use it. Also, we currently build one JAR for Driver and Evaluator anyhow, and therefore not having this dependency won't have benefits in JAR size.
Markus
Re: REEF driver info endpoint in reef-webserver for Java driver
restarts
Posted by Markus Weimer <ma...@weimo.de>.
On 2015-11-18 16:17, Andrew Chung wrote:
> 2. Have Evaluator depend on reef-webserver.
I'd be OK with this. The webserver is rather core to our manageability
story and I don't see a reason not to use it. Also, we currently build
one JAR for Driver and Evaluator anyhow, and therefore not having this
dependency won't have benefits in JAR size.
Markus