You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@reef.apache.org by Andrew Chung <af...@gmail.com> on 2015/10/29 19:30:08 UTC
Gauging interest in a "DriverTerminator"
Hi REEF devs,
This came up in a recent chat with Beysim.
Currently, REEF terminates when there are no more outstanding requests for
an Evaluator and when there are no more running Tasks.
There are certain scenarios where the driver can be idle for an extended
period of time - for instance we can imagine the Driver as a server that
takes incoming requests to run certain tasks. On startup, before no
requests arrive, the driver can be idle for a while. For now, in order for
the user to keep the Driver alive, they will either have to run an "empty
task" or they will need to make repeated timer calls, something that is not
entirely intuitive.
To support this scenario, we can have a "DriverTerminator" that returns a
class, which:
1. Allows the user to specify whether or not the Driver should be
terminated if it is idle
2. Allows a String to be returned to RM as a diagnostics message on
termination
How does this sound?
Thanks,
Andrew
Re: Gauging interest in a "DriverTerminator"
Posted by Andrew Chung <af...@gmail.com>.
Hi Julia,
The DriverTerminator is not used to terminate the Driver, rather it is to
determine if the Driver should be terminated when nothing is "active."
Perhaps the name is kind of misleading.
Yes, I am aware that Evaluators don't have to be closed when the tasks have
finished, but this is precisely what we are trying to avoid - empty
evaluators/tasks that don't do anything other than to keep the driver alive.
The DriverTerminator will be user-injected, same as ProgressProvider.
Thanks,
Andrew
On Thu, Oct 29, 2015 at 11:49 AM, Julia Wang (QIUHE) <
Qiuhe.Wang@microsoft.com> wrote:
> When tasks are completed, if the user would like to keep driver running,
> he can retain the evaluator as long as the context is not closed. This can
> be controlled in client's driver.
>
> Client can also terminate driver through http request. There are some
> implementation in HttpServerReefEventHander, we can extend those commands
> there or client can also implement those logic in their own http handlers.
>
> How will the DriverTerminator be hooked to the driver?
>
> Thanks,
> Julia
>
> -----Original Message-----
> From: Byung-Gon Chun [mailto:bgchun@gmail.com]
> Sent: Thursday, October 29, 2015 11:33 AM
> To: dev@reef.incubator.apache.org
> Subject: Re: Gauging interest in a "DriverTerminator"
>
> Hi Andrew,
>
> Sounds great.
> I like the idea. Presumably we can use the current termination condition
> as the default DriverTerminator implementation.
>
> Thanks,
> Gon
>
>
> On Fri, Oct 30, 2015 at 3:30 AM, Andrew Chung <af...@gmail.com> wrote:
>
> > Hi REEF devs,
> >
> > This came up in a recent chat with Beysim.
> >
> > Currently, REEF terminates when there are no more outstanding requests
> > for an Evaluator and when there are no more running Tasks.
> > There are certain scenarios where the driver can be idle for an
> > extended period of time - for instance we can imagine the Driver as a
> > server that takes incoming requests to run certain tasks. On startup,
> > before no requests arrive, the driver can be idle for a while. For
> > now, in order for the user to keep the Driver alive, they will either
> > have to run an "empty task" or they will need to make repeated timer
> > calls, something that is not entirely intuitive.
> >
> > To support this scenario, we can have a "DriverTerminator" that
> > returns a class, which:
> > 1. Allows the user to specify whether or not the Driver should be
> > terminated if it is idle 2. Allows a String to be returned to RM as a
> > diagnostics message on termination
> >
> > How does this sound?
> >
> > Thanks,
> > Andrew
> >
>
>
>
> --
> Byung-Gon Chun
>
RE: Gauging interest in a "DriverTerminator"
Posted by "Julia Wang (QIUHE)" <Qi...@microsoft.com>.
When tasks are completed, if the user would like to keep driver running, he can retain the evaluator as long as the context is not closed. This can be controlled in client's driver.
Client can also terminate driver through http request. There are some implementation in HttpServerReefEventHander, we can extend those commands there or client can also implement those logic in their own http handlers.
How will the DriverTerminator be hooked to the driver?
Thanks,
Julia
-----Original Message-----
From: Byung-Gon Chun [mailto:bgchun@gmail.com]
Sent: Thursday, October 29, 2015 11:33 AM
To: dev@reef.incubator.apache.org
Subject: Re: Gauging interest in a "DriverTerminator"
Hi Andrew,
Sounds great.
I like the idea. Presumably we can use the current termination condition as the default DriverTerminator implementation.
Thanks,
Gon
On Fri, Oct 30, 2015 at 3:30 AM, Andrew Chung <af...@gmail.com> wrote:
> Hi REEF devs,
>
> This came up in a recent chat with Beysim.
>
> Currently, REEF terminates when there are no more outstanding requests
> for an Evaluator and when there are no more running Tasks.
> There are certain scenarios where the driver can be idle for an
> extended period of time - for instance we can imagine the Driver as a
> server that takes incoming requests to run certain tasks. On startup,
> before no requests arrive, the driver can be idle for a while. For
> now, in order for the user to keep the Driver alive, they will either
> have to run an "empty task" or they will need to make repeated timer
> calls, something that is not entirely intuitive.
>
> To support this scenario, we can have a "DriverTerminator" that
> returns a class, which:
> 1. Allows the user to specify whether or not the Driver should be
> terminated if it is idle 2. Allows a String to be returned to RM as a
> diagnostics message on termination
>
> How does this sound?
>
> Thanks,
> Andrew
>
--
Byung-Gon Chun
Re: Gauging interest in a "DriverTerminator"
Posted by Byung-Gon Chun <bg...@gmail.com>.
Hi Andrew,
Sounds great.
I like the idea. Presumably we can use the current termination condition as
the default DriverTerminator implementation.
Thanks,
Gon
On Fri, Oct 30, 2015 at 3:30 AM, Andrew Chung <af...@gmail.com> wrote:
> Hi REEF devs,
>
> This came up in a recent chat with Beysim.
>
> Currently, REEF terminates when there are no more outstanding requests for
> an Evaluator and when there are no more running Tasks.
> There are certain scenarios where the driver can be idle for an extended
> period of time - for instance we can imagine the Driver as a server that
> takes incoming requests to run certain tasks. On startup, before no
> requests arrive, the driver can be idle for a while. For now, in order for
> the user to keep the Driver alive, they will either have to run an "empty
> task" or they will need to make repeated timer calls, something that is not
> entirely intuitive.
>
> To support this scenario, we can have a "DriverTerminator" that returns a
> class, which:
> 1. Allows the user to specify whether or not the Driver should be
> terminated if it is idle
> 2. Allows a String to be returned to RM as a diagnostics message on
> termination
>
> How does this sound?
>
> Thanks,
> Andrew
>
--
Byung-Gon Chun
Re: Gauging interest in a "DriverTerminator"
Posted by Markus Weimer <ma...@weimo.de>.
Hi,
this looks good! As we move forward on this, it would be good to
understand the relationship with the `DriverIdlenessSource` interface
that is used for that right now. I can see how goal 1 below could be
implemented using just that, but not the second goal.
Markus
On 2015-10-29 19:30, Andrew Chung wrote:
> Hi REEF devs,
>
> This came up in a recent chat with Beysim.
>
> Currently, REEF terminates when there are no more outstanding requests for
> an Evaluator and when there are no more running Tasks.
> There are certain scenarios where the driver can be idle for an extended
> period of time - for instance we can imagine the Driver as a server that
> takes incoming requests to run certain tasks. On startup, before no
> requests arrive, the driver can be idle for a while. For now, in order for
> the user to keep the Driver alive, they will either have to run an "empty
> task" or they will need to make repeated timer calls, something that is not
> entirely intuitive.
>
> To support this scenario, we can have a "DriverTerminator" that returns a
> class, which:
> 1. Allows the user to specify whether or not the Driver should be
> terminated if it is idle
> 2. Allows a String to be returned to RM as a diagnostics message on
> termination
>
> How does this sound?
>
> Thanks,
> Andrew
>