You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@reef.apache.org by Markus Weimer <ma...@weimo.de> on 2017/10/30 19:55:18 UTC
REEF.NET: Canonical way to know that a Driver is done?
Hi,
I am working on REEF-1271, a new test framework for REEF. One stumbling
block I have: How do I know for sure when a Driver is done? Ideally, I'd
want to do this in a way which doesn't have to know on which runtime the
Driver is running (YARN, local, ...).
Thanks,
Markus
Re: REEF.NET: Canonical way to know that a Driver is done?
Posted by Markus Weimer <ma...@weimo.de>.
I found a way to make this work:
https://github.com/apache/reef/pull/1408
Markus
On Mon, Oct 30, 2017 at 4:56 PM, Markus Weimer <ma...@weimo.de> wrote:
> On Mon, Oct 30, 2017 at 2:55 PM, Julia Wang (QIUHE) <
> Qiuhe.Wang@microsoft.com.invalid> wrote:
>
>> " hanging request" - evaluator request or driver not able to shut down?
>>
>
> I meant a hanging HTTP request from the Client to the Driver. That way,
> the Driver can effectively raise an event with the Client.
>
> I filled REEF-1942 to track this:
>
> https://issues.apache.org/jira/browse/REEF-1942
>
> Markus
>
Re: REEF.NET: Canonical way to know that a Driver is done?
Posted by Markus Weimer <ma...@weimo.de>.
On Mon, Oct 30, 2017 at 2:55 PM, Julia Wang (QIUHE) <
Qiuhe.Wang@microsoft.com.invalid> wrote:
> " hanging request" - evaluator request or driver not able to shut down?
>
I meant a hanging HTTP request from the Client to the Driver. That way, the
Driver can effectively raise an event with the Client.
I filled REEF-1942 to track this:
https://issues.apache.org/jira/browse/REEF-1942
Markus
RE: REEF.NET: Canonical way to know that a Driver is done?
Posted by "Julia Wang (QIUHE)" <Qi...@microsoft.com.INVALID>.
" hanging request" - evaluator request or driver not able to shut down?
Julia
-----Original Message-----
From: Markus Weimer [mailto:markus@weimo.de]
Sent: Monday, October 30, 2017 2:35 PM
To: REEF Developers Mailinglist <de...@reef.apache.org>
Subject: Re: REEF.NET: Canonical way to know that a Driver is done?
On Mon, Oct 30, 2017 at 2:23 PM, Julia Wang (QIUHE) < Qiuhe.Wang@microsoft.com.invalid> wrote:
> Not aware of.
Bummer. I am thinking of making one which uses the Driver's HTTP endpoint on a hanging request. The request will be answered by the Driver as part of the DriverStop event handler. Does this make sense? It is a bit of a hack, but could work. Trouble is that REEF.NET doesn't have the DriverStop message. Hence, this HTTP endpoint needs to be implemented in Java.
WDYT?
If we had such an API, the client may not have time to call it before the
> driver shut down I guess.
>
In Java, we use an Event for this on the client. But even if not, we could always cache the outcome in the client.
Markus
Re: REEF.NET: Canonical way to know that a Driver is done?
Posted by Markus Weimer <ma...@weimo.de>.
On Mon, Oct 30, 2017 at 2:23 PM, Julia Wang (QIUHE) <
Qiuhe.Wang@microsoft.com.invalid> wrote:
> Not aware of.
Bummer. I am thinking of making one which uses the Driver's HTTP endpoint
on a hanging request. The request will be answered by the Driver as part of
the DriverStop event handler. Does this make sense? It is a bit of a hack,
but could work. Trouble is that REEF.NET doesn't have the DriverStop
message. Hence, this HTTP endpoint needs to be implemented in Java.
WDYT?
If we had such an API, the client may not have time to call it before the
> driver shut down I guess.
>
In Java, we use an Event for this on the client. But even if not, we could
always cache the outcome in the client.
Markus
RE: REEF.NET: Canonical way to know that a Driver is done?
Posted by "Julia Wang (QIUHE)" <Qi...@microsoft.com.INVALID>.
Not aware of. If we had such an API, the client may not have time to call it before the driver shut down I guess.
Julia
-----Original Message-----
From: Markus Weimer [mailto:markus@weimo.de]
Sent: Monday, October 30, 2017 2:11 PM
To: REEF Developers Mailinglist <de...@reef.apache.org>
Subject: Re: REEF.NET: Canonical way to know that a Driver is done?
On Mon, Oct 30, 2017 at 12:57 PM, Julia Wang (QIUHE) < Qiuhe.Wang@microsoft.com.invalid> wrote:
> Looking for log like below:
>
> INFO reef.runtime.common.driver.DriverStatusManager.onComplete
> Thread-13
> | Clean shutdown of the Driver.
>
Thanks! However, I am looking for a clean API to determine Driver shutdown.
Something like the `JobCompleted` event in REEF Java. Do we have such an API?
Thanks,
Markus
Re: REEF.NET: Canonical way to know that a Driver is done?
Posted by Markus Weimer <ma...@weimo.de>.
On Mon, Oct 30, 2017 at 12:57 PM, Julia Wang (QIUHE) <
Qiuhe.Wang@microsoft.com.invalid> wrote:
> Looking for log like below:
>
> INFO reef.runtime.common.driver.DriverStatusManager.onComplete Thread-13
> | Clean shutdown of the Driver.
>
Thanks! However, I am looking for a clean API to determine Driver shutdown.
Something like the `JobCompleted` event in REEF Java. Do we have such an
API?
Thanks,
Markus
RE: REEF.NET: Canonical way to know that a Driver is done?
Posted by "Julia Wang (QIUHE)" <Qi...@microsoft.com.INVALID>.
Looking for log like below:
INFO reef.runtime.common.driver.DriverStatusManager.onComplete Thread-13 | Clean shutdown of the Driver.
-----Original Message-----
From: Markus Weimer [mailto:markus@weimo.de]
Sent: Monday, October 30, 2017 12:55 PM
To: REEF Developers Mailinglist <de...@reef.apache.org>
Subject: REEF.NET: Canonical way to know that a Driver is done?
Hi,
I am working on REEF-1271, a new test framework for REEF. One stumbling block I have: How do I know for sure when a Driver is done? Ideally, I'd want to do this in a way which doesn't have to know on which runtime the Driver is running (YARN, local, ...).
Thanks,
Markus