You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@reef.apache.org by Dhruv Mahajan <dh...@gmail.com> on 2016/06/08 04:34:27 UTC
how to send message from task to driver
Hi
So I want to implement the following semantics:
a) Task does something and then sends message to driver that it is done and
then waits on a semaphore or something like that.
b) Driver receives the message, get the messages from all the evaluators
and then use Dispose() call to send messages to each task.
c) Upon receiving ICloseEvent, each task resumes itself and then exits.
What is the handler to send the message from task to driver?
Dhruv
Re: how to send message from task to driver
Posted by Markus Weimer <ma...@weimo.de>.
On 2016-06-08 10:22, Dhruv Mahajan wrote:
> Just to give u an example, consider a parameter server where we have many
> workers but we still want to control the degree of parallelism. With
> Zookeeper it will be extremely easy to do this dynamically.
Yes. We can also use Zookeeper to keep state we don't want to loose upon
individual machine failure. If we spread Zookeeper over a couple of
Evaluators, any one (or more) machine can fail, including the Driver.
Markus
Re: how to send message from task to driver
Posted by Dhruv Mahajan <dh...@gmail.com>.
Just to give u an example, consider a parameter server where we have many
workers but we still want to control the degree of parallelism. With
Zookeeper it will be extremely easy to do this dynamically.
Dhruv
On Wed, Jun 8, 2016 at 10:21 AM, Markus Weimer <ma...@weimo.de> wrote:
> On 2016-06-08 10:15, Dhruv Mahajan wrote:
>
>> Agreed...But since you mentioned it, hundreds of scenarios ran in my mind
>> in ML where this will make life much easier. So if its not too much an
>> effort I am inclined for its REEF integration in very near future.
>>
>
> +1 on that. If we can find the time, having ZooKeeper as a service run on
> a bunch of Evaluators has been a long-standing dream :)
>
> Markus
>
>
Re: how to send message from task to driver
Posted by Markus Weimer <ma...@weimo.de>.
On 2016-06-08 10:15, Dhruv Mahajan wrote:
> Agreed...But since you mentioned it, hundreds of scenarios ran in my mind
> in ML where this will make life much easier. So if its not too much an
> effort I am inclined for its REEF integration in very near future.
+1 on that. If we can find the time, having ZooKeeper as a service run
on a bunch of Evaluators has been a long-standing dream :)
Markus
Re: how to send message from task to driver
Posted by Dhruv Mahajan <dh...@gmail.com>.
Agreed...But since you mentioned it, hundreds of scenarios ran in my mind
in ML where this will make life much easier. So if its not too much an
effort I am inclined for its REEF integration in very near future.
Dhruv
On Wed, Jun 8, 2016 at 10:13 AM, Markus Weimer <ma...@weimo.de> wrote:
> On 2016-06-08 10:08, Dhruv Mahajan wrote:
>
>> Apache Zookeeper is a good idea. If I am right, its sort of shared state
>> management kind of service? However, should we do it for REEF-1407 or
>> later?
>>
>
> Later. The way memory sizes are trending on a per-server basis, many
> meaningful IMRU workloads can be done on dozens of servers, not thousands.
> So it isn't a pressing need for REEF-1407 :)
>
> Markus
>
>
Re: how to send message from task to driver
Posted by Markus Weimer <ma...@weimo.de>.
On 2016-06-08 10:08, Dhruv Mahajan wrote:
> Apache Zookeeper is a good idea. If I am right, its sort of shared state
> management kind of service? However, should we do it for REEF-1407 or later?
Later. The way memory sizes are trending on a per-server basis, many
meaningful IMRU workloads can be done on dozens of servers, not
thousands. So it isn't a pressing need for REEF-1407 :)
Markus
Re: how to send message from task to driver
Posted by Dhruv Mahajan <dh...@gmail.com>.
Apache Zookeeper is a good idea. If I am right, its sort of shared state
management kind of service? However, should we do it for REEF-1407 or later?
On Wed, Jun 8, 2016 at 10:04 AM, Markus Weimer <ma...@weimo.de> wrote:
> On 2016-06-08 09:13, Dhruv Mahajan wrote:
>
>> in general I agree that if we can introduce an extra push based
>> handler for task for this that would be good.
>>
>
> The challenge with that is the potential to overwhelm the Driver. If we
> have 1000s of Tasks all doing that at the same time, the Driver will be the
> bottleneck. This isn't an issue for a few dozens of Tasks, though.
>
> For very large deployments, we might want to look into Apache ZooKeeper to
> derive consensus amongst the Tasks.
>
> Markus
>
Re: how to send message from task to driver
Posted by Dhruv Mahajan <dh...@gmail.com>.
But I believe, in driver Default logging handler was already bounded for
this right? so if the introduced message handling is also cheaper does it
matter?
Dhruv
On Wed, Jun 8, 2016 at 10:04 AM, Markus Weimer <ma...@weimo.de> wrote:
> On 2016-06-08 09:13, Dhruv Mahajan wrote:
>
>> in general I agree that if we can introduce an extra push based
>> handler for task for this that would be good.
>>
>
> The challenge with that is the potential to overwhelm the Driver. If we
> have 1000s of Tasks all doing that at the same time, the Driver will be the
> bottleneck. This isn't an issue for a few dozens of Tasks, though.
>
> For very large deployments, we might want to look into Apache ZooKeeper to
> derive consensus amongst the Tasks.
>
> Markus
>
Re: how to send message from task to driver
Posted by Markus Weimer <ma...@weimo.de>.
On 2016-06-08 09:13, Dhruv Mahajan wrote:
> in general I agree that if we can introduce an extra push based
> handler for task for this that would be good.
The challenge with that is the potential to overwhelm the Driver. If we
have 1000s of Tasks all doing that at the same time, the Driver will be
the bottleneck. This isn't an issue for a few dozens of Tasks, though.
For very large deployments, we might want to look into Apache ZooKeeper
to derive consensus amongst the Tasks.
Markus
Re: how to send message from task to driver
Posted by Dhruv Mahajan <dh...@gmail.com>.
Hi
I am planning to use it only to stop all tasks in a synchronous fashion so
shud be ok. However, in general I agree that if we can introduce an extra
push based handler for task for this that would be good.
Dhruv
On Wed, Jun 8, 2016 at 7:55 AM, Markus Weimer <ma...@weimo.de> wrote:
>
> On 2016-06-07 22:22, Dhruv Mahajan wrote:
>
>> Ok on further looking, on Driver side I need to register,
>> IObserver<ITaskMessage> and on task side it is ITaskMessageSource.
>> Periodically, with the heartbeats these messages will be read from
>> task and sent to the driver.
>>
>> Can somebody confirm this?
>>
>
> Confirmed. Note that this path is beholden to the heartbeat interval. If
> you need faster push notifications, you either need to use Wake or HTTP
> directly or we have to address this with a REEF extension. It is a
> somewhat frequent request.
>
> Markus
>
Re: how to send message from task to driver
Posted by Markus Weimer <ma...@weimo.de>.
On 2016-06-07 22:22, Dhruv Mahajan wrote:
> Ok on further looking, on Driver side I need to register,
> IObserver<ITaskMessage> and on task side it is ITaskMessageSource.
> Periodically, with the heartbeats these messages will be read from
> task and sent to the driver.
>
> Can somebody confirm this?
Confirmed. Note that this path is beholden to the heartbeat interval. If
you need faster push notifications, you either need to use Wake or HTTP
directly or we have to address this with a REEF extension. It is a
somewhat frequent request.
Markus
Re: how to send message from task to driver
Posted by Dhruv Mahajan <dh...@gmail.com>.
It does not which makes things even simpler :) . Currently, I think once
*Host tasks were done they were exiting. So the task could exit because: a)
its completed b) some failure happened c) driver asked it. b) we cannot
avoid, but now we are saying that c) is the only way.
Dhruv
On Wed, Jun 8, 2016 at 10:02 AM, Markus Weimer <ma...@weimo.de> wrote:
> On 2016-06-08 09:59, Dhruv Mahajan wrote:
>
>> I looked at TaskManager code and there we are calling Dispose to call
>> the ICloseEvent on the Task Side. We should simply extend it by
>> sending another "completed" sort of message and extending the FT to
>> consider it.
>>
>
> `IRunningTask.Dispose()` can take a `byte[]` to convey such messages for
> different kinds of shutdown. However, what difference does it make to
> the IMRU tasks *why* they are being shutdown?
>
> Markus
>
>
Re: how to send message from task to driver
Posted by Markus Weimer <ma...@weimo.de>.
On 2016-06-08 09:59, Dhruv Mahajan wrote:
> I looked at TaskManager code and there we are calling Dispose to call
> the ICloseEvent on the Task Side. We should simply extend it by
> sending another "completed" sort of message and extending the FT to
> consider it.
`IRunningTask.Dispose()` can take a `byte[]` to convey such messages for
different kinds of shutdown. However, what difference does it make to
the IMRU tasks *why* they are being shutdown?
Markus
Re: how to send message from task to driver
Posted by Dhruv Mahajan <dh...@gmail.com>.
Hi Julia
I looked at TaskManager code and there we are calling Dispose to call the
ICloseEvent on the Task Side. We should simply extend it by sending another
"completed" sort of message and extending the FT to consider it.
Dhruv
On Wed, Jun 8, 2016 at 12:53 AM, Julia Wang (QIUHE) <
Qiuhe.Wang@microsoft.com> wrote:
> The handlers work. However, for successful scenario, we cannot use
> Dispose to stop the task in this case because it would conflict with fault
> tolerant design. I have put some comments in the Jira and let's continue
> the discussion there.
>
> -----Original Message-----
> From: Dhruv Mahajan [mailto:dhruv.mahajan@gmail.com]
> Sent: Tuesday, June 7, 2016 10:23 PM
> To: dev@reef.apache.org
> Subject: Re: how to send message from task to driver
>
> Ok on further looking, on Driver side I need to register,
> IObserver<ITaskMessage> and on task side it is ITaskMessageSource.
> Periodically, with the heartbeats these messages will be read from task
> and sent to the driver.
>
> Can somebody confirm this?
> And now an important question: Is this part of code very well tested :) .
> How confident are we? :)
>
> Dhruv
>
> On Tue, Jun 7, 2016 at 9:34 PM, Dhruv Mahajan <dh...@gmail.com>
> wrote:
>
> > Hi
> > So I want to implement the following semantics:
> >
> > a) Task does something and then sends message to driver that it is
> > done and then waits on a semaphore or something like that.
> > b) Driver receives the message, get the messages from all the
> > evaluators and then use Dispose() call to send messages to each task.
> > c) Upon receiving ICloseEvent, each task resumes itself and then exits.
> >
> > What is the handler to send the message from task to driver?
> >
> > Dhruv
> >
>
Re: how to send message from task to driver
Posted by Dhruv Mahajan <dh...@gmail.com>.
Yes, that's what I am using. It also combines well with the current IMRU FT
changes since this handler is already bounded, and we just need to sen an
extra "completed" message form driver to these tasks.
Dhruv
On Wed, Jun 8, 2016 at 10:00 AM, Markus Weimer <ma...@weimo.de> wrote:
> On 2016-06-08 09:08, Dhruv Mahajan wrote:
>
>> So i an ICloseEvent is associated with Task and driver wants to call it,
>> what is the function call?
>>
>
> `IRunningTask.Dispose()`.
>
> Markus
>
Re: how to send message from task to driver
Posted by Markus Weimer <ma...@weimo.de>.
On 2016-06-08 09:08, Dhruv Mahajan wrote:
> So i an ICloseEvent is associated with Task and driver wants to call it,
> what is the function call?
`IRunningTask.Dispose()`.
Markus
Re: how to send message from task to driver
Posted by Dhruv Mahajan <dh...@gmail.com>.
So i an ICloseEvent is associated with Task and driver wants to call it,
what is the function call?
Dhruv
On Jun 8, 2016 7:57 AM, "Markus Weimer" <ma...@weimo.de> wrote:
> On 2016-06-08 00:53, Julia Wang (QIUHE) wrote:
>
>> However, for successful scenario, we cannot use Dispose to stop the
>> task in this case because it would conflict with fault tolerant
>> design.
>>
>
> When is a Task ever stopped by calling `.Dispose()`? At least on the Java
> side, we always use the stop handler for this.
>
> Markus
>
Re: how to send message from task to driver
Posted by Markus Weimer <ma...@weimo.de>.
On 2016-06-08 00:53, Julia Wang (QIUHE) wrote:
> However, for successful scenario, we cannot use Dispose to stop the
> task in this case because it would conflict with fault tolerant
> design.
When is a Task ever stopped by calling `.Dispose()`? At least on the
Java side, we always use the stop handler for this.
Markus
RE: how to send message from task to driver
Posted by "Julia Wang (QIUHE)" <Qi...@microsoft.com>.
The handlers work. However, for successful scenario, we cannot use Dispose to stop the task in this case because it would conflict with fault tolerant design. I have put some comments in the Jira and let's continue the discussion there.
-----Original Message-----
From: Dhruv Mahajan [mailto:dhruv.mahajan@gmail.com]
Sent: Tuesday, June 7, 2016 10:23 PM
To: dev@reef.apache.org
Subject: Re: how to send message from task to driver
Ok on further looking, on Driver side I need to register, IObserver<ITaskMessage> and on task side it is ITaskMessageSource.
Periodically, with the heartbeats these messages will be read from task and sent to the driver.
Can somebody confirm this?
And now an important question: Is this part of code very well tested :) .
How confident are we? :)
Dhruv
On Tue, Jun 7, 2016 at 9:34 PM, Dhruv Mahajan <dh...@gmail.com>
wrote:
> Hi
> So I want to implement the following semantics:
>
> a) Task does something and then sends message to driver that it is
> done and then waits on a semaphore or something like that.
> b) Driver receives the message, get the messages from all the
> evaluators and then use Dispose() call to send messages to each task.
> c) Upon receiving ICloseEvent, each task resumes itself and then exits.
>
> What is the handler to send the message from task to driver?
>
> Dhruv
>
Re: how to send message from task to driver
Posted by Dhruv Mahajan <dh...@gmail.com>.
Ok on further looking, on Driver side I need to register,
IObserver<ITaskMessage> and on task side it is ITaskMessageSource.
Periodically, with the heartbeats these messages will be read from task and
sent to the driver.
Can somebody confirm this?
And now an important question: Is this part of code very well tested :) .
How confident are we? :)
Dhruv
On Tue, Jun 7, 2016 at 9:34 PM, Dhruv Mahajan <dh...@gmail.com>
wrote:
> Hi
> So I want to implement the following semantics:
>
> a) Task does something and then sends message to driver that it is done
> and then waits on a semaphore or something like that.
> b) Driver receives the message, get the messages from all the evaluators
> and then use Dispose() call to send messages to each task.
> c) Upon receiving ICloseEvent, each task resumes itself and then exits.
>
> What is the handler to send the message from task to driver?
>
> Dhruv
>