You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@reef.apache.org by Byung-Gon Chun <bg...@gmail.com> on 2014/11/01 03:43:41 UTC

Re: reef examples

About Evaluator-to-Driver communication, there's need to push TaskMessage's
immediately instead of tagging it on a heartbeat. This is from our
experience on building Surf, an in-memory caching service on REEF.
Although the default should use the heartbeat mechanism, we should provide
an option to push a TaskMessage. Otherwise, we have to set up a separate
channel between Evaluator and Driver.




On Thu, Oct 30, 2014 at 3:45 AM, Markus Weimer <ma...@weimo.de> wrote:

> Hi,
>
>  A. Is heartbeating inherent to WAKE and automatically enabled on driver?
>> Does it require configuration (like interval length)?
>>
> REEF always sets up the communications between Evaluators and the Driver.
> The heartbeat mechanism is configurable somewhere, but we don't really
> expose that at this point (we should, though).
>
>  B. Is heartbeating taking care of all driver/task handlers including
>> ON_SEND_MESSAGE? Why are call(), getMessage() become synchronized with
>> task/driver messaging?
>>
>
> This is where a lot of confusion enters the picture. So allow me to be
> verbose.
>
> Driver-to-Evaluator communication is always push based. That is, calling
> RunningTask.send() will immediately proceed to send that message to the
> Task in question. On the Evaluator side, the message is received and routed
> to the handler configured in TaskConfiguration.ON_MESSAGE.
>
> Evaluator-to-Driver communication is heartbeat based, for the most part
> (I'll talk about the exceptions to this below). When you want to send
> something from the Task to the Driver, you register a handler with
> TaskConfiguration.ON_SEND_MESSAGE. This handler will be invoked by the
> Evaluator whenever a heartbeat is about to be sent. In other words: The
> Task has no way of initiating the communication. This seems (and in fact
> is) annoying, but it serves a purpose: When dealing with thousands of
> Tasks, a single Driver can easily be overwhelmed if they all choose to send
> large messages at once, leading to congestion and ultimately poor
> performance. Allowing the framework to be in charge of the sending allows
> us to pace things just right. Ideally, we'd have a dynamic heartbeat
> interval that takes the current load on the Driver into account, for
> instance. We currently don't have that, though (contributions welcome ;-)).
>
> Now the exception to that setup: State changes of an Evaluator are in fact
> pushed. That is: When e.g. a Task finishes, we immediately send a message
> to the Driver. This drastically reduces the latency compared to pull based
> control flows such as the one found in Hadoop 1. It is safe, because we
> still control the size and timing of these messages.
>
> Markus
>



-- 
Byung-Gon Chun

Re: reef examples

Posted by Byung-Gon Chun <bg...@gmail.com>.
Hi Markus,

I will open a JIRA about this issue. I think a clean way is to provide a
TaskMessage push interface in REEF as an option to reuse the RemoteManager
REEF uses.

Thanks.
-Gon

On Sun, Nov 2, 2014 at 9:06 AM, Markus Weimer <ma...@weimo.de> wrote:

> Hi,
>
> agreed. This comes up too often to ignore. There is a mechanism today using
> the RemoteManager interface. How shall we go beyond that? Can you write up
> your needs as a JIRA such that we can structure the discussion?
>
> Thanks,
>
> Markus
>
> On Fri, Oct 31, 2014 at 7:43 PM, Byung-Gon Chun <bg...@gmail.com> wrote:
>
> > About Evaluator-to-Driver communication, there's need to push
> TaskMessage's
> > immediately instead of tagging it on a heartbeat. This is from our
> > experience on building Surf, an in-memory caching service on REEF.
> > Although the default should use the heartbeat mechanism, we should
> provide
> > an option to push a TaskMessage. Otherwise, we have to set up a separate
> > channel between Evaluator and Driver.
> >
> >
> >
> >
> > On Thu, Oct 30, 2014 at 3:45 AM, Markus Weimer <ma...@weimo.de> wrote:
> >
> > > Hi,
> > >
> > >  A. Is heartbeating inherent to WAKE and automatically enabled on
> driver?
> > >> Does it require configuration (like interval length)?
> > >>
> > > REEF always sets up the communications between Evaluators and the
> Driver.
> > > The heartbeat mechanism is configurable somewhere, but we don't really
> > > expose that at this point (we should, though).
> > >
> > >  B. Is heartbeating taking care of all driver/task handlers including
> > >> ON_SEND_MESSAGE? Why are call(), getMessage() become synchronized with
> > >> task/driver messaging?
> > >>
> > >
> > > This is where a lot of confusion enters the picture. So allow me to be
> > > verbose.
> > >
> > > Driver-to-Evaluator communication is always push based. That is,
> calling
> > > RunningTask.send() will immediately proceed to send that message to the
> > > Task in question. On the Evaluator side, the message is received and
> > routed
> > > to the handler configured in TaskConfiguration.ON_MESSAGE.
> > >
> > > Evaluator-to-Driver communication is heartbeat based, for the most part
> > > (I'll talk about the exceptions to this below). When you want to send
> > > something from the Task to the Driver, you register a handler with
> > > TaskConfiguration.ON_SEND_MESSAGE. This handler will be invoked by the
> > > Evaluator whenever a heartbeat is about to be sent. In other words: The
> > > Task has no way of initiating the communication. This seems (and in
> fact
> > > is) annoying, but it serves a purpose: When dealing with thousands of
> > > Tasks, a single Driver can easily be overwhelmed if they all choose to
> > send
> > > large messages at once, leading to congestion and ultimately poor
> > > performance. Allowing the framework to be in charge of the sending
> allows
> > > us to pace things just right. Ideally, we'd have a dynamic heartbeat
> > > interval that takes the current load on the Driver into account, for
> > > instance. We currently don't have that, though (contributions welcome
> > ;-)).
> > >
> > > Now the exception to that setup: State changes of an Evaluator are in
> > fact
> > > pushed. That is: When e.g. a Task finishes, we immediately send a
> message
> > > to the Driver. This drastically reduces the latency compared to pull
> > based
> > > control flows such as the one found in Hadoop 1. It is safe, because we
> > > still control the size and timing of these messages.
> > >
> > > Markus
> > >
> >
> >
> >
> > --
> > Byung-Gon Chun
> >
>



-- 
Byung-Gon Chun

Re: reef examples

Posted by Markus Weimer <ma...@weimo.de>.
Hi,

agreed. This comes up too often to ignore. There is a mechanism today using
the RemoteManager interface. How shall we go beyond that? Can you write up
your needs as a JIRA such that we can structure the discussion?

Thanks,

Markus

On Fri, Oct 31, 2014 at 7:43 PM, Byung-Gon Chun <bg...@gmail.com> wrote:

> About Evaluator-to-Driver communication, there's need to push TaskMessage's
> immediately instead of tagging it on a heartbeat. This is from our
> experience on building Surf, an in-memory caching service on REEF.
> Although the default should use the heartbeat mechanism, we should provide
> an option to push a TaskMessage. Otherwise, we have to set up a separate
> channel between Evaluator and Driver.
>
>
>
>
> On Thu, Oct 30, 2014 at 3:45 AM, Markus Weimer <ma...@weimo.de> wrote:
>
> > Hi,
> >
> >  A. Is heartbeating inherent to WAKE and automatically enabled on driver?
> >> Does it require configuration (like interval length)?
> >>
> > REEF always sets up the communications between Evaluators and the Driver.
> > The heartbeat mechanism is configurable somewhere, but we don't really
> > expose that at this point (we should, though).
> >
> >  B. Is heartbeating taking care of all driver/task handlers including
> >> ON_SEND_MESSAGE? Why are call(), getMessage() become synchronized with
> >> task/driver messaging?
> >>
> >
> > This is where a lot of confusion enters the picture. So allow me to be
> > verbose.
> >
> > Driver-to-Evaluator communication is always push based. That is, calling
> > RunningTask.send() will immediately proceed to send that message to the
> > Task in question. On the Evaluator side, the message is received and
> routed
> > to the handler configured in TaskConfiguration.ON_MESSAGE.
> >
> > Evaluator-to-Driver communication is heartbeat based, for the most part
> > (I'll talk about the exceptions to this below). When you want to send
> > something from the Task to the Driver, you register a handler with
> > TaskConfiguration.ON_SEND_MESSAGE. This handler will be invoked by the
> > Evaluator whenever a heartbeat is about to be sent. In other words: The
> > Task has no way of initiating the communication. This seems (and in fact
> > is) annoying, but it serves a purpose: When dealing with thousands of
> > Tasks, a single Driver can easily be overwhelmed if they all choose to
> send
> > large messages at once, leading to congestion and ultimately poor
> > performance. Allowing the framework to be in charge of the sending allows
> > us to pace things just right. Ideally, we'd have a dynamic heartbeat
> > interval that takes the current load on the Driver into account, for
> > instance. We currently don't have that, though (contributions welcome
> ;-)).
> >
> > Now the exception to that setup: State changes of an Evaluator are in
> fact
> > pushed. That is: When e.g. a Task finishes, we immediately send a message
> > to the Driver. This drastically reduces the latency compared to pull
> based
> > control flows such as the one found in Hadoop 1. It is safe, because we
> > still control the size and timing of these messages.
> >
> > Markus
> >
>
>
>
> --
> Byung-Gon Chun
>