RFC: Framework <-> Executor Message Passing Optimization Removal

[Benjamin Mahler <be...@gmail.com>]

The existing Mesos API provides unreliable messaging passing for framework
<-> executor communication:

--> Schedulers can call 'sendFrameworkMessage(executor, slave, data)' on
the driver [1], this sends a message to the executor. This has a
best-effort optimization to bypass the master, and send the message to the
slave directly.

--> Executors can call 'sendFrameworkMessage(data)' on the driver [2],
which sends a message to the scheduler. This has a best-effort optimization
to bypass the master, and send the message to the scheduler driver directly
(via the slave).

As part of the HTTP API [3], schedulers can only make Calls against the
master, and all Events must be streamed back on the scheduler-initiated
connection to the master. This means that we can no longer easily support
bypassing the master as an optimization.

The plan is also to remove this optimization in the existing driver, in
order to conform to the upcoming Event/Call messages [4] used in the HTTP
API, so:


*** If anyone is relying on this best-effort optimization, please chime in!
***


[1]
https://github.com/apache/mesos/blob/0.22.1/include/mesos/scheduler.hpp#L289
[2]
https://github.com/apache/mesos/blob/0.22.1/include/mesos/executor.hpp#L185
[3]
https://docs.google.com/document/d/1pnIY_HckimKNvpqhKRhbc9eSItWNFT-priXh_urR-T0/edit
[4]
https://github.com/apache/mesos/blob/0.22.1/include/mesos/scheduler/scheduler.proto

Re: RFC: Framework <-> Executor Message Passing Optimization Removal

[Benjamin Mahler <be...@gmail.com>]

Hey Tom, to be clear, the best-effort messaging primitive will remain
either way.

What I'm referring to is the fact that we when we see offers for a slave in
the driver, we store each slave's address so that we can send messages
directly to the slave, rather than sending through the master. That is, we
try to avoid the extra hop through the master, since it's likely to add
non-trivial latency.

The point I'd like to stress here is that it is dangerous for frameworks to
_rely_ on this optimization, as it's possible to get into situations where
the driver must send all messages through the master (e.g. scheduler
failover, followed by no offers for the slaves if they are full).

On Tue, Jun 30, 2015 at 3:42 AM, Tom Arnfeld <to...@duedil.com> wrote:

> We're using it for streaming realtime logs to the framework. In our
> short-lived framework for building Docker images, the executor streams back
> stdout/stderr logs from the build to the client for ease of use/debugging
> and the executor->framework best-effort messaging stuff made this
> effortless.
>
> --
>
> Tom Arnfeld
> Developer // DueDil
>
>
> On Mon, Jun 29, 2015 at 10:48 PM, Benjamin Mahler <
> benjamin.mahler@gmail.com> wrote:
>
>> FYI Some folks reached out off thread that they are using this
>> optimization
>> for distributed health checking of tasks. This is on the order of
>> O(10,000)
>> framework messages per second for them, which may not be possible through
>> the master.
>>
>> On Tue, Jun 23, 2015 at 6:08 PM, Benjamin Mahler <
>> benjamin.mahler@gmail.com>
>> wrote:
>>
>> > The existing Mesos API provides unreliable messaging passing for
>> framework
>> > <-> executor communication:
>> >
>> > --> Schedulers can call 'sendFrameworkMessage(executor, slave, data)'
>> on
>> > the driver [1], this sends a message to the executor. This has a
>> > best-effort optimization to bypass the master, and send the message to
>> the
>> > slave directly.
>> >
>> > --> Executors can call 'sendFrameworkMessage(data)' on the driver [2],
>> > which sends a message to the scheduler. This has a best-effort
>> optimization
>> > to bypass the master, and send the message to the scheduler driver
>> directly
>> > (via the slave).
>> >
>> > As part of the HTTP API [3], schedulers can only make Calls against the
>> > master, and all Events must be streamed back on the scheduler-initiated
>> > connection to the master. This means that we can no longer easily
>> support
>> > bypassing the master as an optimization.
>> >
>> > The plan is also to remove this optimization in the existing driver, in
>> > order to conform to the upcoming Event/Call messages [4] used in the
>> HTTP
>> > API, so:
>> >
>> >
>> > *** If anyone is relying on this best-effort optimization, please chime
>> > in! ***
>> >
>> >
>> > [1]
>> >
>> https://github.com/apache/mesos/blob/0.22.1/include/mesos/scheduler.hpp#L289
>> > [2]
>> >
>> https://github.com/apache/mesos/blob/0.22.1/include/mesos/executor.hpp#L185
>> > [3]
>> >
>> https://docs.google.com/document/d/1pnIY_HckimKNvpqhKRhbc9eSItWNFT-priXh_urR-T0/edit
>> > [4]
>> >
>> https://github.com/apache/mesos/blob/0.22.1/include/mesos/scheduler/scheduler.proto
>> >
>>
>
>

Re: RFC: Framework <-> Executor Message Passing Optimization Removal

[Benjamin Mahler <be...@gmail.com>]

Hey Tom, to be clear, the best-effort messaging primitive will remain
either way.

What I'm referring to is the fact that we when we see offers for a slave in
the driver, we store each slave's address so that we can send messages
directly to the slave, rather than sending through the master. That is, we
try to avoid the extra hop through the master, since it's likely to add
non-trivial latency.

The point I'd like to stress here is that it is dangerous for frameworks to
_rely_ on this optimization, as it's possible to get into situations where
the driver must send all messages through the master (e.g. scheduler
failover, followed by no offers for the slaves if they are full).

On Tue, Jun 30, 2015 at 3:42 AM, Tom Arnfeld <to...@duedil.com> wrote:

> We're using it for streaming realtime logs to the framework. In our
> short-lived framework for building Docker images, the executor streams back
> stdout/stderr logs from the build to the client for ease of use/debugging
> and the executor->framework best-effort messaging stuff made this
> effortless.
>
> --
>
> Tom Arnfeld
> Developer // DueDil
>
>
> On Mon, Jun 29, 2015 at 10:48 PM, Benjamin Mahler <
> benjamin.mahler@gmail.com> wrote:
>
>> FYI Some folks reached out off thread that they are using this
>> optimization
>> for distributed health checking of tasks. This is on the order of
>> O(10,000)
>> framework messages per second for them, which may not be possible through
>> the master.
>>
>> On Tue, Jun 23, 2015 at 6:08 PM, Benjamin Mahler <
>> benjamin.mahler@gmail.com>
>> wrote:
>>
>> > The existing Mesos API provides unreliable messaging passing for
>> framework
>> > <-> executor communication:
>> >
>> > --> Schedulers can call 'sendFrameworkMessage(executor, slave, data)'
>> on
>> > the driver [1], this sends a message to the executor. This has a
>> > best-effort optimization to bypass the master, and send the message to
>> the
>> > slave directly.
>> >
>> > --> Executors can call 'sendFrameworkMessage(data)' on the driver [2],
>> > which sends a message to the scheduler. This has a best-effort
>> optimization
>> > to bypass the master, and send the message to the scheduler driver
>> directly
>> > (via the slave).
>> >
>> > As part of the HTTP API [3], schedulers can only make Calls against the
>> > master, and all Events must be streamed back on the scheduler-initiated
>> > connection to the master. This means that we can no longer easily
>> support
>> > bypassing the master as an optimization.
>> >
>> > The plan is also to remove this optimization in the existing driver, in
>> > order to conform to the upcoming Event/Call messages [4] used in the
>> HTTP
>> > API, so:
>> >
>> >
>> > *** If anyone is relying on this best-effort optimization, please chime
>> > in! ***
>> >
>> >
>> > [1]
>> >
>> https://github.com/apache/mesos/blob/0.22.1/include/mesos/scheduler.hpp#L289
>> > [2]
>> >
>> https://github.com/apache/mesos/blob/0.22.1/include/mesos/executor.hpp#L185
>> > [3]
>> >
>> https://docs.google.com/document/d/1pnIY_HckimKNvpqhKRhbc9eSItWNFT-priXh_urR-T0/edit
>> > [4]
>> >
>> https://github.com/apache/mesos/blob/0.22.1/include/mesos/scheduler/scheduler.proto
>> >
>>
>
>

Re: RFC: Framework <-> Executor Message Passing Optimization Removal

[Tom Arnfeld <to...@duedil.com>]

We're using it for streaming realtime logs to the framework. In our short-lived framework for building Docker images, the executor streams back stdout/stderr logs from the build to the client for ease of use/debugging and the executor->framework best-effort messaging stuff made this effortless.


--


Tom Arnfeld

Developer // DueDil

On Mon, Jun 29, 2015 at 10:48 PM, Benjamin Mahler
<be...@gmail.com> wrote:

> FYI Some folks reached out off thread that they are using this optimization
> for distributed health checking of tasks. This is on the order of O(10,000)
> framework messages per second for them, which may not be possible through
> the master.
> On Tue, Jun 23, 2015 at 6:08 PM, Benjamin Mahler <be...@gmail.com>
> wrote:
>> The existing Mesos API provides unreliable messaging passing for framework
>> <-> executor communication:
>>
>> --> Schedulers can call 'sendFrameworkMessage(executor, slave, data)' on
>> the driver [1], this sends a message to the executor. This has a
>> best-effort optimization to bypass the master, and send the message to the
>> slave directly.
>>
>> --> Executors can call 'sendFrameworkMessage(data)' on the driver [2],
>> which sends a message to the scheduler. This has a best-effort optimization
>> to bypass the master, and send the message to the scheduler driver directly
>> (via the slave).
>>
>> As part of the HTTP API [3], schedulers can only make Calls against the
>> master, and all Events must be streamed back on the scheduler-initiated
>> connection to the master. This means that we can no longer easily support
>> bypassing the master as an optimization.
>>
>> The plan is also to remove this optimization in the existing driver, in
>> order to conform to the upcoming Event/Call messages [4] used in the HTTP
>> API, so:
>>
>>
>> *** If anyone is relying on this best-effort optimization, please chime
>> in! ***
>>
>>
>> [1]
>> https://github.com/apache/mesos/blob/0.22.1/include/mesos/scheduler.hpp#L289
>> [2]
>> https://github.com/apache/mesos/blob/0.22.1/include/mesos/executor.hpp#L185
>> [3]
>> https://docs.google.com/document/d/1pnIY_HckimKNvpqhKRhbc9eSItWNFT-priXh_urR-T0/edit
>> [4]
>> https://github.com/apache/mesos/blob/0.22.1/include/mesos/scheduler/scheduler.proto
>>

Re: RFC: Framework <-> Executor Message Passing Optimization Removal

[Tom Arnfeld <to...@duedil.com>]

We're using it for streaming realtime logs to the framework. In our short-lived framework for building Docker images, the executor streams back stdout/stderr logs from the build to the client for ease of use/debugging and the executor->framework best-effort messaging stuff made this effortless.


--


Tom Arnfeld

Developer // DueDil

On Mon, Jun 29, 2015 at 10:48 PM, Benjamin Mahler
<be...@gmail.com> wrote:

> FYI Some folks reached out off thread that they are using this optimization
> for distributed health checking of tasks. This is on the order of O(10,000)
> framework messages per second for them, which may not be possible through
> the master.
> On Tue, Jun 23, 2015 at 6:08 PM, Benjamin Mahler <be...@gmail.com>
> wrote:
>> The existing Mesos API provides unreliable messaging passing for framework
>> <-> executor communication:
>>
>> --> Schedulers can call 'sendFrameworkMessage(executor, slave, data)' on
>> the driver [1], this sends a message to the executor. This has a
>> best-effort optimization to bypass the master, and send the message to the
>> slave directly.
>>
>> --> Executors can call 'sendFrameworkMessage(data)' on the driver [2],
>> which sends a message to the scheduler. This has a best-effort optimization
>> to bypass the master, and send the message to the scheduler driver directly
>> (via the slave).
>>
>> As part of the HTTP API [3], schedulers can only make Calls against the
>> master, and all Events must be streamed back on the scheduler-initiated
>> connection to the master. This means that we can no longer easily support
>> bypassing the master as an optimization.
>>
>> The plan is also to remove this optimization in the existing driver, in
>> order to conform to the upcoming Event/Call messages [4] used in the HTTP
>> API, so:
>>
>>
>> *** If anyone is relying on this best-effort optimization, please chime
>> in! ***
>>
>>
>> [1]
>> https://github.com/apache/mesos/blob/0.22.1/include/mesos/scheduler.hpp#L289
>> [2]
>> https://github.com/apache/mesos/blob/0.22.1/include/mesos/executor.hpp#L185
>> [3]
>> https://docs.google.com/document/d/1pnIY_HckimKNvpqhKRhbc9eSItWNFT-priXh_urR-T0/edit
>> [4]
>> https://github.com/apache/mesos/blob/0.22.1/include/mesos/scheduler/scheduler.proto
>>

Re: RFC: Framework <-> Executor Message Passing Optimization Removal

[Benjamin Mahler <be...@gmail.com>]

FYI Some folks reached out off thread that they are using this optimization
for distributed health checking of tasks. This is on the order of O(10,000)
framework messages per second for them, which may not be possible through
the master.

On Tue, Jun 23, 2015 at 6:08 PM, Benjamin Mahler <be...@gmail.com>
wrote:

> The existing Mesos API provides unreliable messaging passing for framework
> <-> executor communication:
>
> --> Schedulers can call 'sendFrameworkMessage(executor, slave, data)' on
> the driver [1], this sends a message to the executor. This has a
> best-effort optimization to bypass the master, and send the message to the
> slave directly.
>
> --> Executors can call 'sendFrameworkMessage(data)' on the driver [2],
> which sends a message to the scheduler. This has a best-effort optimization
> to bypass the master, and send the message to the scheduler driver directly
> (via the slave).
>
> As part of the HTTP API [3], schedulers can only make Calls against the
> master, and all Events must be streamed back on the scheduler-initiated
> connection to the master. This means that we can no longer easily support
> bypassing the master as an optimization.
>
> The plan is also to remove this optimization in the existing driver, in
> order to conform to the upcoming Event/Call messages [4] used in the HTTP
> API, so:
>
>
> *** If anyone is relying on this best-effort optimization, please chime
> in! ***
>
>
> [1]
> https://github.com/apache/mesos/blob/0.22.1/include/mesos/scheduler.hpp#L289
> [2]
> https://github.com/apache/mesos/blob/0.22.1/include/mesos/executor.hpp#L185
> [3]
> https://docs.google.com/document/d/1pnIY_HckimKNvpqhKRhbc9eSItWNFT-priXh_urR-T0/edit
> [4]
> https://github.com/apache/mesos/blob/0.22.1/include/mesos/scheduler/scheduler.proto
>

Re: RFC: Framework <-> Executor Message Passing Optimization Removal

[Benjamin Mahler <be...@gmail.com>]

FYI Some folks reached out off thread that they are using this optimization
for distributed health checking of tasks. This is on the order of O(10,000)
framework messages per second for them, which may not be possible through
the master.

On Tue, Jun 23, 2015 at 6:08 PM, Benjamin Mahler <be...@gmail.com>
wrote:

> The existing Mesos API provides unreliable messaging passing for framework
> <-> executor communication:
>
> --> Schedulers can call 'sendFrameworkMessage(executor, slave, data)' on
> the driver [1], this sends a message to the executor. This has a
> best-effort optimization to bypass the master, and send the message to the
> slave directly.
>
> --> Executors can call 'sendFrameworkMessage(data)' on the driver [2],
> which sends a message to the scheduler. This has a best-effort optimization
> to bypass the master, and send the message to the scheduler driver directly
> (via the slave).
>
> As part of the HTTP API [3], schedulers can only make Calls against the
> master, and all Events must be streamed back on the scheduler-initiated
> connection to the master. This means that we can no longer easily support
> bypassing the master as an optimization.
>
> The plan is also to remove this optimization in the existing driver, in
> order to conform to the upcoming Event/Call messages [4] used in the HTTP
> API, so:
>
>
> *** If anyone is relying on this best-effort optimization, please chime
> in! ***
>
>
> [1]
> https://github.com/apache/mesos/blob/0.22.1/include/mesos/scheduler.hpp#L289
> [2]
> https://github.com/apache/mesos/blob/0.22.1/include/mesos/executor.hpp#L185
> [3]
> https://docs.google.com/document/d/1pnIY_HckimKNvpqhKRhbc9eSItWNFT-priXh_urR-T0/edit
> [4]
> https://github.com/apache/mesos/blob/0.22.1/include/mesos/scheduler/scheduler.proto
>