Slave<->Executor protocol instead of C++ driver + language bindings

[Vetoshkin Nikita <ni...@gmail.com>]

Hi!
Have you ever considered providing a protocol for implementing Executor in
foreign language additionally to writing various bindings yourself?
I think that protocol isn't that hard as you already use protobuf to
exchange information, but I haven't dug deep in the sources. Documenting
wire protocol could open ways to implement Executors in pure python or
golang or any other language. It could provide some benefits:
  * in Python one could use gevent which is pretty popular
  * golang has it's own eventloop builtin
  * pure language binding could save a lot of trouble bridging with
unfriendly C++
  * building and using pure language client could be performed using native
tools like `pip install` or `go get` without need to establish c++ dev
environment.
Going further you could open Framework<->Master protocol to allow crazy
stuff like using all fancy Netty, Akka, Finagle, whatever... network tools,
frameworks and libraries to talk to Mesos.

Sorry to bother if it's a wrong place or there's already an answer in the
wild. Thanks in advance!

Re: Slave<->Executor protocol instead of C++ driver + language bindings

[Vetoshkin Nikita <ni...@gmail.com>]

As I promised (a little late though) - ticket for discussion and
implementation is created. https://issues.apache.org/jira/browse/MESOS-930


On Mon, Jan 13, 2014 at 11:13 PM, Vetoshkin Nikita <
nikita.vetoshkin@gmail.com> wrote:

> Great! Thanks for the answers.
> Now I think I'm ready to create JIRA, describing all thoughts and
> preliminary plan, how can we achieve the goal. Hopefully I'll find sometime
> to do it during this week.
>
>
> On Mon, Jan 13, 2014 at 9:32 PM, Vinod Kone <vi...@gmail.com> wrote:
>
>> >
>> > Yeah, found it. Pretty strange way to encode (why not use custom HTTP
>> > headers e.g. X-Libprocess-From). Are you suggesting that this protocol
>> > should not be documented and made external, instead some other protocol
>> > should be used for communication?
>> >
>>
>> No. I'm in favor of documenting the protocol and sticking with HTTP
>> wrapped
>> protobufs. I'm not familiar with custom HTTP headers but is
>> 'X-Libprocess-From' a way to optionally specify the libprocess id? If yes,
>> that seems like a good option. Note that we can not completely get rid of
>> the libprocess id in the protocol messages because most of our tests bring
>> up an in-process mesos cluster, where different components (master, slave,
>> etc) have same 'ip:port' but only differ in the 'id'.
>>
>>
>> > And another quick question about communication - how does executor
>> driver
>> > knows the address (PID, host:port) of the slave?
>>
>>
>> Slave sets an environment variable "MESOS_SLAVE_PID" before exec()ing an
>> executor. See MesosExecutorDriver::start().
>>
>> Cheers,
>> Vinod
>>
>>
>>
>> > On Mon, Jan 13, 2014 at 10:59 AM, Vinod Kone <vi...@gmail.com>
>> wrote:
>> >
>> > > On Sun, Jan 12, 2014 at 10:35 PM, Vetoshkin Nikita <
>> > > nikita.vetoshkin@gmail.com> wrote:
>> > >
>> > > > but to me it seems that
>> > > > protocol is protobuf messages inside HTTP POST body. I just couldn't
>> > find
>> > > > yet where to does executor driver send them (UNIX socket? TCP?).
>> > Wouldn't
>> > > > simply documenting it be enough for a start?
>> > > >
>> > >
>> > > Look for 'send()' calls inside exec.cpp. It is a libprocess method
>> that
>> > > sends messages to an end point. Similarly 'install<>()' is a
>> libprocess
>> > > method to receive messages. libprocess does use HTTP for
>> > sending/receiving
>> > > protobuf messages but it expects them to be in a certain format.
>> > > Specifically it expects 'from' and 'to' fields in addition to the
>> > protobuf
>> > > message. 'from/to' correspond to libprocess endpoint (internally
>> called
>> > > PID) : <id>@ip:port.
>> > >
>> >
>>
>
>

Re: Slave<->Executor protocol instead of C++ driver + language bindings

[Vetoshkin Nikita <ni...@gmail.com>]

Great! Thanks for the answers.
Now I think I'm ready to create JIRA, describing all thoughts and
preliminary plan, how can we achieve the goal. Hopefully I'll find sometime
to do it during this week.


On Mon, Jan 13, 2014 at 9:32 PM, Vinod Kone <vi...@gmail.com> wrote:

> >
> > Yeah, found it. Pretty strange way to encode (why not use custom HTTP
> > headers e.g. X-Libprocess-From). Are you suggesting that this protocol
> > should not be documented and made external, instead some other protocol
> > should be used for communication?
> >
>
> No. I'm in favor of documenting the protocol and sticking with HTTP wrapped
> protobufs. I'm not familiar with custom HTTP headers but is
> 'X-Libprocess-From' a way to optionally specify the libprocess id? If yes,
> that seems like a good option. Note that we can not completely get rid of
> the libprocess id in the protocol messages because most of our tests bring
> up an in-process mesos cluster, where different components (master, slave,
> etc) have same 'ip:port' but only differ in the 'id'.
>
>
> > And another quick question about communication - how does executor driver
> > knows the address (PID, host:port) of the slave?
>
>
> Slave sets an environment variable "MESOS_SLAVE_PID" before exec()ing an
> executor. See MesosExecutorDriver::start().
>
> Cheers,
> Vinod
>
>
>
> > On Mon, Jan 13, 2014 at 10:59 AM, Vinod Kone <vi...@gmail.com>
> wrote:
> >
> > > On Sun, Jan 12, 2014 at 10:35 PM, Vetoshkin Nikita <
> > > nikita.vetoshkin@gmail.com> wrote:
> > >
> > > > but to me it seems that
> > > > protocol is protobuf messages inside HTTP POST body. I just couldn't
> > find
> > > > yet where to does executor driver send them (UNIX socket? TCP?).
> > Wouldn't
> > > > simply documenting it be enough for a start?
> > > >
> > >
> > > Look for 'send()' calls inside exec.cpp. It is a libprocess method that
> > > sends messages to an end point. Similarly 'install<>()' is a libprocess
> > > method to receive messages. libprocess does use HTTP for
> > sending/receiving
> > > protobuf messages but it expects them to be in a certain format.
> > > Specifically it expects 'from' and 'to' fields in addition to the
> > protobuf
> > > message. 'from/to' correspond to libprocess endpoint (internally called
> > > PID) : <id>@ip:port.
> > >
> >
>

Re: Slave<->Executor protocol instead of C++ driver + language bindings

[Vinod Kone <vi...@gmail.com>]

>
> Yeah, found it. Pretty strange way to encode (why not use custom HTTP
> headers e.g. X-Libprocess-From). Are you suggesting that this protocol
> should not be documented and made external, instead some other protocol
> should be used for communication?
>

No. I'm in favor of documenting the protocol and sticking with HTTP wrapped
protobufs. I'm not familiar with custom HTTP headers but is
'X-Libprocess-From' a way to optionally specify the libprocess id? If yes,
that seems like a good option. Note that we can not completely get rid of
the libprocess id in the protocol messages because most of our tests bring
up an in-process mesos cluster, where different components (master, slave,
etc) have same 'ip:port' but only differ in the 'id'.


> And another quick question about communication - how does executor driver
> knows the address (PID, host:port) of the slave?


Slave sets an environment variable "MESOS_SLAVE_PID" before exec()ing an
executor. See MesosExecutorDriver::start().

Cheers,
Vinod



> On Mon, Jan 13, 2014 at 10:59 AM, Vinod Kone <vi...@gmail.com> wrote:
>
> > On Sun, Jan 12, 2014 at 10:35 PM, Vetoshkin Nikita <
> > nikita.vetoshkin@gmail.com> wrote:
> >
> > > but to me it seems that
> > > protocol is protobuf messages inside HTTP POST body. I just couldn't
> find
> > > yet where to does executor driver send them (UNIX socket? TCP?).
> Wouldn't
> > > simply documenting it be enough for a start?
> > >
> >
> > Look for 'send()' calls inside exec.cpp. It is a libprocess method that
> > sends messages to an end point. Similarly 'install<>()' is a libprocess
> > method to receive messages. libprocess does use HTTP for
> sending/receiving
> > protobuf messages but it expects them to be in a certain format.
> > Specifically it expects 'from' and 'to' fields in addition to the
> protobuf
> > message. 'from/to' correspond to libprocess endpoint (internally called
> > PID) : <id>@ip:port.
> >
>

Re: Slave<->Executor protocol instead of C++ driver + language bindings

[Vetoshkin Nikita <ni...@gmail.com>]

Yeah, found it. Pretty strange way to encode (why not use custom HTTP
headers e.g. X-Libprocess-From). Are you suggesting that this protocol
should not be documented and made external, instead some other protocol
should be used for communication?
And another quick question about communication - how does executor driver
knows the address (PID, host:port) of the slave?


On Mon, Jan 13, 2014 at 10:59 AM, Vinod Kone <vi...@gmail.com> wrote:

> On Sun, Jan 12, 2014 at 10:35 PM, Vetoshkin Nikita <
> nikita.vetoshkin@gmail.com> wrote:
>
> > but to me it seems that
> > protocol is protobuf messages inside HTTP POST body. I just couldn't find
> > yet where to does executor driver send them (UNIX socket? TCP?). Wouldn't
> > simply documenting it be enough for a start?
> >
>
> Look for 'send()' calls inside exec.cpp. It is a libprocess method that
> sends messages to an end point. Similarly 'install<>()' is a libprocess
> method to receive messages. libprocess does use HTTP for sending/receiving
> protobuf messages but it expects them to be in a certain format.
> Specifically it expects 'from' and 'to' fields in addition to the protobuf
> message. 'from/to' correspond to libprocess endpoint (internally called
> PID) : <id>@ip:port.
>

Re: Slave<->Executor protocol instead of C++ driver + language bindings

[Vinod Kone <vi...@gmail.com>]

On Sun, Jan 12, 2014 at 10:35 PM, Vetoshkin Nikita <
nikita.vetoshkin@gmail.com> wrote:

> but to me it seems that
> protocol is protobuf messages inside HTTP POST body. I just couldn't find
> yet where to does executor driver send them (UNIX socket? TCP?). Wouldn't
> simply documenting it be enough for a start?
>

Look for 'send()' calls inside exec.cpp. It is a libprocess method that
sends messages to an end point. Similarly 'install<>()' is a libprocess
method to receive messages. libprocess does use HTTP for sending/receiving
protobuf messages but it expects them to be in a certain format.
Specifically it expects 'from' and 'to' fields in addition to the protobuf
message. 'from/to' correspond to libprocess endpoint (internally called
PID) : <id>@ip:port.

Re: Slave<->Executor protocol instead of C++ driver + language bindings

[Vetoshkin Nikita <ni...@gmail.com>]

Thank you for your answer and interest in this!
Just a small note about scheduler logic. Mozilla guys created Kazoo - pure
python implementation of Zookeeper protocol (which isn't trivial with all
retries and timeouts) and it seems to work. So - nothing extraordinary,
just careful programming :) But, yes, let's start with executors though.
I don't quite understand why would we need to change libprocess. I haven't
dug too deep into sources (and I'm no C++ guy) - but to me it seems that
protocol is protobuf messages inside HTTP POST body. I just couldn't find
yet where to does executor driver send them (UNIX socket? TCP?). Wouldn't
simply documenting it be enough for a start?
I'll gladly create JIRA, just want to have better understanding of this
undertaking.



On Mon, Jan 13, 2014 at 7:25 AM, Vinod Kone <vi...@gmail.com> wrote:

> Hey Vetoshkin,
>
> Thanks for bringing up the question. As Dave alluded to we have discussed
> the pros and cons of getting away from language bindings in the past.
>
> One of the advantages of us proving language bindings and frameworks
> linking to it is that we have to implement non-trivial scheduler/executor
> specific logic in a single place, the driver (sched.cpp and exec.cpp
> respectively). These drivers are not simply wrappers that call in to the
> scheduler/executor after getting a message from Mesos.
>
> For example the scheduler driver does authentication and mesos master
> discovery on behalf of the scheduler. If we were to get rid of the driver,
> every framework (or language) has to cook up their own solutions for these
> in addition to handling the transport of the protocol messages, which makes
> writing a new framework rather tedious.
>
> That said the executor driver unlike the scheduler driver, has very little
> logic in it besides wrapping the executor. So we could conceivably get rid
> of the executor bindings in favor of providing pure language
> implementations. This requires some (substantial?) refactoring of
> libprocess (mesos's communication framework) to send/receive HTTP wrapped
> protobuf messages. Do you mind creating a ticket to track this work?
>
> HTH,
>
>
> On Sat, Jan 11, 2014 at 7:01 PM, Dave Lester <dave@ischool.berkeley.edu
> >wrote:
>
> > HI Vetoshkin,
> >
> > At face value, I think the goal of making language bindings unnecessary
> in
> > favor of adopting a protocol makes sense -- I'd love to see Mesos expand
> > its support for framework languages, and this could potentially help make
> > that easier.
> >
> > If I remember correctly, Vinod (cc'd) was discussing a similar idea a few
> > months back. Perhaps he'd like to weigh in.
> >
> > Dave
> >
> >
> > On Thu, Jan 9, 2014 at 10:09 PM, Vetoshkin Nikita <
> > nikita.vetoshkin@gmail.com> wrote:
> >
> >> Hi!
> >> Have you ever considered providing a protocol for implementing Executor
> in
> >> foreign language additionally to writing various bindings yourself?
> >> I think that protocol isn't that hard as you already use protobuf to
> >> exchange information, but I haven't dug deep in the sources. Documenting
> >> wire protocol could open ways to implement Executors in pure python or
> >> golang or any other language. It could provide some benefits:
> >>   * in Python one could use gevent which is pretty popular
> >>   * golang has it's own eventloop builtin
> >>   * pure language binding could save a lot of trouble bridging with
> >> unfriendly C++
> >>   * building and using pure language client could be performed using
> >> native
> >> tools like `pip install` or `go get` without need to establish c++ dev
> >> environment.
> >> Going further you could open Framework<->Master protocol to allow crazy
> >> stuff like using all fancy Netty, Akka, Finagle, whatever... network
> >> tools,
> >> frameworks and libraries to talk to Mesos.
> >>
> >> Sorry to bother if it's a wrong place or there's already an answer in
> the
> >> wild. Thanks in advance!
> >>
> >
> >
>

Re: Slave<->Executor protocol instead of C++ driver + language bindings

[Vinod Kone <vi...@gmail.com>]

Hey Vetoshkin,

Thanks for bringing up the question. As Dave alluded to we have discussed
the pros and cons of getting away from language bindings in the past.

One of the advantages of us proving language bindings and frameworks
linking to it is that we have to implement non-trivial scheduler/executor
specific logic in a single place, the driver (sched.cpp and exec.cpp
respectively). These drivers are not simply wrappers that call in to the
scheduler/executor after getting a message from Mesos.

For example the scheduler driver does authentication and mesos master
discovery on behalf of the scheduler. If we were to get rid of the driver,
every framework (or language) has to cook up their own solutions for these
in addition to handling the transport of the protocol messages, which makes
writing a new framework rather tedious.

That said the executor driver unlike the scheduler driver, has very little
logic in it besides wrapping the executor. So we could conceivably get rid
of the executor bindings in favor of providing pure language
implementations. This requires some (substantial?) refactoring of
libprocess (mesos's communication framework) to send/receive HTTP wrapped
protobuf messages. Do you mind creating a ticket to track this work?

HTH,


On Sat, Jan 11, 2014 at 7:01 PM, Dave Lester <da...@ischool.berkeley.edu>wrote:

> HI Vetoshkin,
>
> At face value, I think the goal of making language bindings unnecessary in
> favor of adopting a protocol makes sense -- I'd love to see Mesos expand
> its support for framework languages, and this could potentially help make
> that easier.
>
> If I remember correctly, Vinod (cc'd) was discussing a similar idea a few
> months back. Perhaps he'd like to weigh in.
>
> Dave
>
>
> On Thu, Jan 9, 2014 at 10:09 PM, Vetoshkin Nikita <
> nikita.vetoshkin@gmail.com> wrote:
>
>> Hi!
>> Have you ever considered providing a protocol for implementing Executor in
>> foreign language additionally to writing various bindings yourself?
>> I think that protocol isn't that hard as you already use protobuf to
>> exchange information, but I haven't dug deep in the sources. Documenting
>> wire protocol could open ways to implement Executors in pure python or
>> golang or any other language. It could provide some benefits:
>>   * in Python one could use gevent which is pretty popular
>>   * golang has it's own eventloop builtin
>>   * pure language binding could save a lot of trouble bridging with
>> unfriendly C++
>>   * building and using pure language client could be performed using
>> native
>> tools like `pip install` or `go get` without need to establish c++ dev
>> environment.
>> Going further you could open Framework<->Master protocol to allow crazy
>> stuff like using all fancy Netty, Akka, Finagle, whatever... network
>> tools,
>> frameworks and libraries to talk to Mesos.
>>
>> Sorry to bother if it's a wrong place or there's already an answer in the
>> wild. Thanks in advance!
>>
>
>

Re: Slave<->Executor protocol instead of C++ driver + language bindings

[Dave Lester <da...@ischool.berkeley.edu>]

HI Vetoshkin,

At face value, I think the goal of making language bindings unnecessary in
favor of adopting a protocol makes sense -- I'd love to see Mesos expand
its support for framework languages, and this could potentially help make
that easier.

If I remember correctly, Vinod (cc'd) was discussing a similar idea a few
months back. Perhaps he'd like to weigh in.

Dave


On Thu, Jan 9, 2014 at 10:09 PM, Vetoshkin Nikita <
nikita.vetoshkin@gmail.com> wrote:

> Hi!
> Have you ever considered providing a protocol for implementing Executor in
> foreign language additionally to writing various bindings yourself?
> I think that protocol isn't that hard as you already use protobuf to
> exchange information, but I haven't dug deep in the sources. Documenting
> wire protocol could open ways to implement Executors in pure python or
> golang or any other language. It could provide some benefits:
>   * in Python one could use gevent which is pretty popular
>   * golang has it's own eventloop builtin
>   * pure language binding could save a lot of trouble bridging with
> unfriendly C++
>   * building and using pure language client could be performed using native
> tools like `pip install` or `go get` without need to establish c++ dev
> environment.
> Going further you could open Framework<->Master protocol to allow crazy
> stuff like using all fancy Netty, Akka, Finagle, whatever... network tools,
> frameworks and libraries to talk to Mesos.
>
> Sorry to bother if it's a wrong place or there's already an answer in the
> wild. Thanks in advance!
>