Design doc: ordered delivery in libprocess

[Neil Conway <ne...@gmail.com>]

A few months ago, there was a dev list thread on whether libprocess
should provide ordered delivery [1]. The consensus then was that
libprocess doesn't provide ordered delivery in a few corner cases, but
that we should fix that behavior to guarantee ordered (but unreliable)
message delivery.

Please see this design doc for a proposal of how to achieve this:

https://docs.google.com/document/d/1oqv42-GYXfh0TsyXGCBhxakXw8ckXMCnEb0T-ydEyYM/edit#

Comments welcome.

Thanks,
Neil

[1] https://mail-archives.apache.org/mod_mbox/mesos-dev/201511.mbox/%3CCAOW5sYaANyaRD-Mbk7E_VsWwF=XrTCVhszUvmuNJ4sQ8JdT-Zg@mail.gmail.com%3E

Re: Design doc: ordered delivery in libprocess

[Neil Conway <ne...@gmail.com>]

Hi all,

It turns out this design doesn't work. The reason is that we made the
following assumption: given a client like:

s1 = connect(); send(s1, "msg1"); close(s1);
s2 = connect(); send(s2, "msg2"); close(s2);

And a server like:

char buf[...];
while (true) {
  s = accept();
  recv(s, buf);
  close(s);
}

Our design required that the server-side socket corresponding to "s1"
would always be accepted _before_ the server-side socket for "s2".
That is, that we can rely on the order in which server-side sockets
are accept()'d to infer the order in which client-side sockets are
connect()'d, as long as the client only makes at most one connect() to
a given host at a given time.

It turns out that this isn't the case on either OSX or Linux. I
haven't dug too deeply into the reasons why, but one plausible
explanation is that the kernel is using multiple accept() queues and
dispatching inbound connections to a queue in some unpredictable way.

It seems that implementing ordered delivery will require adding
metadata to the libprocess socket-establishment protocol to give the
recipient enough information to determine which of the inbound
connections from a given libprocess instance is the "newest".

Neil


On Fri, Mar 25, 2016 at 12:50 PM, Neil Conway <ne...@gmail.com> wrote:
> A few months ago, there was a dev list thread on whether libprocess
> should provide ordered delivery [1]. The consensus then was that
> libprocess doesn't provide ordered delivery in a few corner cases, but
> that we should fix that behavior to guarantee ordered (but unreliable)
> message delivery.
>
> Please see this design doc for a proposal of how to achieve this:
>
> https://docs.google.com/document/d/1oqv42-GYXfh0TsyXGCBhxakXw8ckXMCnEb0T-ydEyYM/edit#
>
> Comments welcome.
>
> Thanks,
> Neil
>
> [1] https://mail-archives.apache.org/mod_mbox/mesos-dev/201511.mbox/%3CCAOW5sYaANyaRD-Mbk7E_VsWwF=XrTCVhszUvmuNJ4sQ8JdT-Zg@mail.gmail.com%3E