programming with async loops in libprocess

[Yan Xu <ya...@jxu.me>]

In fixing MESOS-884 <https://issues.apache.org/jira/browse/MESOS-884> I
realized that it might be one of more commonly made but easy to overlook
mistakes so I thought I'd share with the dev group.

In libprocess' async programming model loops are expressed using

1) A call to a method that returns a Future
2) A callback registered with this future invokes another such call
3) The chain/graph of calls that consists of 1) and 2) forms a loop. e.g.
    A -> B -> A or,
    A -> B -> C -> B etc.

In MESOS-884 <https://issues.apache.org/jira/browse/MESOS-884> one such
"node" invokes two 1) calls which forked another loop. The outer loop can
repeatedly fork the inner loop.

I found it's important to reason about the termination and divergence of
the loops.
Drawing the call graph helps detect the loops: if there are multiple cycles
in the connected graph then it's possible for multiple copies of the child
loop to get launched again and again which leads to ever-growing dispatch
queue and nondeterministic results.

In general, if a Process runs multiple loops it's better to keep them
separate: launching them in initialize() and making sure they don't cross
their paths is good practice.

--
Jiang Yan Xu <ya...@jxu.me> @xujyan <http://twitter.com/xujyan>