[jira] [Commented] (MESOS-2735) Change the interaction between the slave and the resource estimator from polling to pushing

["Benjamin Mahler (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/MESOS-2735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14567860#comment-14567860 ] 

Benjamin Mahler commented on MESOS-2735:
----------------------------------------

{quote}
Finally, one of the advantages of the pull model is that it's easier to reason about because we don't have "anonymous" lambdas that cause execution in some other random place in the code (i.e., you can easily see in the slave where the future that gets returned from `ResourceEstimator::estimate()` gets handled). In addition, the ResourceEstimator remains "functional" in the sense that it just has to return some value (or a future) from it's functions versus invoking some callback that causes something to get run some other place (and in fact, may also block, so isn't it safer for the ResourceEstimator to invoke the callback in it's own `async`?).

The invocation of the `ResourceEstimator::estimate()` followed by the `.then` is a nice pattern that let's us compose with other things as well, which is harder to do with the lambda style callbacks and why we've avoided it where we've been able (in fact, I'm curious which place in the code are you imitating here?).
{quote}

The anonymous lambdas are because we do not have an abstraction to represent an asychronous stream of results:

{code}
// Returns a stream of the unused resources that can be oversubscribed.
Stream<Resources> unused();
{code}

This allows the caller to hold the stream and do composition on the next() Future result, so we achieve push but we keep the ability to do composition. {{http::Pipe}} is a string-specialized form of this and {{process::Queue<T>}} is an infinite-only version of {{process::Stream<T>}}. An important additional property of {{process::Stream<T>}} is that it should provide "asynchronous back-pressure" (e.g. unix pipes provide synchronous back-pressure by blocking writes, however we need an asynchronous mechanism to back-pressure the writes), to control the DoS concerns you guys have mentioned.

Take the allocator for example, it seems non-intuitive if the master had to continually call a method to obtain the next allocation from {{Future<Allocation> allocation()}}. At least, it requires more understanding than what it expressed in the return type. Whereas, if we have a {{Stream<Allocation> initialize()}} or {{Stream<Allocation> allocations()}}, we're clearly capturing the semantics within the return type. Back-pressure is the key to ensure that the master's speed of consumption limits the rate at which the allocator makes allocations. Today, {{process::Queue<T>}} is an equivalent replacement since allocations occur as an infinite stream and there is no back-pressuring.

> Change the interaction between the slave and the resource estimator from polling to pushing 
> --------------------------------------------------------------------------------------------
>
>                 Key: MESOS-2735
>                 URL: https://issues.apache.org/jira/browse/MESOS-2735
>             Project: Mesos
>          Issue Type: Bug
>            Reporter: Jie Yu
>            Assignee: Jie Yu
>              Labels: twitter
>
> This will make the semantics more clear. The resource estimator can control the speed of sending resources estimation to the slave.
> To avoid cyclic dependency, slave will register a callback with the resource estimator and the resource estimator will simply invoke that callback when there's a new estimation ready. The callback will be a defer to the slave's main event queue.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)