You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@mesos.apache.org by Chun-Hung Hsiao <ch...@mesosphere.io> on 2017/09/12 02:32:53 UTC
Adding process::Executor::execute()
Hi,
I'm thinking about extending `process::Executor` with a new `execute()`
interface.
The need of this new interface surfaced when I'm working on
https://issues.apache.org/jira/browse/MESOS-7964
Summary:
1. A disk GC might execute multiple `rmdirs` callbacks, and some of them
are heavy duty. We don't want to run them on `GarbageCollectorProcess` so
that it won't block other events of the process.
Currently we do the following:
async(rmdirs).onAny(...);
2. `async` puts each `rmdir` callback in an actor. When there are many
heavy-duty `rmdirs` callbacks, the actors end up occupying all worker
threads and blocking other actors for minutes.
Yan suggested me to use `process::Executor` such that:
1. The `rmdirs` callbacks are not executed on `GarbaceGollectorProcess`
2. All `rmdirs` callbacks are executed on a single thread
Since the `Executor` class only contains a `defer()` function that returns
a `_Deferred` structure,
I'm doing the following:
executor.defer(rmdirs).operator
std::function<Future<Nothing>()>()().onAny(...)
Would it make sense to add another `execute()` function to directly return
a `Future`?
- Chun-Hung
Re: Adding process::Executor::execute()
Posted by Chun-Hung Hsiao <ch...@mesosphere.io>.
Thanks Ben. Just posted a patch: https://reviews.apache.org/r/62252/.
- Chun-Hung
On Mon, Sep 11, 2017 at 9:15 PM, Benjamin Hindman <be...@mesosphere.io>
wrote:
> Quick clarification: you'll have a single `process::Executor` and queue up
> all the rmdirs on that, correct? So you'll still tie up a worker thread,
> but only one of them.
>
> Either way it makes sense to add `process::Executor::execute()`. I'm
> happy to shepherd that for you Chun, send me a patch!
>
> On Mon, Sep 11, 2017 at 7:32 PM, Chun-Hung Hsiao <ch...@mesosphere.io>
> wrote:
>
>> Hi,
>>
>> I'm thinking about extending `process::Executor` with a new `execute()`
>> interface.
>> The need of this new interface surfaced when I'm working on
>> https://issues.apache.org/jira/browse/MESOS-7964
>> Summary:
>> 1. A disk GC might execute multiple `rmdirs` callbacks, and some of them
>> are heavy duty. We don't want to run them on `GarbageCollectorProcess` so
>> that it won't block other events of the process.
>> Currently we do the following:
>> async(rmdirs).onAny(...);
>> 2. `async` puts each `rmdir` callback in an actor. When there are many
>> heavy-duty `rmdirs` callbacks, the actors end up occupying all worker
>> threads and blocking other actors for minutes.
>>
>> Yan suggested me to use `process::Executor` such that:
>> 1. The `rmdirs` callbacks are not executed on `GarbaceGollectorProcess`
>> 2. All `rmdirs` callbacks are executed on a single thread
>> Since the `Executor` class only contains a `defer()` function that
>> returns a `_Deferred` structure,
>> I'm doing the following:
>> executor.defer(rmdirs).operator std::function<Future<Nothing>(
>> )>()().onAny(...)
>>
>> Would it make sense to add another `execute()` function to directly
>> return a `Future`?
>>
>> - Chun-Hung
>>
>>
>
>
> --
> Benjamin Hindman
> Founder of Mesosphere and Co-Creator of Apache Mesos
> Mesosphere Inc. <http://www.mesosphere.io/>
>
> Follow us on Twitter: @mesosphere <http://twitter.com/mesosphere>
>
> [image: All New DC/OS 1.10]
> <http://smart.mesosphere.io/v2/a/dcos1_10_ver3/59b75f7429a6455da34fd9e4-g0PWg/httpsmesosphere.comblogdcos-1_10-kubernetes>
>
>
Re: Adding process::Executor::execute()
Posted by Benjamin Hindman <be...@mesosphere.io>.
Quick clarification: you'll have a single `process::Executor` and queue up
all the rmdirs on that, correct? So you'll still tie up a worker thread,
but only one of them.
Either way it makes sense to add `process::Executor::execute()`. I'm happy
to shepherd that for you Chun, send me a patch!
On Mon, Sep 11, 2017 at 7:32 PM, Chun-Hung Hsiao <ch...@mesosphere.io>
wrote:
> Hi,
>
> I'm thinking about extending `process::Executor` with a new `execute()`
> interface.
> The need of this new interface surfaced when I'm working on
> https://issues.apache.org/jira/browse/MESOS-7964
> Summary:
> 1. A disk GC might execute multiple `rmdirs` callbacks, and some of them
> are heavy duty. We don't want to run them on `GarbageCollectorProcess` so
> that it won't block other events of the process.
> Currently we do the following:
> async(rmdirs).onAny(...);
> 2. `async` puts each `rmdir` callback in an actor. When there are many
> heavy-duty `rmdirs` callbacks, the actors end up occupying all worker
> threads and blocking other actors for minutes.
>
> Yan suggested me to use `process::Executor` such that:
> 1. The `rmdirs` callbacks are not executed on `GarbaceGollectorProcess`
> 2. All `rmdirs` callbacks are executed on a single thread
> Since the `Executor` class only contains a `defer()` function that returns
> a `_Deferred` structure,
> I'm doing the following:
> executor.defer(rmdirs).operator std::function<Future<Nothing>(
> )>()().onAny(...)
>
> Would it make sense to add another `execute()` function to directly return
> a `Future`?
>
> - Chun-Hung
>
>
--
Benjamin Hindman
Founder of Mesosphere and Co-Creator of Apache Mesos
Mesosphere Inc. <http://www.mesosphere.io/>
Follow us on Twitter: @mesosphere <http://twitter.com/mesosphere>