You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@mesos.apache.org by uyang Zhang W <wu...@bytedance.com> on 2019/07/10 00:06:42 UTC
Simulate Mesos for Large Scale Framework Test
Dear all,
I developed a framework and would like to test the scheduling algorithm by simulating it in a large scale environment.
I found an old doc at https://docs.google.com/document/d/1Ygq9MPWrqcQLf0J-mraVeEIRYk3xNXXQ0xRHFqTiXsQ/edit# <https://docs.google.com/document/d/1Ygq9MPWrqcQLf0J-mraVeEIRYk3xNXXQ0xRHFqTiXsQ/edit#>. It basically satisfies all the requirements. However, I cannot find any implementations.
To specify, I have the following expectations.
1) Simulating 1k nodes with heterogeneous resources.
2) Loading job traces with defined running time.
3) Test the scheduling algorithm in this simulated environment.
Can you please give a pointer to do that?
Best,
Wuyang
Re: [External Email] Re: Simulate Mesos for Large Scale Framework Test
Posted by uyang Zhang W <wu...@bytedance.com>.
Hi Meng,
Thanks a lot for providing detailed solutions! It will be really helpful.
I will try both of them today and see how they fit to the current demand.
Best,
Wuyang
On Wed, Jul 10, 2019 at 5:45 PM Meng Zhu <mz...@mesosphere.com> wrote:
> Hi Wuyang:
>
> I am not sure if the design was actually implemented. At least, I am not
> aware of such simulators.
>
> If you want to simulate with your own framework, one possibility is to run
> multiple agents with fake resources on the same (or a few) node.
>
> Alternatively, we have a lightweight benchmark test suite inside Mesos for
> the allocator:
> https://github.com/apache/mesos/blob/master/src/tests/hierarchical_allocator_benchmarks.cpp#L128
> You can easily specify agent and framework
> <https://github.com/apache/mesos/blob/dcd73437549413790751d1ff127989dbb29bd753/src/tests/hierarchical_allocator_benchmarks.cpp#L70-L116> profiles
> and spawn them. However, currently, it is not possible to directly plug-in
> your own framework. Here is an example simulation
> <https://github.com/apache/mesos/blob/dcd73437549413790751d1ff127989dbb29bd753/src/tests/hierarchical_allocator_benchmarks.cpp#L324-L328> that
> uses the fixture.
> You can extend the framework profile and fixture to either encode more
> behaviors per your preference.
>
> In both cases, the Mesos default allocator is used. You can change the
> allocation algorithm by using your own allocator implementation.
>
> Hope it helps.
>
> -Meng
>
> On Tue, Jul 9, 2019 at 5:06 PM uyang Zhang W <wu...@bytedance.com>
> wrote:
>
>> Dear all,
>>
>> I developed a framework and would like to test the scheduling algorithm
>> by simulating it in a large scale environment.
>>
>> I found an old doc at
>> https://docs.google.com/document/d/1Ygq9MPWrqcQLf0J-mraVeEIRYk3xNXXQ0xRHFqTiXsQ/edit#.
>> It basically satisfies all the requirements. However, I cannot find any
>> implementations.
>>
>> To specify, I have the following expectations.
>> 1) Simulating 1k nodes with heterogeneous resources.
>> 2) Loading job traces with defined running time.
>> 3) Test the scheduling algorithm in this simulated environment.
>>
>> Can you please give a pointer to do that?
>>
>> Best,
>> Wuyang
>>
>
Re: Simulate Mesos for Large Scale Framework Test
Posted by Meng Zhu <mz...@mesosphere.com>.
Hi Wuyang:
I am not sure if the design was actually implemented. At least, I am not
aware of such simulators.
If you want to simulate with your own framework, one possibility is to run
multiple agents with fake resources on the same (or a few) node.
Alternatively, we have a lightweight benchmark test suite inside Mesos for
the allocator:
https://github.com/apache/mesos/blob/master/src/tests/hierarchical_allocator_benchmarks.cpp#L128
You can easily specify agent and framework
<https://github.com/apache/mesos/blob/dcd73437549413790751d1ff127989dbb29bd753/src/tests/hierarchical_allocator_benchmarks.cpp#L70-L116>
profiles
and spawn them. However, currently, it is not possible to directly plug-in
your own framework. Here is an example simulation
<https://github.com/apache/mesos/blob/dcd73437549413790751d1ff127989dbb29bd753/src/tests/hierarchical_allocator_benchmarks.cpp#L324-L328>
that
uses the fixture.
You can extend the framework profile and fixture to either encode more
behaviors per your preference.
In both cases, the Mesos default allocator is used. You can change the
allocation algorithm by using your own allocator implementation.
Hope it helps.
-Meng
On Tue, Jul 9, 2019 at 5:06 PM uyang Zhang W <wu...@bytedance.com>
wrote:
> Dear all,
>
> I developed a framework and would like to test the scheduling algorithm by
> simulating it in a large scale environment.
>
> I found an old doc at
> https://docs.google.com/document/d/1Ygq9MPWrqcQLf0J-mraVeEIRYk3xNXXQ0xRHFqTiXsQ/edit#.
> It basically satisfies all the requirements. However, I cannot find any
> implementations.
>
> To specify, I have the following expectations.
> 1) Simulating 1k nodes with heterogeneous resources.
> 2) Loading job traces with defined running time.
> 3) Test the scheduling algorithm in this simulated environment.
>
> Can you please give a pointer to do that?
>
> Best,
> Wuyang
>