Re: [External Email] Re: Simulate Mesos for Large Scale Framework Test

[uyang Zhang W <wu...@bytedance.com>]

Hi Meng,

Thanks a lot for providing detailed solutions! It will be really helpful.

I will try both of them today and see how they fit to the current demand.

Best,
Wuyang

On Wed, Jul 10, 2019 at 5:45 PM Meng Zhu <mz...@mesosphere.com> wrote:

> Hi Wuyang:
>
> I am not sure if the design was actually implemented. At least, I am not
> aware of such simulators.
>
> If you want to simulate with your own framework, one possibility is to run
> multiple agents with fake resources on the same (or a few) node.
>
> Alternatively, we have a lightweight benchmark test suite inside Mesos for
> the allocator:
> https://github.com/apache/mesos/blob/master/src/tests/hierarchical_allocator_benchmarks.cpp#L128
> You can easily specify agent and framework
> <https://github.com/apache/mesos/blob/dcd73437549413790751d1ff127989dbb29bd753/src/tests/hierarchical_allocator_benchmarks.cpp#L70-L116> profiles
> and spawn them. However, currently, it is not possible to directly plug-in
> your own framework. Here is an example simulation
> <https://github.com/apache/mesos/blob/dcd73437549413790751d1ff127989dbb29bd753/src/tests/hierarchical_allocator_benchmarks.cpp#L324-L328> that
> uses the fixture.
> You can extend the framework profile and fixture to either encode more
> behaviors per your preference.
>
> In both cases, the Mesos default allocator is used. You can change the
> allocation algorithm by using your own allocator implementation.
>
> Hope it helps.
>
> -Meng
>
> On Tue, Jul 9, 2019 at 5:06 PM uyang Zhang W <wu...@bytedance.com>
> wrote:
>
>> Dear all,
>>
>> I developed a framework and would like to test the scheduling algorithm
>> by simulating it in a large scale environment.
>>
>> I found an old doc at
>> https://docs.google.com/document/d/1Ygq9MPWrqcQLf0J-mraVeEIRYk3xNXXQ0xRHFqTiXsQ/edit#.
>> It basically satisfies all the requirements. However, I cannot find any
>> implementations.
>>
>> To specify, I have the following expectations.
>> 1) Simulating 1k nodes with heterogeneous resources.
>> 2) Loading  job traces with defined running time.
>> 3) Test the scheduling algorithm in this simulated environment.
>>
>> Can you please give a pointer to do that?
>>
>> Best,
>> Wuyang
>>
>