You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@mesos.apache.org by Joris Van Remoortere <jo...@mesosphere.io> on 2016/07/11 07:28:37 UTC
Re: MESOS: Improve the performance of Resources.
+Dev
Hey Klaus,
Using Stout's `Optional` to represent the `optional` concept of a message
in protobuf is definitely a step in the right direction.
Regarding your comment in slack yesterday: From my version of the protobuf
generated code there definitely is dynamic allocation even for scalars.
It looks like in our case there is a minimum of 3 dynamic allocations per
Resource object:
> void Resource::SharedDtor() {
> if (name_ != &::google::protobuf::internal::kEmptyString) {
> delete name_;
> }
> if (role_ != _default_role_) {
> delete role_;
> }
> if (this != default_instance_) {
> delete scalar_;
> delete ranges_;
> delete set_;
> delete reservation_;
> delete disk_;
> delete revocable_;
> }
> }
The 2x number I mentioned came from running some of the existing
benchmarks. I didn't explore further because it didn't have as big an
impact as I had hoped. The first battle is simplifying some of the
algorithms in the Sorter / Resources. Once that is done then the resource
arithmetic will be more of a bounding factor.
I agree with Ben that we should focus on writing some basic benchmarks that
represent the common uses of Resources in the allocator. We should scale
these benchmarks to represent some of the more stressful environments that
could occur. For example, had we had such a benchmark, we would have
realized much earlier on that we needed to aggregate only quantities in the
Sorter, and that using the existing form of Resources would have led to a
grinding halt if a reservation were made on every machine.
Is there a regular call that is scheduled to discuss this? I think there
are some other folks also working on benchmarks and interested in the
discussion.
—
*Joris Van Remoortere*
Mesosphere
On Sun, Jul 10, 2016 at 8:50 PM, Klaus Ma <kl...@gmail.com> wrote:
> + more devs :).
>
> ----
> Da (Klaus), Ma (马达) | PMP® | Software Architect
> Platform OpenSource Technology, STG, IBM GCG
> +86-10-8245 4084 | klaus1982.cn@gmail.com | http://k82.me
>
> On Mon, Jul 11, 2016 at 7:43 AM, Klaus Ma <kl...@gmail.com> wrote:
>
>> Hi Joris,
>>
>> I think `Option` is helpful to the performance improvement, it used
>> `placement new` to avoid dynamic allocation. Suppose you're using Option
>> for optional member in protobuf, and using class instance directly
>> (operator=).
>>
>> I'm adding some benchmark for `Resources`, especially for the `Resources`
>> with Rang, DiskInfo and ReservationInfo
>>
>> Draft PR for Benchmark of Resources:
>> https://github.com/k82cn/mesos/commit/09ca215cb37b1f89eb7d622228a8cf2249eb641c
>>
>>
>> ----
>> Da (Klaus), Ma (马达) | PMP® | Software Architect
>> Platform OpenSource Technology, STG, IBM GCG
>> +86-10-8245 4084 | klaus1982.cn@gmail.com | http://k82.me
>>
>
>
Re: MESOS: Improve the performance of Resources.
Posted by Joris Van Remoortere <jo...@mesosphere.io>.
I used callgrind to instrument the `allocate` function during one of the
iterations of the allocator benchmarks. The ~2x came from the cpu cycle
count reduction.
As I said, that number wasn't within the range I was interested in, so I
didn't spend much time on it. Rather I was using the callgrind results to
find other bottlenecks mentioned earlier in the thread.
—
*Joris Van Remoortere*
Mesosphere
On Tue, Jul 12, 2016 at 2:26 AM, Guangya Liu <gy...@gmail.com> wrote:
> Hi Joris,
>
> For `2x` number, can you please show more detail for how did you do and
> evaluate the test? How did you compare the result? Did you have any test
> code to share?
>
> Thanks,
>
> Guangya
>
> On Mon, Jul 11, 2016 at 6:05 PM, Klaus Ma <kl...@gmail.com> wrote:
>
>> Hi Joris,
>>
>> For `Scalars`, yes, it's also dynamic allocated in
>> `Resources::mutable_scalar()`.
>>
>> For `2x` number, do you have any patch to share? Tested cases this
>> afternoon: add resources including 100 roles (1 CPU for each); the
>> performance is downgrade a lot; so I agree with you to improve some
>> algorithms in Resources/Sorter.
>>
>> For 'basic benchmarks', temporary tracking in my personal github (
>> https://github.com/k82cn/mesos/blob/resources_benchmark/src/tests/resources_tests.cpp).
>> The following cases are in my mind to add:
>> 1. simple resources, e.g. 1 cpu
>> 2. resources with port, e.g. [1-2], [3-4], ... [101-102]
>> 3. resources with reservation, cpus(r1):1;cpus(r2):1; .... cpus(r10):1
>> 4. resources with diskInfo
>> 5. resources with revocableInfo
>>
>> The operators will be +, +=, -, -=, cpus(), contains.
>>
>> I booked a weekly call to discuss for allocaiton performance and sent the
>> invitation to the dev@.
>>
>> If any comments, please let me know.
>>
>> ----
>> Da (Klaus), Ma (马达) | PMP® | Software Architect
>> Platform OpenSource Technology, STG, IBM GCG
>> +86-10-8245 4084 | klaus1982.cn@gmail.com | http://k82.me
>>
>> On Mon, Jul 11, 2016 at 3:28 PM, Joris Van Remoortere <
>> joris@mesosphere.io> wrote:
>>
>>> +Dev
>>>
>>> Hey Klaus,
>>>
>>> Using Stout's `Optional` to represent the `optional` concept of a
>>> message in protobuf is definitely a step in the right direction.
>>> Regarding your comment in slack yesterday: From my version of the
>>> protobuf generated code there definitely is dynamic allocation even for
>>> scalars.
>>>
>>> It looks like in our case there is a minimum of 3 dynamic allocations
>>> per Resource object:
>>>
>>>> void Resource::SharedDtor() {
>>>> if (name_ != &::google::protobuf::internal::kEmptyString) {
>>>> delete name_;
>>>> }
>>>> if (role_ != _default_role_) {
>>>> delete role_;
>>>> }
>>>> if (this != default_instance_) {
>>>> delete scalar_;
>>>> delete ranges_;
>>>> delete set_;
>>>> delete reservation_;
>>>> delete disk_;
>>>> delete revocable_;
>>>> }
>>>> }
>>>
>>>
>>> The 2x number I mentioned came from running some of the existing
>>> benchmarks. I didn't explore further because it didn't have as big an
>>> impact as I had hoped. The first battle is simplifying some of the
>>> algorithms in the Sorter / Resources. Once that is done then the resource
>>> arithmetic will be more of a bounding factor.
>>>
>>> I agree with Ben that we should focus on writing some basic benchmarks
>>> that represent the common uses of Resources in the allocator. We should
>>> scale these benchmarks to represent some of the more stressful environments
>>> that could occur. For example, had we had such a benchmark, we would have
>>> realized much earlier on that we needed to aggregate only quantities in the
>>> Sorter, and that using the existing form of Resources would have led to a
>>> grinding halt if a reservation were made on every machine.
>>>
>>> Is there a regular call that is scheduled to discuss this? I think there
>>> are some other folks also working on benchmarks and interested in the
>>> discussion.
>>>
>>> —
>>> *Joris Van Remoortere*
>>> Mesosphere
>>>
>>> On Sun, Jul 10, 2016 at 8:50 PM, Klaus Ma <kl...@gmail.com>
>>> wrote:
>>>
>>>> + more devs :).
>>>>
>>>> ----
>>>> Da (Klaus), Ma (马达) | PMP® | Software Architect
>>>> Platform OpenSource Technology, STG, IBM GCG
>>>> +86-10-8245 4084 | klaus1982.cn@gmail.com | http://k82.me
>>>>
>>>> On Mon, Jul 11, 2016 at 7:43 AM, Klaus Ma <kl...@gmail.com>
>>>> wrote:
>>>>
>>>>> Hi Joris,
>>>>>
>>>>> I think `Option` is helpful to the performance improvement, it used
>>>>> `placement new` to avoid dynamic allocation. Suppose you're using Option
>>>>> for optional member in protobuf, and using class instance directly
>>>>> (operator=).
>>>>>
>>>>> I'm adding some benchmark for `Resources`, especially for the
>>>>> `Resources` with Rang, DiskInfo and ReservationInfo
>>>>>
>>>>> Draft PR for Benchmark of Resources:
>>>>> https://github.com/k82cn/mesos/commit/09ca215cb37b1f89eb7d622228a8cf2249eb641c
>>>>>
>>>>>
>>>>> ----
>>>>> Da (Klaus), Ma (马达) | PMP® | Software Architect
>>>>> Platform OpenSource Technology, STG, IBM GCG
>>>>> +86-10-8245 4084 | klaus1982.cn@gmail.com | http://k82.me
>>>>>
>>>>
>>>>
>>>
>>
>
Re: MESOS: Improve the performance of Resources.
Posted by Guangya Liu <gy...@gmail.com>.
Hi Joris,
For `2x` number, can you please show more detail for how did you do and
evaluate the test? How did you compare the result? Did you have any test
code to share?
Thanks,
Guangya
On Mon, Jul 11, 2016 at 6:05 PM, Klaus Ma <kl...@gmail.com> wrote:
> Hi Joris,
>
> For `Scalars`, yes, it's also dynamic allocated in
> `Resources::mutable_scalar()`.
>
> For `2x` number, do you have any patch to share? Tested cases this
> afternoon: add resources including 100 roles (1 CPU for each); the
> performance is downgrade a lot; so I agree with you to improve some
> algorithms in Resources/Sorter.
>
> For 'basic benchmarks', temporary tracking in my personal github (
> https://github.com/k82cn/mesos/blob/resources_benchmark/src/tests/resources_tests.cpp).
> The following cases are in my mind to add:
> 1. simple resources, e.g. 1 cpu
> 2. resources with port, e.g. [1-2], [3-4], ... [101-102]
> 3. resources with reservation, cpus(r1):1;cpus(r2):1; .... cpus(r10):1
> 4. resources with diskInfo
> 5. resources with revocableInfo
>
> The operators will be +, +=, -, -=, cpus(), contains.
>
> I booked a weekly call to discuss for allocaiton performance and sent the
> invitation to the dev@.
>
> If any comments, please let me know.
>
> ----
> Da (Klaus), Ma (马达) | PMP® | Software Architect
> Platform OpenSource Technology, STG, IBM GCG
> +86-10-8245 4084 | klaus1982.cn@gmail.com | http://k82.me
>
> On Mon, Jul 11, 2016 at 3:28 PM, Joris Van Remoortere <joris@mesosphere.io
> > wrote:
>
>> +Dev
>>
>> Hey Klaus,
>>
>> Using Stout's `Optional` to represent the `optional` concept of a message
>> in protobuf is definitely a step in the right direction.
>> Regarding your comment in slack yesterday: From my version of the
>> protobuf generated code there definitely is dynamic allocation even for
>> scalars.
>>
>> It looks like in our case there is a minimum of 3 dynamic allocations per
>> Resource object:
>>
>>> void Resource::SharedDtor() {
>>> if (name_ != &::google::protobuf::internal::kEmptyString) {
>>> delete name_;
>>> }
>>> if (role_ != _default_role_) {
>>> delete role_;
>>> }
>>> if (this != default_instance_) {
>>> delete scalar_;
>>> delete ranges_;
>>> delete set_;
>>> delete reservation_;
>>> delete disk_;
>>> delete revocable_;
>>> }
>>> }
>>
>>
>> The 2x number I mentioned came from running some of the existing
>> benchmarks. I didn't explore further because it didn't have as big an
>> impact as I had hoped. The first battle is simplifying some of the
>> algorithms in the Sorter / Resources. Once that is done then the resource
>> arithmetic will be more of a bounding factor.
>>
>> I agree with Ben that we should focus on writing some basic benchmarks
>> that represent the common uses of Resources in the allocator. We should
>> scale these benchmarks to represent some of the more stressful environments
>> that could occur. For example, had we had such a benchmark, we would have
>> realized much earlier on that we needed to aggregate only quantities in the
>> Sorter, and that using the existing form of Resources would have led to a
>> grinding halt if a reservation were made on every machine.
>>
>> Is there a regular call that is scheduled to discuss this? I think there
>> are some other folks also working on benchmarks and interested in the
>> discussion.
>>
>> —
>> *Joris Van Remoortere*
>> Mesosphere
>>
>> On Sun, Jul 10, 2016 at 8:50 PM, Klaus Ma <kl...@gmail.com> wrote:
>>
>>> + more devs :).
>>>
>>> ----
>>> Da (Klaus), Ma (马达) | PMP® | Software Architect
>>> Platform OpenSource Technology, STG, IBM GCG
>>> +86-10-8245 4084 | klaus1982.cn@gmail.com | http://k82.me
>>>
>>> On Mon, Jul 11, 2016 at 7:43 AM, Klaus Ma <kl...@gmail.com>
>>> wrote:
>>>
>>>> Hi Joris,
>>>>
>>>> I think `Option` is helpful to the performance improvement, it used
>>>> `placement new` to avoid dynamic allocation. Suppose you're using Option
>>>> for optional member in protobuf, and using class instance directly
>>>> (operator=).
>>>>
>>>> I'm adding some benchmark for `Resources`, especially for the
>>>> `Resources` with Rang, DiskInfo and ReservationInfo
>>>>
>>>> Draft PR for Benchmark of Resources:
>>>> https://github.com/k82cn/mesos/commit/09ca215cb37b1f89eb7d622228a8cf2249eb641c
>>>>
>>>>
>>>> ----
>>>> Da (Klaus), Ma (马达) | PMP® | Software Architect
>>>> Platform OpenSource Technology, STG, IBM GCG
>>>> +86-10-8245 4084 | klaus1982.cn@gmail.com | http://k82.me
>>>>
>>>
>>>
>>
>
Re: MESOS: Improve the performance of Resources.
Posted by Klaus Ma <kl...@gmail.com>.
Hi Joris,
For `Scalars`, yes, it's also dynamic allocated in
`Resources::mutable_scalar()`.
For `2x` number, do you have any patch to share? Tested cases this
afternoon: add resources including 100 roles (1 CPU for each); the
performance is downgrade a lot; so I agree with you to improve some
algorithms in Resources/Sorter.
For 'basic benchmarks', temporary tracking in my personal github (
https://github.com/k82cn/mesos/blob/resources_benchmark/src/tests/resources_tests.cpp).
The following cases are in my mind to add:
1. simple resources, e.g. 1 cpu
2. resources with port, e.g. [1-2], [3-4], ... [101-102]
3. resources with reservation, cpus(r1):1;cpus(r2):1; .... cpus(r10):1
4. resources with diskInfo
5. resources with revocableInfo
The operators will be +, +=, -, -=, cpus(), contains.
I booked a weekly call to discuss for allocaiton performance and sent the
invitation to the dev@.
If any comments, please let me know.
----
Da (Klaus), Ma (马达) | PMP® | Software Architect
Platform OpenSource Technology, STG, IBM GCG
+86-10-8245 4084 | klaus1982.cn@gmail.com | http://k82.me
On Mon, Jul 11, 2016 at 3:28 PM, Joris Van Remoortere <jo...@mesosphere.io>
wrote:
> +Dev
>
> Hey Klaus,
>
> Using Stout's `Optional` to represent the `optional` concept of a message
> in protobuf is definitely a step in the right direction.
> Regarding your comment in slack yesterday: From my version of the protobuf
> generated code there definitely is dynamic allocation even for scalars.
>
> It looks like in our case there is a minimum of 3 dynamic allocations per
> Resource object:
>
>> void Resource::SharedDtor() {
>> if (name_ != &::google::protobuf::internal::kEmptyString) {
>> delete name_;
>> }
>> if (role_ != _default_role_) {
>> delete role_;
>> }
>> if (this != default_instance_) {
>> delete scalar_;
>> delete ranges_;
>> delete set_;
>> delete reservation_;
>> delete disk_;
>> delete revocable_;
>> }
>> }
>
>
> The 2x number I mentioned came from running some of the existing
> benchmarks. I didn't explore further because it didn't have as big an
> impact as I had hoped. The first battle is simplifying some of the
> algorithms in the Sorter / Resources. Once that is done then the resource
> arithmetic will be more of a bounding factor.
>
> I agree with Ben that we should focus on writing some basic benchmarks
> that represent the common uses of Resources in the allocator. We should
> scale these benchmarks to represent some of the more stressful environments
> that could occur. For example, had we had such a benchmark, we would have
> realized much earlier on that we needed to aggregate only quantities in the
> Sorter, and that using the existing form of Resources would have led to a
> grinding halt if a reservation were made on every machine.
>
> Is there a regular call that is scheduled to discuss this? I think there
> are some other folks also working on benchmarks and interested in the
> discussion.
>
> —
> *Joris Van Remoortere*
> Mesosphere
>
> On Sun, Jul 10, 2016 at 8:50 PM, Klaus Ma <kl...@gmail.com> wrote:
>
>> + more devs :).
>>
>> ----
>> Da (Klaus), Ma (马达) | PMP® | Software Architect
>> Platform OpenSource Technology, STG, IBM GCG
>> +86-10-8245 4084 | klaus1982.cn@gmail.com | http://k82.me
>>
>> On Mon, Jul 11, 2016 at 7:43 AM, Klaus Ma <kl...@gmail.com> wrote:
>>
>>> Hi Joris,
>>>
>>> I think `Option` is helpful to the performance improvement, it used
>>> `placement new` to avoid dynamic allocation. Suppose you're using Option
>>> for optional member in protobuf, and using class instance directly
>>> (operator=).
>>>
>>> I'm adding some benchmark for `Resources`, especially for the
>>> `Resources` with Rang, DiskInfo and ReservationInfo
>>>
>>> Draft PR for Benchmark of Resources:
>>> https://github.com/k82cn/mesos/commit/09ca215cb37b1f89eb7d622228a8cf2249eb641c
>>>
>>>
>>> ----
>>> Da (Klaus), Ma (马达) | PMP® | Software Architect
>>> Platform OpenSource Technology, STG, IBM GCG
>>> +86-10-8245 4084 | klaus1982.cn@gmail.com | http://k82.me
>>>
>>
>>
>