You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@mesos.apache.org by Meng Zhu <mz...@mesosphere.io> on 2018/10/20 06:05:37 UTC
Review Request 69098: Added a benchmark to compare quota and nonquota
allocation performance.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69098/
-----------------------------------------------------------
Review request for mesos, Benjamin Mahler and Gastón Kleiman.
Bugs: MESOS-6630
https://issues.apache.org/jira/browse/MESOS-6630
Repository: mesos
Description
-------
This benchmark evaluates the performance difference between nonquota
and quota settings. In both settings, the same allocations are made
for fair comparison.
Diffs
-----
src/tests/hierarchical_allocator_benchmarks.cpp bf9167b63747f7b8a402d950947028436307082a
Diff: https://reviews.apache.org/r/69098/diff/1/
Testing
-------
Non-optimized build benchmark output: (more comprehensive evaluation result across versions coming soon)
[ RUN ] NonQuotaVsQuotaParam/HierarchicalAllocator_BENCHMARK_WithNonQuotaVsQuotaParam.NonQuotaVsQuota/0
Added 20 agents in 7.75017ms
Added 20 frameworks in 18.32336ms
Benchmark setup: 20 agents, 10 roles, 20 frameworks
Start nonQuota allocation
Made 20 allocations in 17.017677ms for nonquota roles
Start Quota allocation
Made 20 allocations in 50.428526ms for quota roles
[ OK ] NonQuotaVsQuotaParam/HierarchicalAllocator_BENCHMARK_WithNonQuotaVsQuotaParam.NonQuotaVsQuota/0 (99 ms)
[ RUN ] NonQuotaVsQuotaParam/HierarchicalAllocator_BENCHMARK_WithNonQuotaVsQuotaParam.NonQuotaVsQuota/1
Added 200 agents in 45.158088ms
Added 200 frameworks in 649.806597ms
Benchmark setup: 200 agents, 100 roles, 200 frameworks
Start nonQuota allocation
Made 200 allocations in 293.943471ms for nonquota roles
Start Quota allocation
Made 200 allocations in 1.945455585secs for quota roles
[ OK ] NonQuotaVsQuotaParam/HierarchicalAllocator_BENCHMARK_WithNonQuotaVsQuotaParam.NonQuotaVsQuota/1 (2974 ms)
[ RUN ] NonQuotaVsQuotaParam/HierarchicalAllocator_BENCHMARK_WithNonQuotaVsQuotaParam.NonQuotaVsQuota/2
Added 2000 agents in 310.070738ms
Added 2000 frameworks in 49.139360942secs
Benchmark setup: 2000 agents, 1000 roles, 2000 frameworks
Start nonQuota allocation
Made 2000 allocations in 19.216226126secs for nonquota roles
Start Quota allocation
Made 2000 allocations in 3.00175930081667mins for quota roles
[ OK ] NonQuotaVsQuotaParam/HierarchicalAllocator_BENCHMARK_WithNonQuotaVsQuotaParam.NonQuotaVsQuota/2 (250392 ms)
Thanks,
Meng Zhu
Re: Review Request 69098: Added a benchmark to compare quota and
nonquota allocation performance.
Posted by Mesos Reviewbot Windows <re...@mesos.apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69098/#review209819
-----------------------------------------------------------
PASS: Mesos patch 69098 was successfully built and tested.
Reviews applied: `['69092', '69093', '69094', '69095', '69096', '69097', '69098']`
All the build artifacts available at: http://dcos-win.westus2.cloudapp.azure.com/artifacts/mesos-reviewbot-testing/2497/mesos-review-69098
- Mesos Reviewbot Windows
On Oct. 20, 2018, 6:05 a.m., Meng Zhu wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/69098/
> -----------------------------------------------------------
>
> (Updated Oct. 20, 2018, 6:05 a.m.)
>
>
> Review request for mesos, Benjamin Mahler and Gastón Kleiman.
>
>
> Bugs: MESOS-6630
> https://issues.apache.org/jira/browse/MESOS-6630
>
>
> Repository: mesos
>
>
> Description
> -------
>
> This benchmark evaluates the performance difference between nonquota
> and quota settings. In both settings, the same allocations are made
> for fair comparison.
>
>
> Diffs
> -----
>
> src/tests/hierarchical_allocator_benchmarks.cpp bf9167b63747f7b8a402d950947028436307082a
>
>
> Diff: https://reviews.apache.org/r/69098/diff/1/
>
>
> Testing
> -------
>
> Non-optimized build benchmark output: (more comprehensive evaluation result across versions coming soon)
>
> [ RUN ] NonQuotaVsQuotaParam/HierarchicalAllocator_BENCHMARK_WithNonQuotaVsQuotaParam.NonQuotaVsQuota/0
> Added 20 agents in 7.75017ms
> Added 20 frameworks in 18.32336ms
> Benchmark setup: 20 agents, 10 roles, 20 frameworks
> Start nonQuota allocation
> Made 20 allocations in 17.017677ms for nonquota roles
> Start Quota allocation
> Made 20 allocations in 50.428526ms for quota roles
> [ OK ] NonQuotaVsQuotaParam/HierarchicalAllocator_BENCHMARK_WithNonQuotaVsQuotaParam.NonQuotaVsQuota/0 (99 ms)
>
>
> [ RUN ] NonQuotaVsQuotaParam/HierarchicalAllocator_BENCHMARK_WithNonQuotaVsQuotaParam.NonQuotaVsQuota/1
> Added 200 agents in 45.158088ms
> Added 200 frameworks in 649.806597ms
> Benchmark setup: 200 agents, 100 roles, 200 frameworks
> Start nonQuota allocation
> Made 200 allocations in 293.943471ms for nonquota roles
> Start Quota allocation
> Made 200 allocations in 1.945455585secs for quota roles
> [ OK ] NonQuotaVsQuotaParam/HierarchicalAllocator_BENCHMARK_WithNonQuotaVsQuotaParam.NonQuotaVsQuota/1 (2974 ms)
>
>
> [ RUN ] NonQuotaVsQuotaParam/HierarchicalAllocator_BENCHMARK_WithNonQuotaVsQuotaParam.NonQuotaVsQuota/2
> Added 2000 agents in 310.070738ms
> Added 2000 frameworks in 49.139360942secs
> Benchmark setup: 2000 agents, 1000 roles, 2000 frameworks
> Start nonQuota allocation
> Made 2000 allocations in 19.216226126secs for nonquota roles
> Start Quota allocation
> Made 2000 allocations in 3.00175930081667mins for quota roles
> [ OK ] NonQuotaVsQuotaParam/HierarchicalAllocator_BENCHMARK_WithNonQuotaVsQuotaParam.NonQuotaVsQuota/2 (250392 ms)
>
>
> Thanks,
>
> Meng Zhu
>
>
Re: Review Request 69098: Added a benchmark to compare quota and
nonquota allocation performance.
Posted by Mesos Reviewbot <re...@mesos.apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69098/#review209826
-----------------------------------------------------------
Patch looks great!
Reviews applied: [69092, 69093, 69094, 69095, 69096, 69097, 69098]
Passed command: export OS='ubuntu:14.04' BUILDTOOL='autotools' COMPILER='gcc' CONFIGURATION='--verbose --disable-libtool-wrappers' ENVIRONMENT='GLOG_v=1 MESOS_VERBOSE=1'; ./support/docker-build.sh
- Mesos Reviewbot
On Oct. 20, 2018, 8:05 a.m., Meng Zhu wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/69098/
> -----------------------------------------------------------
>
> (Updated Oct. 20, 2018, 8:05 a.m.)
>
>
> Review request for mesos, Benjamin Mahler and Gastón Kleiman.
>
>
> Bugs: MESOS-6630
> https://issues.apache.org/jira/browse/MESOS-6630
>
>
> Repository: mesos
>
>
> Description
> -------
>
> This benchmark evaluates the performance difference between nonquota
> and quota settings. In both settings, the same allocations are made
> for fair comparison.
>
>
> Diffs
> -----
>
> src/tests/hierarchical_allocator_benchmarks.cpp bf9167b63747f7b8a402d950947028436307082a
>
>
> Diff: https://reviews.apache.org/r/69098/diff/1/
>
>
> Testing
> -------
>
> Non-optimized build benchmark output: (more comprehensive evaluation result across versions coming soon)
>
> [ RUN ] NonQuotaVsQuotaParam/HierarchicalAllocator_BENCHMARK_WithNonQuotaVsQuotaParam.NonQuotaVsQuota/0
> Added 20 agents in 7.75017ms
> Added 20 frameworks in 18.32336ms
> Benchmark setup: 20 agents, 10 roles, 20 frameworks
> Start nonQuota allocation
> Made 20 allocations in 17.017677ms for nonquota roles
> Start Quota allocation
> Made 20 allocations in 50.428526ms for quota roles
> [ OK ] NonQuotaVsQuotaParam/HierarchicalAllocator_BENCHMARK_WithNonQuotaVsQuotaParam.NonQuotaVsQuota/0 (99 ms)
>
>
> [ RUN ] NonQuotaVsQuotaParam/HierarchicalAllocator_BENCHMARK_WithNonQuotaVsQuotaParam.NonQuotaVsQuota/1
> Added 200 agents in 45.158088ms
> Added 200 frameworks in 649.806597ms
> Benchmark setup: 200 agents, 100 roles, 200 frameworks
> Start nonQuota allocation
> Made 200 allocations in 293.943471ms for nonquota roles
> Start Quota allocation
> Made 200 allocations in 1.945455585secs for quota roles
> [ OK ] NonQuotaVsQuotaParam/HierarchicalAllocator_BENCHMARK_WithNonQuotaVsQuotaParam.NonQuotaVsQuota/1 (2974 ms)
>
>
> [ RUN ] NonQuotaVsQuotaParam/HierarchicalAllocator_BENCHMARK_WithNonQuotaVsQuotaParam.NonQuotaVsQuota/2
> Added 2000 agents in 310.070738ms
> Added 2000 frameworks in 49.139360942secs
> Benchmark setup: 2000 agents, 1000 roles, 2000 frameworks
> Start nonQuota allocation
> Made 2000 allocations in 19.216226126secs for nonquota roles
> Start Quota allocation
> Made 2000 allocations in 3.00175930081667mins for quota roles
> [ OK ] NonQuotaVsQuotaParam/HierarchicalAllocator_BENCHMARK_WithNonQuotaVsQuotaParam.NonQuotaVsQuota/2 (250392 ms)
>
>
> Thanks,
>
> Meng Zhu
>
>
Re: Review Request 69098: Added a benchmark to compare quota and
nonquota allocation performance.
Posted by Mesos Reviewbot Windows <re...@mesos.apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69098/#review211073
-----------------------------------------------------------
PASS: Mesos patch 69098 was successfully built and tested.
Reviews applied: `['69092', '69093', '69094', '69096', '69097', '69098']`
All the build artifacts available at: http://dcos-win.westus2.cloudapp.azure.com/artifacts/mesos-reviewbot-testing/2673/mesos-review-69098
- Mesos Reviewbot Windows
On Oct. 22, 2018, 3:39 p.m., Meng Zhu wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/69098/
> -----------------------------------------------------------
>
> (Updated Oct. 22, 2018, 3:39 p.m.)
>
>
> Review request for mesos, Benjamin Mahler and Gastón Kleiman.
>
>
> Bugs: MESOS-6630
> https://issues.apache.org/jira/browse/MESOS-6630
>
>
> Repository: mesos
>
>
> Description
> -------
>
> This benchmark evaluates the performance difference between nonquota
> and quota settings. In both settings, the same allocations are made
> for fair comparison. In particular, since the agent will always be
> allocated as a whole in nonquota settings, we should also avoid
> agent chopping in quota setting as well. Thus in this benchmark,
> quotas are only set to be multiples of whole agent resources.
> This is also why we have this dedicated benchmark for comparison
> rather than extending the existing quota benchmarks (which involves
> agent chopping).
>
>
> Diffs
> -----
>
> src/tests/hierarchical_allocator_benchmarks.cpp 6c6330e8cdbc705be322a7e2445b295c35ee6952
>
>
> Diff: https://reviews.apache.org/r/69098/diff/2/
>
>
> Testing
> -------
>
> Result from optimized build on a multicore 2.2GHz machine:
>
> Benchmark setup: 20 agents, 10 roles, 20 frameworks
> Made 20 allocations in 3.508676ms for nonquota roles
> Made 20 allocations in 7.901269ms for quota roles
>
> Benchmark setup: 200 agents, 100 roles, 200 frameworks
> Made 200 allocations in 63.407391ms for nonquota roles
> Made 200 allocations in 279.002319ms for quota roles
>
> Benchmark setup: 2000 agents, 1000 roles, 2000 frameworks
> Made 2000 allocations in 4.003802373secs for nonquota roles
> Made 2000 allocations in 20.503188535secs for quota roles
>
>
> Thanks,
>
> Meng Zhu
>
>
Re: Review Request 69098: Added a benchmark to compare quota and
nonquota allocation performance.
Posted by Meng Zhu <mz...@mesosphere.io>.
> On Nov. 2, 2018, 3:23 p.m., Benjamin Mahler wrote:
> > Hm.. why is this one not just extending the one you added in the previous review?
>
> Meng Zhu wrote:
> This benchmark compares quota vs. non-quota. For non-quota setting, there will be no chopping. So for fair comparison, quota setting should have no chopping either. The previous test focuses quota only and chopping is an important part of it.
>
> Benjamin Mahler wrote:
> Ok, can you spell it out a bit more in the comments and description, it just seems to say the same allocations are made, but would be good to explain how that was done and why other benchmarks might fail to do it?
Done.
- Meng
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69098/#review210294
-----------------------------------------------------------
On Oct. 22, 2018, 3:39 p.m., Meng Zhu wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/69098/
> -----------------------------------------------------------
>
> (Updated Oct. 22, 2018, 3:39 p.m.)
>
>
> Review request for mesos, Benjamin Mahler and Gastón Kleiman.
>
>
> Bugs: MESOS-6630
> https://issues.apache.org/jira/browse/MESOS-6630
>
>
> Repository: mesos
>
>
> Description
> -------
>
> This benchmark evaluates the performance difference between nonquota
> and quota settings. In both settings, the same allocations are made
> for fair comparison.
>
>
> Diffs
> -----
>
> src/tests/hierarchical_allocator_benchmarks.cpp bf9167b63747f7b8a402d950947028436307082a
>
>
> Diff: https://reviews.apache.org/r/69098/diff/1/
>
>
> Testing
> -------
>
> Result from optimized build on a multicore 2.2GHz machine:
>
> Benchmark setup: 20 agents, 10 roles, 20 frameworks
> Made 20 allocations in 3.508676ms for nonquota roles
> Made 20 allocations in 7.901269ms for quota roles
>
> Benchmark setup: 200 agents, 100 roles, 200 frameworks
> Made 200 allocations in 63.407391ms for nonquota roles
> Made 200 allocations in 279.002319ms for quota roles
>
> Benchmark setup: 2000 agents, 1000 roles, 2000 frameworks
> Made 2000 allocations in 4.003802373secs for nonquota roles
> Made 2000 allocations in 20.503188535secs for quota roles
>
>
> Thanks,
>
> Meng Zhu
>
>
Re: Review Request 69098: Added a benchmark to compare quota and
nonquota allocation performance.
Posted by Meng Zhu <mz...@mesosphere.io>.
> On Nov. 2, 2018, 3:23 p.m., Benjamin Mahler wrote:
> > Hm.. why is this one not just extending the one you added in the previous review?
This benchmark compares quota vs. non-quota. For non-quota setting, there will be no chopping. So for fair comparison, quota setting should have no chopping either. The previous test focuses quota only and chopping is an important part of it.
- Meng
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69098/#review210294
-----------------------------------------------------------
On Oct. 22, 2018, 3:39 p.m., Meng Zhu wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/69098/
> -----------------------------------------------------------
>
> (Updated Oct. 22, 2018, 3:39 p.m.)
>
>
> Review request for mesos, Benjamin Mahler and Gastón Kleiman.
>
>
> Bugs: MESOS-6630
> https://issues.apache.org/jira/browse/MESOS-6630
>
>
> Repository: mesos
>
>
> Description
> -------
>
> This benchmark evaluates the performance difference between nonquota
> and quota settings. In both settings, the same allocations are made
> for fair comparison.
>
>
> Diffs
> -----
>
> src/tests/hierarchical_allocator_benchmarks.cpp bf9167b63747f7b8a402d950947028436307082a
>
>
> Diff: https://reviews.apache.org/r/69098/diff/1/
>
>
> Testing
> -------
>
> Result from optimized build on a multicore 2.2GHz machine:
>
> Benchmark setup: 20 agents, 10 roles, 20 frameworks
> Made 20 allocations in 3.508676ms for nonquota roles
> Made 20 allocations in 7.901269ms for quota roles
>
> Benchmark setup: 200 agents, 100 roles, 200 frameworks
> Made 200 allocations in 63.407391ms for nonquota roles
> Made 200 allocations in 279.002319ms for quota roles
>
> Benchmark setup: 2000 agents, 1000 roles, 2000 frameworks
> Made 2000 allocations in 4.003802373secs for nonquota roles
> Made 2000 allocations in 20.503188535secs for quota roles
>
>
> Thanks,
>
> Meng Zhu
>
>
Re: Review Request 69098: Added a benchmark to compare quota and
nonquota allocation performance.
Posted by Benjamin Mahler <bm...@apache.org>.
> On Nov. 2, 2018, 10:23 p.m., Benjamin Mahler wrote:
> > Hm.. why is this one not just extending the one you added in the previous review?
>
> Meng Zhu wrote:
> This benchmark compares quota vs. non-quota. For non-quota setting, there will be no chopping. So for fair comparison, quota setting should have no chopping either. The previous test focuses quota only and chopping is an important part of it.
Ok, can you spell it out a bit more in the comments and description, it just seems to say the same allocations are made, but would be good to explain how that was done and why other benchmarks might fail to do it?
- Benjamin
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69098/#review210294
-----------------------------------------------------------
On Oct. 22, 2018, 10:39 p.m., Meng Zhu wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/69098/
> -----------------------------------------------------------
>
> (Updated Oct. 22, 2018, 10:39 p.m.)
>
>
> Review request for mesos, Benjamin Mahler and Gastón Kleiman.
>
>
> Bugs: MESOS-6630
> https://issues.apache.org/jira/browse/MESOS-6630
>
>
> Repository: mesos
>
>
> Description
> -------
>
> This benchmark evaluates the performance difference between nonquota
> and quota settings. In both settings, the same allocations are made
> for fair comparison.
>
>
> Diffs
> -----
>
> src/tests/hierarchical_allocator_benchmarks.cpp bf9167b63747f7b8a402d950947028436307082a
>
>
> Diff: https://reviews.apache.org/r/69098/diff/1/
>
>
> Testing
> -------
>
> Result from optimized build on a multicore 2.2GHz machine:
>
> Benchmark setup: 20 agents, 10 roles, 20 frameworks
> Made 20 allocations in 3.508676ms for nonquota roles
> Made 20 allocations in 7.901269ms for quota roles
>
> Benchmark setup: 200 agents, 100 roles, 200 frameworks
> Made 200 allocations in 63.407391ms for nonquota roles
> Made 200 allocations in 279.002319ms for quota roles
>
> Benchmark setup: 2000 agents, 1000 roles, 2000 frameworks
> Made 2000 allocations in 4.003802373secs for nonquota roles
> Made 2000 allocations in 20.503188535secs for quota roles
>
>
> Thanks,
>
> Meng Zhu
>
>
Re: Review Request 69098: Added a benchmark to compare quota and
nonquota allocation performance.
Posted by Benjamin Mahler <bm...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69098/#review210294
-----------------------------------------------------------
Hm.. why is this one not just extending the one you added in the previous review?
- Benjamin Mahler
On Oct. 22, 2018, 10:39 p.m., Meng Zhu wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/69098/
> -----------------------------------------------------------
>
> (Updated Oct. 22, 2018, 10:39 p.m.)
>
>
> Review request for mesos, Benjamin Mahler and Gastón Kleiman.
>
>
> Bugs: MESOS-6630
> https://issues.apache.org/jira/browse/MESOS-6630
>
>
> Repository: mesos
>
>
> Description
> -------
>
> This benchmark evaluates the performance difference between nonquota
> and quota settings. In both settings, the same allocations are made
> for fair comparison.
>
>
> Diffs
> -----
>
> src/tests/hierarchical_allocator_benchmarks.cpp bf9167b63747f7b8a402d950947028436307082a
>
>
> Diff: https://reviews.apache.org/r/69098/diff/1/
>
>
> Testing
> -------
>
> Result from optimized build on a multicore 2.2GHz machine:
>
> Benchmark setup: 20 agents, 10 roles, 20 frameworks
> Made 20 allocations in 3.508676ms for nonquota roles
> Made 20 allocations in 7.901269ms for quota roles
>
> Benchmark setup: 200 agents, 100 roles, 200 frameworks
> Made 200 allocations in 63.407391ms for nonquota roles
> Made 200 allocations in 279.002319ms for quota roles
>
> Benchmark setup: 2000 agents, 1000 roles, 2000 frameworks
> Made 2000 allocations in 4.003802373secs for nonquota roles
> Made 2000 allocations in 20.503188535secs for quota roles
>
>
> Thanks,
>
> Meng Zhu
>
>
Re: Review Request 69098: Added a benchmark to compare quota and
nonquota allocation performance.
Posted by Meng Zhu <mz...@mesosphere.io>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69098/
-----------------------------------------------------------
(Updated Dec. 6, 2018, 2:42 p.m.)
Review request for mesos, Benjamin Mahler and Gastón Kleiman.
Changes
-------
Addressed Ben's comment.
Bugs: MESOS-6630
https://issues.apache.org/jira/browse/MESOS-6630
Repository: mesos
Description
-------
This benchmark evaluates the performance difference between nonquota
and quota settings. In both settings, the same allocations are made
for fair comparison. In particular, since the agent will always be
allocated as a whole in nonquota settings, we should also avoid
agent chopping in quota setting as well. Thus in this benchmark,
quotas are only set to be multiples of whole agent resources.
This is also why we have this dedicated benchmark for comparison
rather than extending the existing quota benchmarks (which involves
agent chopping).
Diffs (updated)
-----
src/tests/hierarchical_allocator_benchmarks.cpp 6c6330e8cdbc705be322a7e2445b295c35ee6952
Diff: https://reviews.apache.org/r/69098/diff/3/
Changes: https://reviews.apache.org/r/69098/diff/2-3/
Testing (updated)
-------
Nonquota run setup: 20 agents, 10 roles, 20 frameworks
Made 20 allocations in 2.687627ms
Made 0 allocation in 1.575616ms
Quota run setup: 20 agents, 10 roles, 20 frameworks
Made 20 allocations in 6.190203ms
Made 0 allocation in 3.978099ms
Nonquota run setup: 200 agents, 100 roles, 200 frameworks
Made 200 allocations in 45.137172ms
Made 0 allocation in 31.409069ms
Quota run setup: 200 agents, 100 roles, 200 frameworks
Made 200 allocations in 356.202073ms
Made 0 allocation in 246.514222ms
Nonquota run setup: 2000 agents, 1000 roles, 2000 frameworks
Made 2000 allocations in 3.915063634secs
Made 0 allocation in 3.642528426secs
Quota run setup: 2000 agents, 1000 roles, 2000 frameworks
Made 2000 allocations in 36.141305833secs
Made 0 allocation in 26.49620811secs
Thanks,
Meng Zhu
Re: Review Request 69098: Added a benchmark to compare quota and
nonquota allocation performance.
Posted by Meng Zhu <mz...@mesosphere.io>.
> On Dec. 5, 2018, 3:39 p.m., Benjamin Mahler wrote:
> > Thanks for updating the test comment and commit description! Much clearer now and I think I would understand the need for a separate benchmark it if I were coming in fresh.
> >
> > Did you happen to post perf stacks anywhere?
Will upload one in the JIRA.
> On Dec. 5, 2018, 3:39 p.m., Benjamin Mahler wrote:
> > src/tests/hierarchical_allocator_benchmarks.cpp
> > Lines 664-666 (patched)
> > <https://reviews.apache.org/r/69098/diff/2/?file=2112229#file2112229line664>
> >
> > No need to state the name of the variable in its comment:
> >
> > // Determines the the number of agents in the cluster...
> >
> > But if this all that it is, no need for a comment at all, it's clear from the name what it represents.
> >
> > I can't quite understand the second sentence, perhaps you can clarify it? Intuitively it's not clear why the number of agents would influence how many are needed to satisfy a quota.
Updated the comment to:
// This determines the number of agents needed to satisfy a role's quota.
// And total number of agents in the cluster will be
// roleCount * agentsPerRole.
> On Dec. 5, 2018, 3:39 p.m., Benjamin Mahler wrote:
> > src/tests/hierarchical_allocator_benchmarks.cpp
> > Lines 667 (patched)
> > <https://reviews.apache.org/r/69098/diff/2/?file=2112229#file2112229line667>
> >
> > agentsPerRole?
Done.
> On Dec. 5, 2018, 3:39 p.m., Benjamin Mahler wrote:
> > src/tests/hierarchical_allocator_benchmarks.cpp
> > Lines 669 (patched)
> > <https://reviews.apache.org/r/69098/diff/2/?file=2112229#file2112229line669>
> >
> > frameworksPerRole?
Done.
> On Dec. 5, 2018, 3:39 p.m., Benjamin Mahler wrote:
> > src/tests/hierarchical_allocator_benchmarks.cpp
> > Lines 673-683 (patched)
> > <https://reviews.apache.org/r/69098/diff/2/?file=2112229#file2112229line673>
> >
> > Why do we need to store it as a member? Are we doing this anywhere else? I seem to recall we just access GetParam from the test itself?
Removed.
> On Dec. 5, 2018, 3:39 p.m., Benjamin Mahler wrote:
> > src/tests/hierarchical_allocator_benchmarks.cpp
> > Lines 690 (patched)
> > <https://reviews.apache.org/r/69098/diff/2/?file=2112229#file2112229line690>
> >
> > This would be clearer?
> >
> > ```
> > // 10 roles, 10*2 = 20 agents, 10*2 = 20 frameworks.
> > ```
> >
> > etc
Done.
> On Dec. 5, 2018, 3:39 p.m., Benjamin Mahler wrote:
> > src/tests/hierarchical_allocator_benchmarks.cpp
> > Lines 690-694 (patched)
> > <https://reviews.apache.org/r/69098/diff/2/?file=2112229#file2112229line690>
> >
> > periods?
Done.
> On Dec. 5, 2018, 3:39 p.m., Benjamin Mahler wrote:
> > src/tests/hierarchical_allocator_benchmarks.cpp
> > Lines 766 (patched)
> > <https://reviews.apache.org/r/69098/diff/2/?file=2112229#file2112229line766>
> >
> > Can you pre-increment as a habit? I'm not sure if the compiler can optimize away the post-increment copying in the case of iterators, Metric objects, etc. So as a habit pre-incrementing seems like the better approach to adhere to
Ah, good point. Will try to remember that. Thanks for pointing that out!
> On Dec. 5, 2018, 3:39 p.m., Benjamin Mahler wrote:
> > src/tests/hierarchical_allocator_benchmarks.cpp
> > Lines 779-780 (patched)
> > <https://reviews.apache.org/r/69098/diff/2/?file=2112229#file2112229line779>
> >
> > It's certainly not pristine from the perspective of the drf sorter's allocation count and metrics for example. So we probably don't want to say pristine here.
> >
> > While it's a bit brittle of an assumption, it seems ok to me for now with a note about this potentially starting from a dirty state if we change things.
> >
> > Did you consider using the param for this?
Yeah, should have used the param. Updated.
> On Dec. 5, 2018, 3:39 p.m., Benjamin Mahler wrote:
> > src/tests/hierarchical_allocator_benchmarks.cpp
> > Lines 793 (patched)
> > <https://reviews.apache.org/r/69098/diff/2/?file=2112229#file2112229line793>
> >
> > Do we need these two start outputs? It seems like they clutter the output to me.
Removed.
- Meng
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69098/#review211065
-----------------------------------------------------------
On Dec. 6, 2018, 2:42 p.m., Meng Zhu wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/69098/
> -----------------------------------------------------------
>
> (Updated Dec. 6, 2018, 2:42 p.m.)
>
>
> Review request for mesos, Benjamin Mahler and Gastón Kleiman.
>
>
> Bugs: MESOS-6630
> https://issues.apache.org/jira/browse/MESOS-6630
>
>
> Repository: mesos
>
>
> Description
> -------
>
> This benchmark evaluates the performance difference between nonquota
> and quota settings. In both settings, the same allocations are made
> for fair comparison. In particular, since the agent will always be
> allocated as a whole in nonquota settings, we should also avoid
> agent chopping in quota setting as well. Thus in this benchmark,
> quotas are only set to be multiples of whole agent resources.
> This is also why we have this dedicated benchmark for comparison
> rather than extending the existing quota benchmarks (which involves
> agent chopping).
>
>
> Diffs
> -----
>
> src/tests/hierarchical_allocator_benchmarks.cpp 6c6330e8cdbc705be322a7e2445b295c35ee6952
>
>
> Diff: https://reviews.apache.org/r/69098/diff/3/
>
>
> Testing
> -------
>
> Nonquota run setup: 20 agents, 10 roles, 20 frameworks
> Made 20 allocations in 2.687627ms
> Made 0 allocation in 1.575616ms
>
> Quota run setup: 20 agents, 10 roles, 20 frameworks
> Made 20 allocations in 6.190203ms
> Made 0 allocation in 3.978099ms
>
> Nonquota run setup: 200 agents, 100 roles, 200 frameworks
> Made 200 allocations in 45.137172ms
> Made 0 allocation in 31.409069ms
>
> Quota run setup: 200 agents, 100 roles, 200 frameworks
> Made 200 allocations in 356.202073ms
> Made 0 allocation in 246.514222ms
>
> Nonquota run setup: 2000 agents, 1000 roles, 2000 frameworks
> Made 2000 allocations in 3.915063634secs
> Made 0 allocation in 3.642528426secs
>
> Quota run setup: 2000 agents, 1000 roles, 2000 frameworks
> Made 2000 allocations in 36.141305833secs
> Made 0 allocation in 26.49620811secs
>
>
> Thanks,
>
> Meng Zhu
>
>
Re: Review Request 69098: Added a benchmark to compare quota and
nonquota allocation performance.
Posted by Benjamin Mahler <bm...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69098/#review211065
-----------------------------------------------------------
Fix it, then Ship it!
Thanks for updating the test comment and commit description! Much clearer now and I think I would understand the need for a separate benchmark it if I were coming in fresh.
Did you happen to post perf stacks anywhere?
src/tests/hierarchical_allocator_benchmarks.cpp
Lines 664-666 (patched)
<https://reviews.apache.org/r/69098/#comment295923>
No need to state the name of the variable in its comment:
// Determines the the number of agents in the cluster...
But if this all that it is, no need for a comment at all, it's clear from the name what it represents.
I can't quite understand the second sentence, perhaps you can clarify it? Intuitively it's not clear why the number of agents would influence how many are needed to satisfy a quota.
src/tests/hierarchical_allocator_benchmarks.cpp
Lines 667 (patched)
<https://reviews.apache.org/r/69098/#comment295921>
agentsPerRole?
src/tests/hierarchical_allocator_benchmarks.cpp
Lines 669 (patched)
<https://reviews.apache.org/r/69098/#comment295922>
frameworksPerRole?
src/tests/hierarchical_allocator_benchmarks.cpp
Lines 673-683 (patched)
<https://reviews.apache.org/r/69098/#comment295924>
Why do we need to store it as a member? Are we doing this anywhere else? I seem to recall we just access GetParam from the test itself?
src/tests/hierarchical_allocator_benchmarks.cpp
Lines 690 (patched)
<https://reviews.apache.org/r/69098/#comment295926>
This would be clearer?
```
// 10 roles, 10*2 = 20 agents, 10*2 = 20 frameworks.
```
etc
src/tests/hierarchical_allocator_benchmarks.cpp
Lines 690-694 (patched)
<https://reviews.apache.org/r/69098/#comment295927>
periods?
src/tests/hierarchical_allocator_benchmarks.cpp
Lines 766 (patched)
<https://reviews.apache.org/r/69098/#comment295928>
Can you pre-increment as a habit? I'm not sure if the compiler can optimize away the post-increment copying in the case of iterators, Metric objects, etc. So as a habit pre-incrementing seems like the better approach to adhere to
src/tests/hierarchical_allocator_benchmarks.cpp
Lines 779-780 (patched)
<https://reviews.apache.org/r/69098/#comment295929>
It's certainly not pristine from the perspective of the drf sorter's allocation count and metrics for example. So we probably don't want to say pristine here.
While it's a bit brittle of an assumption, it seems ok to me for now with a note about this potentially starting from a dirty state if we change things.
Did you consider using the param for this?
src/tests/hierarchical_allocator_benchmarks.cpp
Lines 793 (patched)
<https://reviews.apache.org/r/69098/#comment295930>
Do we need these two start outputs? It seems like they clutter the output to me.
src/tests/hierarchical_allocator_benchmarks.cpp
Lines 804-807 (patched)
<https://reviews.apache.org/r/69098/#comment295931>
Maybe state why we don't bother recovering the resources here?
- Benjamin Mahler
On Oct. 22, 2018, 10:39 p.m., Meng Zhu wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/69098/
> -----------------------------------------------------------
>
> (Updated Oct. 22, 2018, 10:39 p.m.)
>
>
> Review request for mesos, Benjamin Mahler and Gastón Kleiman.
>
>
> Bugs: MESOS-6630
> https://issues.apache.org/jira/browse/MESOS-6630
>
>
> Repository: mesos
>
>
> Description
> -------
>
> This benchmark evaluates the performance difference between nonquota
> and quota settings. In both settings, the same allocations are made
> for fair comparison. In particular, since the agent will always be
> allocated as a whole in nonquota settings, we should also avoid
> agent chopping in quota setting as well. Thus in this benchmark,
> quotas are only set to be multiples of whole agent resources.
> This is also why we have this dedicated benchmark for comparison
> rather than extending the existing quota benchmarks (which involves
> agent chopping).
>
>
> Diffs
> -----
>
> src/tests/hierarchical_allocator_benchmarks.cpp 6c6330e8cdbc705be322a7e2445b295c35ee6952
>
>
> Diff: https://reviews.apache.org/r/69098/diff/2/
>
>
> Testing
> -------
>
> Result from optimized build on a multicore 2.2GHz machine:
>
> Benchmark setup: 20 agents, 10 roles, 20 frameworks
> Made 20 allocations in 3.508676ms for nonquota roles
> Made 20 allocations in 7.901269ms for quota roles
>
> Benchmark setup: 200 agents, 100 roles, 200 frameworks
> Made 200 allocations in 63.407391ms for nonquota roles
> Made 200 allocations in 279.002319ms for quota roles
>
> Benchmark setup: 2000 agents, 1000 roles, 2000 frameworks
> Made 2000 allocations in 4.003802373secs for nonquota roles
> Made 2000 allocations in 20.503188535secs for quota roles
>
>
> Thanks,
>
> Meng Zhu
>
>
Re: Review Request 69098: Added a benchmark to compare quota and
nonquota allocation performance.
Posted by Meng Zhu <mz...@mesosphere.io>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69098/
-----------------------------------------------------------
(Updated Oct. 22, 2018, 3:39 p.m.)
Review request for mesos, Benjamin Mahler and Gastón Kleiman.
Changes
-------
Posted optimized build benchmark result.
Bugs: MESOS-6630
https://issues.apache.org/jira/browse/MESOS-6630
Repository: mesos
Description
-------
This benchmark evaluates the performance difference between nonquota
and quota settings. In both settings, the same allocations are made
for fair comparison.
Diffs
-----
src/tests/hierarchical_allocator_benchmarks.cpp bf9167b63747f7b8a402d950947028436307082a
Diff: https://reviews.apache.org/r/69098/diff/1/
Testing (updated)
-------
Result from optimized build on a multicore 2.2GHz machine:
Benchmark setup: 20 agents, 10 roles, 20 frameworks
Made 20 allocations in 3.508676ms for nonquota roles
Made 20 allocations in 7.901269ms for quota roles
Benchmark setup: 200 agents, 100 roles, 200 frameworks
Made 200 allocations in 63.407391ms for nonquota roles
Made 200 allocations in 279.002319ms for quota roles
Benchmark setup: 2000 agents, 1000 roles, 2000 frameworks
Made 2000 allocations in 4.003802373secs for nonquota roles
Made 2000 allocations in 20.503188535secs for quota roles
Thanks,
Meng Zhu