You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@mesos.apache.org by Meng Zhu <mz...@mesosphere.io> on 2019/03/28 02:10:53 UTC
Review Request 70330: Used `ResourceQuantities` in the sorter when
appropriate.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/70330/
-----------------------------------------------------------
Review request for mesos and Benjamin Mahler.
Bugs: MESOS-9504
https://issues.apache.org/jira/browse/MESOS-9504
Repository: mesos
Description
-------
Replaced `Resources` with `ResourceQuantities` when
appropriate in the sorter. This simplifies the code
and improves performance.
Diffs
-----
src/master/allocator/mesos/hierarchical.cpp 8bc749903b8ceb09a02e260919377483479302b5
src/master/allocator/sorter/drf/sorter.hpp 75f90f331fbe2ec514daa3fe00b0b05ad55932e1
src/master/allocator/sorter/drf/sorter.cpp 43c97671d692675df6a347e4482126d83d7e3f24
src/master/allocator/sorter/random/sorter.hpp 2031cb234cc3e29723f07ec7d3a7e8671a26a97f
src/master/allocator/sorter/random/sorter.cpp 6fcfc41f65bb6401cfb60af88866c2b02920887e
src/master/allocator/sorter/sorter.hpp 25ad48dff7e624e7d25072958bdd20513ab83d12
src/tests/sorter_tests.cpp 1e2791f993af2fba592b0e76493864c096a0bb5f
Diff: https://reviews.apache.org/r/70330/diff/1/
Testing
-------
make check
# Benchmark Result
Optimized build on a 2.2GHz/64GB box
Each ran three times for average
**TL;DR: ~20% speedup for nonquota benchmark; ~30% speedup for quota benchmark**
## Non-quota allocation
Ran `BENCHMARK_HierarchicalAllocator_WithNonQuotaVsQuotaParam.NonQuotaVsQuota`
Selected the Nonquota result
Before:
- 20 agents, 10 roles, 20 frameworks
Made 20 allocations in 2.382283667ms
- 200 agents, 100 roles, 200 frameworks
Made 200 allocations in 19.05821233ms
- 2000 agents, 1000 roles, 2000 frameworks
Made 2000 allocations in 792.1674363ms
After:
- 20 agents, 10 roles, 20 frameworks
Made 20 allocations in 2.329543667ms // 2% speedup
- 200 agents, 100 roles, 200 frameworks
Made 200 allocations in 15.40222667ms // 19% speedup
- 2000 agents, 1000 roles, 2000 frameworks
Made 2000 allocations in 644.5257763ms // 19% speedup
## Quota Allocation
Ran `BENCHMARK_HierarchicalAllocator_WithQuotaParam.LargeAndSmallQuota`
Before:
- 30 agents, 30 roles, 30 frameworks
Made 36 allocations in 9.656610667ms
- 300 agents, 300 roles, 300 frameworks
Made 350 allocations in 292.0376493ms
- 3000 agents, 3000 roles, 3000 frameworks
Made 3500 allocations in 22.74423621secs
After:
- 30 agents, 30 roles, 30 frameworks
Made 36 allocations in 6.742066667ms // 30% speedup
- 300 agents, 300 roles, 300 frameworks
Made 350 allocations in 187.105902ms // 35% speedup
- 3000 agents, 3000 roles, 3000 frameworks
Made 3500 allocations in 14.74580913secs // 36% speedup
Thanks,
Meng Zhu
Re: Review Request 70330: Used `ResourceQuantities` in the sorter when
appropriate.
Posted by Mesos Reviewbot <re...@mesos.apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/70330/#review214222
-----------------------------------------------------------
Bad review!
Reviews applied: [70330, 70329, 70318]
Error:
2019-03-29 23:16:07 URL:https://reviews.apache.org/r/70330/diff/raw/ [22834/22834] -> "70330.patch" [1]
error: patch failed: src/master/allocator/mesos/hierarchical.cpp:1709
error: src/master/allocator/mesos/hierarchical.cpp: patch does not apply
- Mesos Reviewbot
On March 27, 2019, 7:10 p.m., Meng Zhu wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/70330/
> -----------------------------------------------------------
>
> (Updated March 27, 2019, 7:10 p.m.)
>
>
> Review request for mesos and Benjamin Mahler.
>
>
> Bugs: MESOS-9504
> https://issues.apache.org/jira/browse/MESOS-9504
>
>
> Repository: mesos
>
>
> Description
> -------
>
> Replaced `Resources` with `ResourceQuantities` when
> appropriate in the sorter. This simplifies the code
> and improves performance.
>
>
> Diffs
> -----
>
> src/master/allocator/mesos/hierarchical.cpp 8bc749903b8ceb09a02e260919377483479302b5
> src/master/allocator/sorter/drf/sorter.hpp 75f90f331fbe2ec514daa3fe00b0b05ad55932e1
> src/master/allocator/sorter/drf/sorter.cpp 43c97671d692675df6a347e4482126d83d7e3f24
> src/master/allocator/sorter/random/sorter.hpp 2031cb234cc3e29723f07ec7d3a7e8671a26a97f
> src/master/allocator/sorter/random/sorter.cpp 6fcfc41f65bb6401cfb60af88866c2b02920887e
> src/master/allocator/sorter/sorter.hpp 25ad48dff7e624e7d25072958bdd20513ab83d12
> src/tests/sorter_tests.cpp 1e2791f993af2fba592b0e76493864c096a0bb5f
>
>
> Diff: https://reviews.apache.org/r/70330/diff/1/
>
>
> Testing
> -------
>
> make check
>
> # Benchmark Result
> Optimized build on a 2.2GHz/64GB box
> Each ran three times for average
>
> **TL;DR: ~20% speedup for nonquota benchmark; ~30% speedup for quota benchmark**
>
> ## Non-quota allocation
>
> Ran `BENCHMARK_HierarchicalAllocator_WithNonQuotaVsQuotaParam.NonQuotaVsQuota`
> Selected the Nonquota result
>
> Before:
>
> - 20 agents, 10 roles, 20 frameworks
> Made 20 allocations in 2.382283667ms
>
> - 200 agents, 100 roles, 200 frameworks
> Made 200 allocations in 19.05821233ms
>
> - 2000 agents, 1000 roles, 2000 frameworks
> Made 2000 allocations in 792.1674363ms
>
> After:
>
> - 20 agents, 10 roles, 20 frameworks
> Made 20 allocations in 2.329543667ms // 2% speedup
>
> - 200 agents, 100 roles, 200 frameworks
> Made 200 allocations in 15.40222667ms // 19% speedup
>
> - 2000 agents, 1000 roles, 2000 frameworks
> Made 2000 allocations in 644.5257763ms // 19% speedup
>
> ## Quota Allocation
>
> Ran `BENCHMARK_HierarchicalAllocator_WithQuotaParam.LargeAndSmallQuota`
>
> Before:
>
> - 30 agents, 30 roles, 30 frameworks
> Made 36 allocations in 9.656610667ms
>
> - 300 agents, 300 roles, 300 frameworks
> Made 350 allocations in 292.0376493ms
>
> - 3000 agents, 3000 roles, 3000 frameworks
> Made 3500 allocations in 22.74423621secs
>
> After:
>
> - 30 agents, 30 roles, 30 frameworks
> Made 36 allocations in 6.742066667ms // 30% speedup
>
> - 300 agents, 300 roles, 300 frameworks
> Made 350 allocations in 187.105902ms // 35% speedup
>
> - 3000 agents, 3000 roles, 3000 frameworks
> Made 3500 allocations in 14.74580913secs // 36% speedup
>
>
> Thanks,
>
> Meng Zhu
>
>
Re: Review Request 70330: Used `ResourceQuantities` in the sorter when
appropriate.
Posted by Mesos Reviewbot Windows <re...@mesos.apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/70330/#review214155
-----------------------------------------------------------
FAIL: Some of the unit tests failed. Please check the relevant logs.
Reviews applied: `['70318', '70319', '70320', '70329', '70330']`
Failed command: `Start-MesosCITesting`
All the build artifacts available at: http://dcos-win.westus2.cloudapp.azure.com/artifacts/mesos-reviewbot-testing/3033/mesos-review-70330
Relevant logs:
- [mesos-tests.log](http://dcos-win.westus2.cloudapp.azure.com/artifacts/mesos-reviewbot-testing/3033/mesos-review-70330/logs/mesos-tests.log):
```
I0328 05:55:13.160986 17808 master.cpp:11685] Removing task e9cb7922-ca4c-4a91-a2cc-86d4e2d784a8 with resources cpus(allocated: *):4; mem(allocated: *):2048; disk(allocated: *):1024; ports(allocated: *):[31000-32000] of framework 6091452b-dbb2-4e9e-bd17-95fc87328817-0000 on agent 6091452b-dbb2-4e9e-bd17-95fc87328817-S0 at slave(500)@192.10.1.4:58249 (windows-01.chtsmhjxogyevckjfayqqcnjda.xx.internal.cloudapp.net)
I0328 05:55:13.163980 17808 master.cpp:1295] Agent 6091452b-dbb2-4e9e-bd17-95fc87328817-S0 at slave(500)@192.10.1.4:58249 (windows-01.chtsmhjxogyevckjfayqqcnjda.xx.internal.cloudapp.net) disconnected
I0328 05:55:13.163980 17808 master.cpp:3330] Disconnecting agent 6091452b-dbb2-4e9e-bd17-95fc87328817-S0 at slave(500)@192.10.1.4:58249 (windows-01.chtsmhjxogyevckjfayqqcnjda.xx.internal.cloudapp.net)
I0328 05:55:13.165027 17808 master.cpp:3349] Deactivating agent 6091452b-dbb2-4e9e-bd17-95fc87328817-S0 at slave(500)@192.10.1.4:58249 (windows-01.chtsmhjxogyevckjfayqqcnjda.xx.internal.cloudapp.net)
I0328 05:55:13.165027 19560 hierarchical.cpp:391] Removed framework 6091452b-dbb2-4e9e-bd17-95fc87328817-0000
I0328 05:55:13.165027 18796 containerizer.cpp:2576] Destroying container 58418bf9-0bf6-43c0-ae50-0c29efb7a31e in RUNNING state
I0328 05:55:13.165027 18796 containerizer.cpp:3278] Transitioning the state of container 58418bf9-0bf6-43c0-ae50-0c29efb7a31e from RUNNING to DESTROYING
I0328 05:55:13.165027 19560 hierarchical.cpp:828] Agent 6091452b-dbb2-4e9e-bd17-95fc87328817-S0 deactivated
I0328 05:55:13.166059 18796 [ OK ] IsolationFlag/MemoryIsolatorTest.ROOT_MemUsage/0 (795 ms)
[----------] 1 test from IsolationFlag/MemoryIsolatorTest (813 ms total)
[----------] Global test environment tear-down
[==========] 1130 tests from 107 test cases ran. (607328 ms total)
[ PASSED ] 1129 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] DockerFetcherPluginTest.INTERNET_CURL_FetchBlob
1 FAILED TEST
YOU HAVE 231 DISABLED TESTS
launcher.cpp:161] Asked to destroy container 58418bf9-0bf6-43c0-ae50-0c29efb7a31e
W0328 05:55:13.167023 16596 process.cpp:838] Failed to recv on socket WindowsFD::Type::SOCKET=3460 to peer '192.10.1.4:60638': IO failed with error code: The specified network name is no longer available.
W0328 05:55:13.167975 16596 process.cpp:1423] Failed to recv on socket WindowsFD::Type::SOCKET=4860 to peer '192.10.1.4:60637': IO failed with error code: The specified network name is no longer available.
I0328 05:55:13.255828 14096 containerizer.cpp:3117] Container 58418bf9-0bf6-43c0-ae50-0c29efb7a31e has exited
I0328 05:55:13.289773 19524 master.cpp:1135] Master terminating
I0328 05:55:13.290771 14096 hierarchical.cpp:679] Removed agent 6091452b-dbb2-4e9e-bd17-95fc87328817-S0
I0328 05:55:13.682775 16596 process.cpp:927] Stopped the socket accept loop
```
- Mesos Reviewbot Windows
On March 28, 2019, 2:10 a.m., Meng Zhu wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/70330/
> -----------------------------------------------------------
>
> (Updated March 28, 2019, 2:10 a.m.)
>
>
> Review request for mesos and Benjamin Mahler.
>
>
> Bugs: MESOS-9504
> https://issues.apache.org/jira/browse/MESOS-9504
>
>
> Repository: mesos
>
>
> Description
> -------
>
> Replaced `Resources` with `ResourceQuantities` when
> appropriate in the sorter. This simplifies the code
> and improves performance.
>
>
> Diffs
> -----
>
> src/master/allocator/mesos/hierarchical.cpp 8bc749903b8ceb09a02e260919377483479302b5
> src/master/allocator/sorter/drf/sorter.hpp 75f90f331fbe2ec514daa3fe00b0b05ad55932e1
> src/master/allocator/sorter/drf/sorter.cpp 43c97671d692675df6a347e4482126d83d7e3f24
> src/master/allocator/sorter/random/sorter.hpp 2031cb234cc3e29723f07ec7d3a7e8671a26a97f
> src/master/allocator/sorter/random/sorter.cpp 6fcfc41f65bb6401cfb60af88866c2b02920887e
> src/master/allocator/sorter/sorter.hpp 25ad48dff7e624e7d25072958bdd20513ab83d12
> src/tests/sorter_tests.cpp 1e2791f993af2fba592b0e76493864c096a0bb5f
>
>
> Diff: https://reviews.apache.org/r/70330/diff/1/
>
>
> Testing
> -------
>
> make check
>
> # Benchmark Result
> Optimized build on a 2.2GHz/64GB box
> Each ran three times for average
>
> **TL;DR: ~20% speedup for nonquota benchmark; ~30% speedup for quota benchmark**
>
> ## Non-quota allocation
>
> Ran `BENCHMARK_HierarchicalAllocator_WithNonQuotaVsQuotaParam.NonQuotaVsQuota`
> Selected the Nonquota result
>
> Before:
>
> - 20 agents, 10 roles, 20 frameworks
> Made 20 allocations in 2.382283667ms
>
> - 200 agents, 100 roles, 200 frameworks
> Made 200 allocations in 19.05821233ms
>
> - 2000 agents, 1000 roles, 2000 frameworks
> Made 2000 allocations in 792.1674363ms
>
> After:
>
> - 20 agents, 10 roles, 20 frameworks
> Made 20 allocations in 2.329543667ms // 2% speedup
>
> - 200 agents, 100 roles, 200 frameworks
> Made 200 allocations in 15.40222667ms // 19% speedup
>
> - 2000 agents, 1000 roles, 2000 frameworks
> Made 2000 allocations in 644.5257763ms // 19% speedup
>
> ## Quota Allocation
>
> Ran `BENCHMARK_HierarchicalAllocator_WithQuotaParam.LargeAndSmallQuota`
>
> Before:
>
> - 30 agents, 30 roles, 30 frameworks
> Made 36 allocations in 9.656610667ms
>
> - 300 agents, 300 roles, 300 frameworks
> Made 350 allocations in 292.0376493ms
>
> - 3000 agents, 3000 roles, 3000 frameworks
> Made 3500 allocations in 22.74423621secs
>
> After:
>
> - 30 agents, 30 roles, 30 frameworks
> Made 36 allocations in 6.742066667ms // 30% speedup
>
> - 300 agents, 300 roles, 300 frameworks
> Made 350 allocations in 187.105902ms // 35% speedup
>
> - 3000 agents, 3000 roles, 3000 frameworks
> Made 3500 allocations in 14.74580913secs // 36% speedup
>
>
> Thanks,
>
> Meng Zhu
>
>
Re: Review Request 70330: Used `ResourceQuantities` in the sorter when
appropriate.
Posted by Mesos Reviewbot <re...@mesos.apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/70330/#review214156
-----------------------------------------------------------
Patch looks great!
Reviews applied: [70318, 70319, 70320, 70329, 70330]
Passed command: export OS='ubuntu:14.04' BUILDTOOL='autotools' COMPILER='gcc' CONFIGURATION='--verbose --disable-libtool-wrappers --disable-parallel-test-execution' ENVIRONMENT='GLOG_v=1 MESOS_VERBOSE=1'; ./support/docker-build.sh
- Mesos Reviewbot
On March 28, 2019, 2:10 a.m., Meng Zhu wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/70330/
> -----------------------------------------------------------
>
> (Updated March 28, 2019, 2:10 a.m.)
>
>
> Review request for mesos and Benjamin Mahler.
>
>
> Bugs: MESOS-9504
> https://issues.apache.org/jira/browse/MESOS-9504
>
>
> Repository: mesos
>
>
> Description
> -------
>
> Replaced `Resources` with `ResourceQuantities` when
> appropriate in the sorter. This simplifies the code
> and improves performance.
>
>
> Diffs
> -----
>
> src/master/allocator/mesos/hierarchical.cpp 8bc749903b8ceb09a02e260919377483479302b5
> src/master/allocator/sorter/drf/sorter.hpp 75f90f331fbe2ec514daa3fe00b0b05ad55932e1
> src/master/allocator/sorter/drf/sorter.cpp 43c97671d692675df6a347e4482126d83d7e3f24
> src/master/allocator/sorter/random/sorter.hpp 2031cb234cc3e29723f07ec7d3a7e8671a26a97f
> src/master/allocator/sorter/random/sorter.cpp 6fcfc41f65bb6401cfb60af88866c2b02920887e
> src/master/allocator/sorter/sorter.hpp 25ad48dff7e624e7d25072958bdd20513ab83d12
> src/tests/sorter_tests.cpp 1e2791f993af2fba592b0e76493864c096a0bb5f
>
>
> Diff: https://reviews.apache.org/r/70330/diff/1/
>
>
> Testing
> -------
>
> make check
>
> # Benchmark Result
> Optimized build on a 2.2GHz/64GB box
> Each ran three times for average
>
> **TL;DR: ~20% speedup for nonquota benchmark; ~30% speedup for quota benchmark**
>
> ## Non-quota allocation
>
> Ran `BENCHMARK_HierarchicalAllocator_WithNonQuotaVsQuotaParam.NonQuotaVsQuota`
> Selected the Nonquota result
>
> Before:
>
> - 20 agents, 10 roles, 20 frameworks
> Made 20 allocations in 2.382283667ms
>
> - 200 agents, 100 roles, 200 frameworks
> Made 200 allocations in 19.05821233ms
>
> - 2000 agents, 1000 roles, 2000 frameworks
> Made 2000 allocations in 792.1674363ms
>
> After:
>
> - 20 agents, 10 roles, 20 frameworks
> Made 20 allocations in 2.329543667ms // 2% speedup
>
> - 200 agents, 100 roles, 200 frameworks
> Made 200 allocations in 15.40222667ms // 19% speedup
>
> - 2000 agents, 1000 roles, 2000 frameworks
> Made 2000 allocations in 644.5257763ms // 19% speedup
>
> ## Quota Allocation
>
> Ran `BENCHMARK_HierarchicalAllocator_WithQuotaParam.LargeAndSmallQuota`
>
> Before:
>
> - 30 agents, 30 roles, 30 frameworks
> Made 36 allocations in 9.656610667ms
>
> - 300 agents, 300 roles, 300 frameworks
> Made 350 allocations in 292.0376493ms
>
> - 3000 agents, 3000 roles, 3000 frameworks
> Made 3500 allocations in 22.74423621secs
>
> After:
>
> - 30 agents, 30 roles, 30 frameworks
> Made 36 allocations in 6.742066667ms // 30% speedup
>
> - 300 agents, 300 roles, 300 frameworks
> Made 350 allocations in 187.105902ms // 35% speedup
>
> - 3000 agents, 3000 roles, 3000 frameworks
> Made 3500 allocations in 14.74580913secs // 36% speedup
>
>
> Thanks,
>
> Meng Zhu
>
>
Re: Review Request 70330: Used `ResourceQuantities` in the sorter when
appropriate.
Posted by Benjamin Mahler <bm...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/70330/#review214188
-----------------------------------------------------------
Ship it!
Very nice!
- Benjamin Mahler
On March 28, 2019, 2:10 a.m., Meng Zhu wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/70330/
> -----------------------------------------------------------
>
> (Updated March 28, 2019, 2:10 a.m.)
>
>
> Review request for mesos and Benjamin Mahler.
>
>
> Bugs: MESOS-9504
> https://issues.apache.org/jira/browse/MESOS-9504
>
>
> Repository: mesos
>
>
> Description
> -------
>
> Replaced `Resources` with `ResourceQuantities` when
> appropriate in the sorter. This simplifies the code
> and improves performance.
>
>
> Diffs
> -----
>
> src/master/allocator/mesos/hierarchical.cpp 8bc749903b8ceb09a02e260919377483479302b5
> src/master/allocator/sorter/drf/sorter.hpp 75f90f331fbe2ec514daa3fe00b0b05ad55932e1
> src/master/allocator/sorter/drf/sorter.cpp 43c97671d692675df6a347e4482126d83d7e3f24
> src/master/allocator/sorter/random/sorter.hpp 2031cb234cc3e29723f07ec7d3a7e8671a26a97f
> src/master/allocator/sorter/random/sorter.cpp 6fcfc41f65bb6401cfb60af88866c2b02920887e
> src/master/allocator/sorter/sorter.hpp 25ad48dff7e624e7d25072958bdd20513ab83d12
> src/tests/sorter_tests.cpp 1e2791f993af2fba592b0e76493864c096a0bb5f
>
>
> Diff: https://reviews.apache.org/r/70330/diff/1/
>
>
> Testing
> -------
>
> make check
>
> # Benchmark Result
> Optimized build on a 2.2GHz/64GB box
> Each ran three times for average
>
> **TL;DR: ~20% speedup for nonquota benchmark; ~30% speedup for quota benchmark**
>
> ## Non-quota allocation
>
> Ran `BENCHMARK_HierarchicalAllocator_WithNonQuotaVsQuotaParam.NonQuotaVsQuota`
> Selected the Nonquota result
>
> Before:
>
> - 20 agents, 10 roles, 20 frameworks
> Made 20 allocations in 2.382283667ms
>
> - 200 agents, 100 roles, 200 frameworks
> Made 200 allocations in 19.05821233ms
>
> - 2000 agents, 1000 roles, 2000 frameworks
> Made 2000 allocations in 792.1674363ms
>
> After:
>
> - 20 agents, 10 roles, 20 frameworks
> Made 20 allocations in 2.329543667ms // 2% speedup
>
> - 200 agents, 100 roles, 200 frameworks
> Made 200 allocations in 15.40222667ms // 19% speedup
>
> - 2000 agents, 1000 roles, 2000 frameworks
> Made 2000 allocations in 644.5257763ms // 19% speedup
>
> ## Quota Allocation
>
> Ran `BENCHMARK_HierarchicalAllocator_WithQuotaParam.LargeAndSmallQuota`
>
> Before:
>
> - 30 agents, 30 roles, 30 frameworks
> Made 36 allocations in 9.656610667ms
>
> - 300 agents, 300 roles, 300 frameworks
> Made 350 allocations in 292.0376493ms
>
> - 3000 agents, 3000 roles, 3000 frameworks
> Made 3500 allocations in 22.74423621secs
>
> After:
>
> - 30 agents, 30 roles, 30 frameworks
> Made 36 allocations in 6.742066667ms // 30% speedup
>
> - 300 agents, 300 roles, 300 frameworks
> Made 350 allocations in 187.105902ms // 35% speedup
>
> - 3000 agents, 3000 roles, 3000 frameworks
> Made 3500 allocations in 14.74580913secs // 36% speedup
>
>
> Thanks,
>
> Meng Zhu
>
>