You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@mesos.apache.org by Meng Zhu <mz...@mesosphere.io> on 2019/04/09 00:01:09 UTC
Review Request 70430: Avoided some recalculation in the random sorter.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/70430/
-----------------------------------------------------------
Review request for mesos and Benjamin Mahler.
Bugs: MESOS-9452
https://issues.apache.org/jira/browse/MESOS-9452
Repository: mesos
Description
-------
This patch keeps the sorting related information in the memory
and accompanies a dirty bit with it. This helps to avoid
unnecessary recalculation of this info in `sort()`.
Diffs
-----
src/master/allocator/sorter/random/sorter.hpp 125ce84761e4c930370912151700ddda35d7b6c1
src/master/allocator/sorter/random/sorter.cpp bbe130dbf3b158ea14f9572bc5d14200fcd85127
Diff: https://reviews.apache.org/r/70430/diff/1/
Testing
-------
make check
Benchmarking:
Optimized build with QuotaParam/BENCHMARK_HierarchicalAllocator_WithQuotaParam.LargeAndSmallQuota/2
tl;dr: Performance is now on par with before the change in r/70429
## Before r/70429:
Added 3000 agents in 85.844373ms
Added 3000 frameworks in 19.713969615secs
Benchmark setup: 3000 agents, 3000 roles, 3000 frameworks
Made 3500 allocations in 13.690538305secs
Made 0 allocation in 9.76855825secs
## After r/70429 and this:
Added 3000 agents in 105.284907ms
Added 3000 frameworks in 19.253697264secs
Benchmark setup: 3000 agents, 3000 roles, 3000 frameworks
Made 3500 allocations in 13.488219803secs
Made 0 allocation in 9.846584089secs
Thanks,
Meng Zhu
Re: Review Request 70430: Avoided some recalculation in the random
sorter.
Posted by Mesos Reviewbot Windows <re...@mesos.apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/70430/#review214871
-----------------------------------------------------------
FAIL: Failed to apply the current review.
Failed command: `python.exe .\support\apply-reviews.py -n -r 70430`
All the build artifacts available at: http://dcos-win.westus2.cloudapp.azure.com/artifacts/mesos-reviewbot-testing/3258/mesos-review-70430
Relevant logs:
- [apply-review-70430.log](http://dcos-win.westus2.cloudapp.azure.com/artifacts/mesos-reviewbot-testing/3258/mesos-review-70430/logs/apply-review-70430.log):
```
error: patch failed: src/master/allocator/sorter/random/sorter.cpp:68
error: src/master/allocator/sorter/random/sorter.cpp: patch does not apply
```
- Mesos Reviewbot Windows
On April 9, 2019, 12:01 a.m., Meng Zhu wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/70430/
> -----------------------------------------------------------
>
> (Updated April 9, 2019, 12:01 a.m.)
>
>
> Review request for mesos and Benjamin Mahler.
>
>
> Bugs: MESOS-9733
> https://issues.apache.org/jira/browse/MESOS-9733
>
>
> Repository: mesos
>
>
> Description
> -------
>
> This patch keeps the sorting related information in the memory
> and accompanies a dirty bit with it. This helps to avoid
> unnecessary recalculation of this info in `sort()`.
>
>
> Diffs
> -----
>
> src/master/allocator/sorter/random/sorter.hpp 125ce84761e4c930370912151700ddda35d7b6c1
> src/master/allocator/sorter/random/sorter.cpp bbe130dbf3b158ea14f9572bc5d14200fcd85127
>
>
> Diff: https://reviews.apache.org/r/70430/diff/3/
>
>
> Testing
> -------
>
> make check
>
> Benchmarking:
> Optimized build with QuotaParam/BENCHMARK_HierarchicalAllocator_WithQuotaParam.LargeAndSmallQuota/2
>
> tl;dr: Performance is now on par with before the change in r/70429
>
> ## Before r/70429:
> Added 3000 agents in 85.844373ms
> Added 3000 frameworks in 19.713969615secs
> Benchmark setup: 3000 agents, 3000 roles, 3000 frameworks
> Made 3500 allocations in 13.690538305secs
> Made 0 allocation in 9.76855825secs
>
> ## After r/70429 and this:
> Added 3000 agents in 105.284907ms
> Added 3000 frameworks in 19.253697264secs
> Benchmark setup: 3000 agents, 3000 roles, 3000 frameworks
> Made 3500 allocations in 13.488219803secs
> Made 0 allocation in 9.846584089secs
>
>
> Thanks,
>
> Meng Zhu
>
>
Re: Review Request 70430: Avoided some recalculation in the random
sorter.
Posted by Benjamin Mahler <bm...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/70430/#review214891
-----------------------------------------------------------
Fix it, then Ship it!
Maybe we can re-run the benchmark and include the results in the commit since we didn't have the dirty bit getting cleared?
src/master/allocator/sorter/random/sorter.cpp
Line 629 (original), 643 (patched)
<https://reviews.apache.org/r/70430/#comment301181>
set dirty bit to false?
- Benjamin Mahler
On April 9, 2019, 12:01 a.m., Meng Zhu wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/70430/
> -----------------------------------------------------------
>
> (Updated April 9, 2019, 12:01 a.m.)
>
>
> Review request for mesos and Benjamin Mahler.
>
>
> Bugs: MESOS-9733
> https://issues.apache.org/jira/browse/MESOS-9733
>
>
> Repository: mesos
>
>
> Description
> -------
>
> This patch keeps the sorting related information in the memory
> and accompanies a dirty bit with it. This helps to avoid
> unnecessary recalculation of this info in `sort()`.
>
>
> Diffs
> -----
>
> src/master/allocator/sorter/random/sorter.hpp 125ce84761e4c930370912151700ddda35d7b6c1
> src/master/allocator/sorter/random/sorter.cpp bbe130dbf3b158ea14f9572bc5d14200fcd85127
>
>
> Diff: https://reviews.apache.org/r/70430/diff/3/
>
>
> Testing
> -------
>
> make check
>
> Benchmarking:
> Optimized build with QuotaParam/BENCHMARK_HierarchicalAllocator_WithQuotaParam.LargeAndSmallQuota/2
>
> tl;dr: Performance is now on par with before the change in r/70429
>
> ## Before r/70429:
> Added 3000 agents in 85.844373ms
> Added 3000 frameworks in 19.713969615secs
> Benchmark setup: 3000 agents, 3000 roles, 3000 frameworks
> Made 3500 allocations in 13.690538305secs
> Made 0 allocation in 9.76855825secs
>
> ## After r/70429 and this:
> Added 3000 agents in 105.284907ms
> Added 3000 frameworks in 19.253697264secs
> Benchmark setup: 3000 agents, 3000 roles, 3000 frameworks
> Made 3500 allocations in 13.488219803secs
> Made 0 allocation in 9.846584089secs
>
>
> Thanks,
>
> Meng Zhu
>
>
Re: Review Request 70430: Avoided some recalculation in the random
sorter.
Posted by Mesos Reviewbot Windows <re...@mesos.apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/70430/#review214730
-----------------------------------------------------------
FAIL: Some of the unit tests failed. Please check the relevant logs.
Reviews applied: `['70429', '70430']`
Failed command: `Start-MesosCITesting`
All the build artifacts available at: http://dcos-win.westus2.cloudapp.azure.com/artifacts/mesos-reviewbot-testing/3194/mesos-review-70430
Relevant logs:
- [mesos-tests.log](http://dcos-win.westus2.cloudapp.azure.com/artifacts/mesos-reviewbot-testing/3194/mesos-review-70430/logs/mesos-tests.log):
```
I0417 20:04:41.052911 60364 master.cpp:3352] Deactivating agent efd9479a-837b-4a3a-a36c-857f3c996ef4-S0 at slave(501)@192.10.1.4:49808 (windows-01.chtsmhjxogyevckjfayqqcnjda.xx.internal.cloudapp.net)
I0417 20:04:41.052911 66860 hierarchical.cpp:392] Removed framework efd9479a-837b-4a3a-a36c-857f3c996ef4-0000
I0417 20:04:41.053927 66860 hierarchical.cpp:829] Agent efd9479a-837b-4a3a-a36c-857f3c996ef4-S0 deactivated
I0417 20:04:41.054914 64488 containerizer.cpp:2576] Destroying container 263075ab-fc67-48ed-bb65-9eef9d6dddd6 in RUNNING state
I0417 20:04:41.054914 64488 containerizer.cpp:3278] Transitioning the state of container 263075ab-fc67-48ed-bb65-9eef9d6dddd6 from RUNNING to DESTROYING
I0417 20:04:41.054914 64488 launcher.cpp:161] Asked to destroy container 263075ab-fc67-48ed-bb65-9eef9d6dddd6
W0417 20:04:41.055905 67104 process.cpp:1423] Failed to recv on socket WindowsFD::Type::SOCKET=3260 to peer '192.10.1.4:52213': IO failed with error code: The specified network name is no longer avail[ OK ] IsolationFlag/MemoryIsolatorTest.ROOT_MemUsage/0 (773 ms)
[----------] 1 test from IsolationFlag/MemoryIsolatorTest (794 ms total)
[----------] Global test environment tear-down
[==========] 1161 tests from 109 test cases ran. (576626 ms total)
[ PASSED ] 1157 tests.
[ FAILED ] 4 tests, listed below:
[ FAILED ] RandomSorterTest.HierarchicalProbabilityDistribution
[ FAILED ] DockerFetcherPluginTest.INTERNET_CURL_FetchManifest
[ FAILED ] DockerFetcherPluginTest.INTERNET_CURL_FetchImage
[ FAILED ] DockerFetcherPluginTest.INTERNET_CURL_InvokeFetchByName
4 FAILED TESTS
YOU HAVE 233 DISABLED TESTS
able.
W0417 20:04:41.056910 67104 process.cpp:838] Failed to recv on socket WindowsFD::Type::SOCKET=4536 to peer '192.10.1.4:52214': IO failed with error code: The specified network name is no longer available.
I0417 20:04:41.121827 60872 containerizer.cpp:3117] Container 263075ab-fc67-48ed-bb65-9eef9d6dddd6 has exited
I0417 20:04:41.150856 62096 master.cpp:1135] Master terminating
I0417 20:04:41.152860 65164 hierarchical.cpp:680] Removed agent efd9479a-837b-4a3a-a36c-857f3c996ef4-S0
I0417 20:04:42.002846 67104 process.cpp:927] Stopped the socket accept loop
```
- Mesos Reviewbot Windows
On April 9, 2019, 12:01 a.m., Meng Zhu wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/70430/
> -----------------------------------------------------------
>
> (Updated April 9, 2019, 12:01 a.m.)
>
>
> Review request for mesos and Benjamin Mahler.
>
>
> Bugs: MESOS-9733
> https://issues.apache.org/jira/browse/MESOS-9733
>
>
> Repository: mesos
>
>
> Description
> -------
>
> This patch keeps the sorting related information in the memory
> and accompanies a dirty bit with it. This helps to avoid
> unnecessary recalculation of this info in `sort()`.
>
>
> Diffs
> -----
>
> src/master/allocator/sorter/random/sorter.hpp 125ce84761e4c930370912151700ddda35d7b6c1
> src/master/allocator/sorter/random/sorter.cpp bbe130dbf3b158ea14f9572bc5d14200fcd85127
>
>
> Diff: https://reviews.apache.org/r/70430/diff/2/
>
>
> Testing
> -------
>
> make check
>
> Benchmarking:
> Optimized build with QuotaParam/BENCHMARK_HierarchicalAllocator_WithQuotaParam.LargeAndSmallQuota/2
>
> tl;dr: Performance is now on par with before the change in r/70429
>
> ## Before r/70429:
> Added 3000 agents in 85.844373ms
> Added 3000 frameworks in 19.713969615secs
> Benchmark setup: 3000 agents, 3000 roles, 3000 frameworks
> Made 3500 allocations in 13.690538305secs
> Made 0 allocation in 9.76855825secs
>
> ## After r/70429 and this:
> Added 3000 agents in 105.284907ms
> Added 3000 frameworks in 19.253697264secs
> Benchmark setup: 3000 agents, 3000 roles, 3000 frameworks
> Made 3500 allocations in 13.488219803secs
> Made 0 allocation in 9.846584089secs
>
>
> Thanks,
>
> Meng Zhu
>
>
Re: Review Request 70430: Avoided some recalculation in the random
sorter.
Posted by Mesos Reviewbot Windows <re...@mesos.apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/70430/#review214489
-----------------------------------------------------------
FAIL: Some of the unit tests failed. Please check the relevant logs.
Reviews applied: `['70429', '70430']`
Failed command: `Start-MesosCITesting`
All the build artifacts available at: http://dcos-win.westus2.cloudapp.azure.com/artifacts/mesos-reviewbot-testing/3123/mesos-review-70430
Relevant logs:
- [mesos-tests.log](http://dcos-win.westus2.cloudapp.azure.com/artifacts/mesos-reviewbot-testing/3123/mesos-review-70430/logs/mesos-tests.log):
```
I0409 08:24:11.416843 41404 master.cpp:1295] Agent bec6eae5-5255-4922-908b-615934fac20f-S0 at slave(500)@192.10.1.4:62080 (windows-01.chtsmhjxogyevckjfayqqcnjda.xx.internal.cloudapp.net) disconnected
I0409 08:24:11.416843 41404 master.cpp:3333] Disconnecting agent bec6eae5-5255-4922-908b-615934fac20f-S0 at slave(500)@192.10.1.4:62080 (windows-01.chtsmhjxogyevckjfayqqcnjda.xx.internal.cloudapp.net)
I0409 08:24:11.416843 41404 master.cpp:3352] Deactivating agent bec6eae5-5255-4922-908b-615934fac20f-S0 at slave(500)@192.10.1.4:62080 (windows-01.chtsmhjxogyevckjfayqqcnjda.xx.internal.cloudapp.net)
I0409 08:24:11.417857 43052 hierarchical.cpp:392] Removed framework bec6eae5-5255-4922-908b-615934fac20f-0000
I0409 08:24:11.417857 43052 hierarchical.cpp:829] Agent bec6eae5-5255-4922-908b-615934fac20f-S0 deactivated
I0409 08:24:11.418853 39300 containerizer.cpp:2576] Destroying container 6e5cddf0-5326-41dd-b978-af6d74c53a13 in RUNNING state
I0409 08:24:11.418853 39300 containerizer.cpp:3278] Transitioning the state of container 6e5cddf0-5326-41dd-b978-af6d74c53a13 from RUNNING to DESTROYING
I0409 08:24:11.419919 39300 launcher.cpp:161] Asked to destroy container 6e5cddf0-5326-41dd-b978-af6d74c53a13
W0409 08:24:11.420856 33548 process.cpp:838] Failed to recv on socket WindowsFD::Type::SOCKET=11732 to peer '192.10.1.4:64478': IO failed with error code: The specified network name is no longer available.
W0409 08:24:11.420856 33548 process.cpp:1423] Failed to recv on socket WindowsFD::Type::SOCKET=11984 to peer '192.10.1.4:64477': IO failed with error code: The specified network name is no longer available.
I0409 08:24:11.441877 3758[ OK ] IsolationFlag/MemoryIsolatorTest.ROOT_MemUsage/0 (680 ms)
[----------] 1 test from IsolationFlag/MemoryIsolatorTest (697 ms total)
[----------] Global test environment tear-down
[==========] 1158 tests from 108 test cases ran. (588292 ms total)
[ PASSED ] 1156 tests.
[ FAILED ] 2 tests, listed below:
[ FAILED ] DockerFetcherPluginTest.INTERNET_CURL_FetchImage
[ FAILED ] DockerFetcherPluginTest.INTERNET_CURL_InvokeFetchByName
2 FAILED TESTS
YOU HAVE 233 DISABLED TESTS
0 containerizer.cpp:3117] Container 6e5cddf0-5326-41dd-b978-af6d74c53a13 has exited
I0409 08:24:11.471849 42244 master.cpp:1135] Master terminating
I0409 08:24:11.473852 38152 hierarchical.cpp:680] Removed agent bec6eae5-5255-4922-908b-615934fac20f-S0
I0409 08:24:12.036937 33548 process.cpp:927] Stopped the socket accept loop
```
- Mesos Reviewbot Windows
On April 9, 2019, 8:01 a.m., Meng Zhu wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/70430/
> -----------------------------------------------------------
>
> (Updated April 9, 2019, 8:01 a.m.)
>
>
> Review request for mesos and Benjamin Mahler.
>
>
> Bugs: MESOS-9452
> https://issues.apache.org/jira/browse/MESOS-9452
>
>
> Repository: mesos
>
>
> Description
> -------
>
> This patch keeps the sorting related information in the memory
> and accompanies a dirty bit with it. This helps to avoid
> unnecessary recalculation of this info in `sort()`.
>
>
> Diffs
> -----
>
> src/master/allocator/sorter/random/sorter.hpp 125ce84761e4c930370912151700ddda35d7b6c1
> src/master/allocator/sorter/random/sorter.cpp bbe130dbf3b158ea14f9572bc5d14200fcd85127
>
>
> Diff: https://reviews.apache.org/r/70430/diff/1/
>
>
> Testing
> -------
>
> make check
>
> Benchmarking:
> Optimized build with QuotaParam/BENCHMARK_HierarchicalAllocator_WithQuotaParam.LargeAndSmallQuota/2
>
> tl;dr: Performance is now on par with before the change in r/70429
>
> ## Before r/70429:
> Added 3000 agents in 85.844373ms
> Added 3000 frameworks in 19.713969615secs
> Benchmark setup: 3000 agents, 3000 roles, 3000 frameworks
> Made 3500 allocations in 13.690538305secs
> Made 0 allocation in 9.76855825secs
>
> ## After r/70429 and this:
> Added 3000 agents in 105.284907ms
> Added 3000 frameworks in 19.253697264secs
> Benchmark setup: 3000 agents, 3000 roles, 3000 frameworks
> Made 3500 allocations in 13.488219803secs
> Made 0 allocation in 9.846584089secs
>
>
> Thanks,
>
> Meng Zhu
>
>
Re: Review Request 70430: Avoided some recalculation in the random
sorter.
Posted by Mesos Reviewbot <re...@mesos.apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/70430/#review214867
-----------------------------------------------------------
Bad review!
Reviews applied: [70430, 70429, 70542]
Error:
2019-04-24 22:44:57 URL:https://reviews.apache.org/r/70430/diff/raw/ [3397/3397] -> "70430.patch" [1]
error: patch failed: src/master/allocator/sorter/random/sorter.cpp:68
error: src/master/allocator/sorter/random/sorter.cpp: patch does not apply
- Mesos Reviewbot
On April 8, 2019, 5:01 p.m., Meng Zhu wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/70430/
> -----------------------------------------------------------
>
> (Updated April 8, 2019, 5:01 p.m.)
>
>
> Review request for mesos and Benjamin Mahler.
>
>
> Bugs: MESOS-9733
> https://issues.apache.org/jira/browse/MESOS-9733
>
>
> Repository: mesos
>
>
> Description
> -------
>
> This patch keeps the sorting related information in the memory
> and accompanies a dirty bit with it. This helps to avoid
> unnecessary recalculation of this info in `sort()`.
>
>
> Diffs
> -----
>
> src/master/allocator/sorter/random/sorter.hpp 125ce84761e4c930370912151700ddda35d7b6c1
> src/master/allocator/sorter/random/sorter.cpp bbe130dbf3b158ea14f9572bc5d14200fcd85127
>
>
> Diff: https://reviews.apache.org/r/70430/diff/3/
>
>
> Testing
> -------
>
> make check
>
> Benchmarking:
> Optimized build with QuotaParam/BENCHMARK_HierarchicalAllocator_WithQuotaParam.LargeAndSmallQuota/2
>
> tl;dr: Performance is now on par with before the change in r/70429
>
> ## Before r/70429:
> Added 3000 agents in 85.844373ms
> Added 3000 frameworks in 19.713969615secs
> Benchmark setup: 3000 agents, 3000 roles, 3000 frameworks
> Made 3500 allocations in 13.690538305secs
> Made 0 allocation in 9.76855825secs
>
> ## After r/70429 and this:
> Added 3000 agents in 105.284907ms
> Added 3000 frameworks in 19.253697264secs
> Benchmark setup: 3000 agents, 3000 roles, 3000 frameworks
> Made 3500 allocations in 13.488219803secs
> Made 0 allocation in 9.846584089secs
>
>
> Thanks,
>
> Meng Zhu
>
>
Re: Review Request 70430: Avoided some recalculation in the random
sorter.
Posted by Mesos Reviewbot <re...@mesos.apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/70430/#review214487
-----------------------------------------------------------
Patch looks great!
Reviews applied: [70429, 70430]
Passed command: export OS='ubuntu:14.04' BUILDTOOL='autotools' COMPILER='gcc' CONFIGURATION='--verbose --disable-libtool-wrappers --disable-parallel-test-execution' ENVIRONMENT='GLOG_v=1 MESOS_VERBOSE=1'; ./support/docker-build.sh
- Mesos Reviewbot
On April 9, 2019, 12:01 a.m., Meng Zhu wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/70430/
> -----------------------------------------------------------
>
> (Updated April 9, 2019, 12:01 a.m.)
>
>
> Review request for mesos and Benjamin Mahler.
>
>
> Bugs: MESOS-9452
> https://issues.apache.org/jira/browse/MESOS-9452
>
>
> Repository: mesos
>
>
> Description
> -------
>
> This patch keeps the sorting related information in the memory
> and accompanies a dirty bit with it. This helps to avoid
> unnecessary recalculation of this info in `sort()`.
>
>
> Diffs
> -----
>
> src/master/allocator/sorter/random/sorter.hpp 125ce84761e4c930370912151700ddda35d7b6c1
> src/master/allocator/sorter/random/sorter.cpp bbe130dbf3b158ea14f9572bc5d14200fcd85127
>
>
> Diff: https://reviews.apache.org/r/70430/diff/1/
>
>
> Testing
> -------
>
> make check
>
> Benchmarking:
> Optimized build with QuotaParam/BENCHMARK_HierarchicalAllocator_WithQuotaParam.LargeAndSmallQuota/2
>
> tl;dr: Performance is now on par with before the change in r/70429
>
> ## Before r/70429:
> Added 3000 agents in 85.844373ms
> Added 3000 frameworks in 19.713969615secs
> Benchmark setup: 3000 agents, 3000 roles, 3000 frameworks
> Made 3500 allocations in 13.690538305secs
> Made 0 allocation in 9.76855825secs
>
> ## After r/70429 and this:
> Added 3000 agents in 105.284907ms
> Added 3000 frameworks in 19.253697264secs
> Benchmark setup: 3000 agents, 3000 roles, 3000 frameworks
> Made 3500 allocations in 13.488219803secs
> Made 0 allocation in 9.846584089secs
>
>
> Thanks,
>
> Meng Zhu
>
>