You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@mesos.apache.org by Benjamin Mahler <bm...@apache.org> on 2018/05/24 21:22:14 UTC
Review Request 67298: Fixed a performance issue in collect/await.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67298/
-----------------------------------------------------------
Review request for mesos, Chun-Hung Hsiao and Gilbert Song.
Repository: mesos
Description
-------
It turns out that even with C++11, some `std::list::size()`
implementations are still linear in complexity.
This was found when running perf against a metrics benchmark
for the master. A follow-up to avoid std::list in the collect.hpp
interfaces entirely is needed.
Diffs
-----
3rdparty/libprocess/include/process/collect.hpp 7e07b1db8fdb577daa58e83a1295e2d9845ffc31
Diff: https://reviews.apache.org/r/67298/diff/1/
Testing
-------
Before:
```
Registered 2000 frameworks
Finished launching the tasks; Sleep 10 seconds ...
Start collecting metrics ...
v0 '/metrics/snapshot' response took 17.751689014secs
v1 'master::call::GetMetrics' application/x-protobuf response took 17.523928635secs
v1 'master::call::GetMetrics' application/json response took 18.111901732secs
```
After:
```
Registered 2000 frameworks
Finished launching the tasks; Sleep 10 seconds ...
Start collecting metrics ...
v0 '/metrics/snapshot' response took 1.730948431secs
v1 'master::call::GetMetrics' application/x-protobuf response took 1.697177667secs
v1 'master::call::GetMetrics' application/json response took 2.160314525secs
```
Thanks,
Benjamin Mahler
Re: Review Request 67298: Fixed a performance issue in collect/await.
Posted by Mesos Reviewbot Windows <re...@mesos.apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67298/#review203823
-----------------------------------------------------------
FAIL: Some of the unit tests failed. Please check the relevant logs.
Failed command: `Start-MesosCITesting`
All the build artifacts available at: http://dcos-win.westus.cloudapp.azure.com/mesos-build/review/67298
Relevant logs:
- [mesos-tests-stdout.log](http://dcos-win.westus.cloudapp.azure.com/mesos-build/review/67298/logs/mesos-tests-stdout.log):
```
[ OK ] Endpoint/SlaveEndpointTest.NoAuthorizer/2 (106 ms)
[----------] 9 tests from Endpoint/SlaveEndpointTest (1015 ms total)
[----------] 2 tests from ContainerizerType/DefaultContainerDNSFlagTest
[ RUN ] ContainerizerType/DefaultContainerDNSFlagTest.ValidateFlag/0
[ OK ] ContainerizerType/DefaultContainerDNSFlagTest.ValidateFlag/0 (32 ms)
[ RUN ] ContainerizerType/DefaultContainerDNSFlagTest.ValidateFlag/1
[ OK ] ContainerizerType/DefaultContainerDNSFlagTest.ValidateFlag/1 (37 ms)
[----------] 2 tests from ContainerizerType/DefaultContainerDNSFlagTest (70 ms total)
[----------] 1 test from IsolationFlag/CpuIsolatorTest
[ RUN ] IsolationFlag/CpuIsolatorTest.ROOT_UserCpuUsage/0
[ OK ] IsolationFlag/CpuIsolatorTest.ROOT_UserCpuUsage/0 (736 ms)
[----------] 1 test from IsolationFlag/CpuIsolatorTest (763 ms total)
[----------] 1 test from IsolationFlag/MemoryIsolatorTest
[ RUN ] IsolationFlag/MemoryIsolatorTest.ROOT_MemUsage/0
[ OK ] IsolationFlag/MemoryIsolatorTest.ROOT_MemUsage/0 (729 ms)
[----------] 1 test from IsolationFlag/MemoryIsolatorTest (753 ms total)
[----------] Global test environment tear-down
[==========] 980 tests from 95 test cases ran. (444938 ms total)
[ PASSED ] 979 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] DockerContainerizerHealthCheckTest.ROOT_DOCKER_DockerHealthStatusChange
1 FAILED TEST
YOU HAVE 220 DISABLED TESTS
```
- [mesos-tests-stderr.log](http://dcos-win.westus.cloudapp.azure.com/mesos-build/review/67298/logs/mesos-tests-stderr.log):
```
I0524 22:15:41.169273 22800 master.cpp:10843] Updating the state of task a7dd3100-fa3d-4680-a658-6e901a9f8e88 of framework 0ec974ec-eae4-4aea-95b1-f539abc239af-0000 (latest state: TASK_KILLED, status update state: TASK_KILLED)
I0524 22:15:41.169273 16100 slave.cpp:3935] Shutting down framework 0ec974ec-eae4-4aea-95b1-f539abc239af-0000
I0524 22:15:41.169273 16100 slave.cpp:6656] Shutting down executor 'a7dd3100-fa3d-4680-a658-6e901a9f8e88' of framework 0ec974ec-eae4-4aea-95b1-f539abc239af-0000 at executor(1)@192.10.1.6:65103
I0524 22:15:41.171253 16100 slave.cpp:929] Agent terminating
W0524 22:15:41.171253 16100 slave.cpp:3931] Ignoring shutdown framework 0ec974ec-eae4-4aea-95b1-f539abc239af-0000 because it is terminating
I0524 22:15:41.171253 22800 master.cpp:10942] Removing task a7dd3100-fa3d-4680-a658-6e901a9f8e88 wiI0524 22:15:41.003270 22036 exec.cpp:162] Version: 1.7.0
I0524 22:15:41.028254 20288 exec.cpp:236] Executor registered on agent 0ec974ec-eae4-4aea-95b1-f539abc239af-S0
I0524 22:15:41.032265 20576 executor.cpp:178] Received SUBSCRIBED event
I0524 22:15:41.038264 20576 executor.cpp:182] Subscribed executor on windows-02.enofukwu14ruplxn0gs3yzmsgf.xx.internal.cloudapp.net
I0524 22:15:41.039264 20576 executor.cpp:178] Received LAUNCH event
I0524 22:15:41.043264 20576 executor.cpp:665] Starting task a7dd3100-fa3d-4680-a658-6e901a9f8e88
I0524 22:15:41.123265 20576 executor.cpp:485] Running 'D:\DCOS\mesos\src\mesos-containerizer.exe launch <POSSIBLY-SENSITIVE-DATA>'
I0524 22:15:41.141259 20576 executor.cpp:678] Forked command at 5872
I0524 22:15:41.171253 21828 exec.cpp:445] Executor asked to shutdown
I0524 22:15:41.171253 17672 executor.cpp:178] Received SHUTDOWN event
I0524 22:15:41.171253 17672 executor.cpp:781] Shutting down
I0524 22:15:41.171253 17672 executor.cpp:894] Sending SIGTERM to process tree at pid 587th resources cpus(allocated: *):4; mem(allocated: *):2048; disk(allocated: *):1024; ports(allocated: *):[31000-32000] of framework 0ec974ec-eae4-4aea-95b1-f539abc239af-0000 on agent 0ec974ec-eae4-4aea-95b1-f539abc239af-S0 at slave(447)@192.10.1.6:65082 (windows-02.enofukwu14ruplxn0gs3yzmsgf.xx.internal.cloudapp.net)
I0524 22:15:41.174266 22800 master.cpp:1293] Agent 0ec974ec-eae4-4aea-95b1-f539abc239af-S0 at slave(447)@192.10.1.6:65082 (windows-02.enofukwu14ruplxn0gs3yzmsgf.xx.internal.cloudapp.net) disconnected
I0524 22:15:41.175266 22800 master.cpp:3303] Disconnecting agent 0ec974ec-eae4-4aea-95b1-f539abc239af-S0 at slave(447)@192.10.1.6:65082 (windows-02.enofukwu14ruplxn0gs3yzmsgf.xx.internal.cloudapp.net)
I0524 22:15:41.175266 22800 master.cpp:3322] Deactivating agent 0ec974ec-eae4-4aea-95b1-f539abc239af-S0 at slave(447)@192.10.1.6:65082 (windows-02.enofukwu14ruplxn0gs3yzmsgf.xx.internal.cloudapp.net)
I0524 22:15:41.175266 22712 hierarchical.cpp:344] Removed framework 0ec974ec-eae4-4aea-95b1-f539abc239af-0000
I0524 22:15:41.176273 17984 containerizer.cpp:2401] Destroying container 5442fd84-f580-4bf0-9fd8-578c2b682856 in RUNNING state
I0524 22:15:41.176273 17984 containerizer.cpp:3015] Transitioning the state of container 5442fd84-f580-4bf0-9fd8-578c2b682856 from RUNNING to DESTROYING
I0524 22:15:41.176273 22712 hierarchical.cpp:766] Agent 0ec974ec-eae4-4aea-95b1-f539abc239af-S0 deactivated
I0524 22:15:41.177273 17984 launcher.cpp:156] Asked to destroy container 5442fd84-f580-4bf0-9fd8-578c2b682856
I0524 22:15:41.206706 22776 containerizer.cpp:2854] Container 5442fd84-f580-4bf0-9fd8-578c2b682856 has exited
I0524 22:15:41.236711 19772 master.cpp:1135] Master terminating
I0524 22:15:41.238689 20340 hierarchical.cpp:609] Removed agent 0ec974ec-eae4-4aea-95b1-f539abc239af-S0
I0524 22:15:41.744712 17688 process.cpp:940] Stopped the socket accept loop
```
- Mesos Reviewbot Windows
On May 24, 2018, 9:22 p.m., Benjamin Mahler wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/67298/
> -----------------------------------------------------------
>
> (Updated May 24, 2018, 9:22 p.m.)
>
>
> Review request for mesos, Chun-Hung Hsiao and Gilbert Song.
>
>
> Bugs: MESOS-8952
> https://issues.apache.org/jira/browse/MESOS-8952
>
>
> Repository: mesos
>
>
> Description
> -------
>
> It turns out that even with C++11, some `std::list::size()`
> implementations are still linear in complexity.
>
> This was found when running perf against a metrics benchmark
> for the master. A follow-up to avoid std::list in the collect.hpp
> interfaces entirely is needed.
>
>
> Diffs
> -----
>
> 3rdparty/libprocess/include/process/collect.hpp 7e07b1db8fdb577daa58e83a1295e2d9845ffc31
>
>
> Diff: https://reviews.apache.org/r/67298/diff/1/
>
>
> Testing
> -------
>
> Before:
>
> ```
> Registered 2000 frameworks
> Finished launching the tasks; Sleep 10 seconds ...
> Start collecting metrics ...
> v0 '/metrics/snapshot' response took 17.751689014secs
> v1 'master::call::GetMetrics' application/x-protobuf response took 17.523928635secs
> v1 'master::call::GetMetrics' application/json response took 18.111901732secs
> ```
>
> After:
>
> ```
> Registered 2000 frameworks
> Finished launching the tasks; Sleep 10 seconds ...
> Start collecting metrics ...
> v0 '/metrics/snapshot' response took 1.730948431secs
> v1 'master::call::GetMetrics' application/x-protobuf response took 1.697177667secs
> v1 'master::call::GetMetrics' application/json response took 2.160314525secs
> ```
>
>
> Thanks,
>
> Benjamin Mahler
>
>
Re: Review Request 67298: Fixed a performance issue in collect/await.
Posted by Gilbert Song <so...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67298/#review203819
-----------------------------------------------------------
Ship it!
Ship It!
- Gilbert Song
On May 24, 2018, 2:22 p.m., Benjamin Mahler wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/67298/
> -----------------------------------------------------------
>
> (Updated May 24, 2018, 2:22 p.m.)
>
>
> Review request for mesos, Chun-Hung Hsiao and Gilbert Song.
>
>
> Repository: mesos
>
>
> Description
> -------
>
> It turns out that even with C++11, some `std::list::size()`
> implementations are still linear in complexity.
>
> This was found when running perf against a metrics benchmark
> for the master. A follow-up to avoid std::list in the collect.hpp
> interfaces entirely is needed.
>
>
> Diffs
> -----
>
> 3rdparty/libprocess/include/process/collect.hpp 7e07b1db8fdb577daa58e83a1295e2d9845ffc31
>
>
> Diff: https://reviews.apache.org/r/67298/diff/1/
>
>
> Testing
> -------
>
> Before:
>
> ```
> Registered 2000 frameworks
> Finished launching the tasks; Sleep 10 seconds ...
> Start collecting metrics ...
> v0 '/metrics/snapshot' response took 17.751689014secs
> v1 'master::call::GetMetrics' application/x-protobuf response took 17.523928635secs
> v1 'master::call::GetMetrics' application/json response took 18.111901732secs
> ```
>
> After:
>
> ```
> Registered 2000 frameworks
> Finished launching the tasks; Sleep 10 seconds ...
> Start collecting metrics ...
> v0 '/metrics/snapshot' response took 1.730948431secs
> v1 'master::call::GetMetrics' application/x-protobuf response took 1.697177667secs
> v1 'master::call::GetMetrics' application/json response took 2.160314525secs
> ```
>
>
> Thanks,
>
> Benjamin Mahler
>
>