You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@mesos.apache.org by Gastón Kleiman <ga...@mesosphere.io> on 2018/07/03 19:16:55 UTC
Review Request 67817: Improved logging for offers and inverse offers.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67817/
-----------------------------------------------------------
Review request for mesos, Chun-Hung Hsiao, Greg Mann, and Vinod Kone.
Repository: mesos
Description
-------
Log offer IDs and inverse offer IDs when sending out offers and
inverse offers so it is easier to match them to their ACCEPT or DECLINE
calls and removals.
Also log at `VLOG(2)` level which resources are offered.
NOTE: It is possible to enable `VLOG(2)` logs just for `master.cpp` by
setting the following env variable when starting the master:
`GLOG_vmodule=master=2`.
Diffs
-----
src/master/master.cpp ddc8df0ea82241be6c733237feef1553c7669eb2
Diff: https://reviews.apache.org/r/67817/diff/1/
Testing
-------
Manual testing =).
Thanks,
Gastón Kleiman
Re: Review Request 67817: Improved logging for offers and inverse
offers.
Posted by Mesos Reviewbot <re...@mesos.apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67817/#review205706
-----------------------------------------------------------
Patch looks great!
Reviews applied: [67817]
Passed command: export OS='ubuntu:14.04' BUILDTOOL='autotools' COMPILER='gcc' CONFIGURATION='--verbose --disable-libtool-wrappers' ENVIRONMENT='GLOG_v=1 MESOS_VERBOSE=1'; ./support/docker-build.sh
- Mesos Reviewbot
On July 3, 2018, 8:56 p.m., Gastón Kleiman wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/67817/
> -----------------------------------------------------------
>
> (Updated July 3, 2018, 8:56 p.m.)
>
>
> Review request for mesos, Chun-Hung Hsiao, Greg Mann, and Vinod Kone.
>
>
> Repository: mesos
>
>
> Description
> -------
>
> Log offer IDs and inverse offer IDs when sending out offers and
> inverse offers so it is easier to match them to their ACCEPT or DECLINE
> calls and removals.
>
> Also log at `VLOG(2)` level which resources are offered.
>
> NOTE: It is possible to enable `VLOG(2)` logs just for `master.cpp` by
> setting the following env variable when starting the master:
> `GLOG_vmodule=master=2`.
>
>
> Diffs
> -----
>
> src/master/master.cpp ddc8df0ea82241be6c733237feef1553c7669eb2
>
>
> Diff: https://reviews.apache.org/r/67817/diff/2/
>
>
> Testing
> -------
>
> Manual testing =).
>
>
> Thanks,
>
> Gastón Kleiman
>
>
Re: Review Request 67817: Improved logging for offers and inverse
offers.
Posted by Mesos Reviewbot Windows <re...@mesos.apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67817/#review205695
-----------------------------------------------------------
FAIL: Some of the unit tests failed. Please check the relevant logs.
Reviews applied: `['67817']`
Failed command: `Start-MesosCITesting`
All the build artifacts available at: http://dcos-win.westus.cloudapp.azure.com/mesos-build/review/67817
Relevant logs:
- [mesos-tests-stdout.log](http://dcos-win.westus.cloudapp.azure.com/mesos-build/review/67817/logs/mesos-tests-stdout.log):
```
[ RUN ] DockerContainerizerTest.ROOT_DOCKER_DestroyWhilePulling
[ OK ] DockerContainerizerTest.ROOT_DOCKER_DestroyWhilePulling (800 ms)
[ RUN ] DockerContainerizerTest.ROOT_DOCKER_DestroyUnknownContainer
[ OK ] DockerContainerizerTest.ROOT_DOCKER_DestroyUnknownContainer (599 ms)
[ RUN ] DockerContainerizerTest.ROOT_DOCKER_ExecutorCleanupWhenLaunchFailed
[ OK ] DockerContainerizerTest.ROOT_DOCKER_ExecutorCleanupWhenLaunchFailed (1223 ms)
[ RUN ] DockerContainerizerTest.ROOT_DOCKER_FetchFailure
[ OK ] DockerContainerizerTest.ROOT_DOCKER_FetchFailure (800 ms)
[ RUN ] DockerContainerizerTest.ROOT_DOCKER_DockerPullFailure
[ OK ] DockerContainerizerTest.ROOT_DOCKER_DockerPullFailure (803 ms)
[ RUN ] DockerContainerizerTest.ROOT_DOCKER_DockerInspectDiscard
[ OK ] DockerContainerizerTest.ROOT_DOCKER_DockerInspectDiscard (1006 ms)
[ RUN ] DockerContainerizerTest.ROOT_DOCKER_WaitUnknownContainer
[ OK ] DockerContainerizerTest.ROOT_DOCKER_WaitUnknownContainer (603 ms)
[ RUN ] DockerContainerizerTest.ROOT_DOCKER_NoTransitionFromKillingToRunning
[ OK ] DockerContainerizerTest.ROOT_DOCKER_NoTransitionFromKillingToRunning (5457 ms)
[ RUN ] DockerContainerizerTest.ROOT_DOCKER_DefaultDNS
[ OK ] DockerContainerizerTest.ROOT_DOCKER_DefaultDNS (5059 ms)
[----------] 24 tests from DockerContainerizerTest (92935 ms total)
[----------] 1 test from HungDockerTest
[ RUN ] HungDockerTest.ROOT_DOCKER_InspectHungDuringPull
d:\dcos\mesos\mesos\src\tests\mock_docker.hpp(155): ERROR: this mock object (used in test HungDockerTest.ROOT_DOCKER_InspectHungDuringPull) should be deleted but never is. Its address is @000000BE8DF5BE90.
d:\dcos\mesos\mesos\src\tests\containerizer\docker_containerizer_tests.cpp(5187): ERROR: this mock object (used in test HungDockerTest.ROOT_DOCKER_InspectHungDuringPull) should be deleted but never is. Its address is @000000BE8DF5C0F0.
d:\dcos\mesos\mesos\src\tests\mock_docker.cpp(48): ERROR: this mock object (used in test HungDockerTest.ROOT_DOCKER_InspectHungDuringPull) should be deleted but never is. Its address is @00000278864CBF20.
d:\dcos\mesos\mesos\3rdparty\libprocess\include\process\gmock.hpp(235): ERROR: this mock object (used in test HungDockerTest.ROOT_DOCKER_InspectHungDuringPull) should be deleted but never is. Its address is @00000278867FAD98.
d:\dcos\mesos\mesos\src\tests\mock_registrar.cpp(54): ERROR: this mock object (used in test HungDockerTest.ROOT_DOCKER_InspectHungDuringPull) should be deleted but never is. Its address is @0000027887327170.
ERROR: 5 leaked mock objects found at program exit.
```
- [mesos-tests-stderr.log](http://dcos-win.westus.cloudapp.azure.com/mesos-build/review/67817/logs/mesos-tests-stderr.log):
```
I0703 21:55:49.018437 8884 authenticatee.cpp:299] Authentication success
I0703 21:55:49.018437 688 master.cpp:9816] Successfully authenticated principal 'test-principal' at scheduler-e3a6c593-a96f-4741-bf65-e160e0540822@192.10.1.6:55266
I0703 21:55:49.019436 9540 sched.cpp:501] Successfully authenticated with master master@192.10.1.6:55266
I0703 21:55:49.020438 8884 master.cpp:2927] Received SUBSCRIBE call for framework 'default' at scheduler-e3a6c593-a96f-4741-bf65-e160e0540822@192.10.1.6:55266
I0703 21:55:49.020438 8884 master.cpp:2234] Authorizing framework principal 'test-principal' to receive offers for roles '{ * }'
I0703 21:55:49.021437 688 master.cpp:3008] Subscribing framework default with checkpointing disabled and capabilities [ MULTI_ROLE, RESERVATION_REFINEMENT ]
I0703 21:55:49.021437 688 master.cpp:10007] Adding framework 8bbc2376-5d5f-4728-a1f3-263be02da6ec-0000 (default) at scheduler-e3a6c593-a96f-4741-bf65-e160e0540822@192.10.1.6:55266 with roles { } suppressed
I0703 21:55:49.022438 1840 sched.cpp:749] Framework registered with 8bbc2376-5d5f-4728-a1f3-263be02da6ec-0000
I0703 21:55:49.023437 4164 hierarchical.cpp:299] Added framework 8bbc2376-5d5f-4728-a1f3-263be02da6ec-0000
E0703 21:55:49.118428 9540 slave.cpp:7289] EXIT with status 1: Failed to perform recovery: Collect failed: Failed to run 'C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\VC\Auxiliary\Build\test-docker.bat -H npipe:////./pipe/docker_engine ps -a': exited with status 1; stderr=''C:\Program' is not recognized as an internal or external command,
operable program or batch file.
'
If recovery failed due to a change in configuration and you want to
keep the current agent id, you might want to change the
`--reconfiguration_policy` flag to a more permissive value.
To restart this agent with a new agent id instead, do as follows:
rm -f C:\Users\jenkins\AppData\Local\Temp\f9JyJV\meta\slaves\latest
This ensures that the agent does not recover old live executors.
If you use the Docker containerizer and think that the Docker
daemon state is broken, you can try to clear it. But be careful:
these commands will erase all containers and images from this host,
not just those started by Mesos!
docker kill $(docker ps -q)
docker rm $(docker ps -a -q)
docker rmi $(docker images -q)
Finally, restart the agent.
```
- Mesos Reviewbot Windows
On July 3, 2018, 8:56 p.m., Gastón Kleiman wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/67817/
> -----------------------------------------------------------
>
> (Updated July 3, 2018, 8:56 p.m.)
>
>
> Review request for mesos, Chun-Hung Hsiao, Greg Mann, and Vinod Kone.
>
>
> Repository: mesos
>
>
> Description
> -------
>
> Log offer IDs and inverse offer IDs when sending out offers and
> inverse offers so it is easier to match them to their ACCEPT or DECLINE
> calls and removals.
>
> Also log at `VLOG(2)` level which resources are offered.
>
> NOTE: It is possible to enable `VLOG(2)` logs just for `master.cpp` by
> setting the following env variable when starting the master:
> `GLOG_vmodule=master=2`.
>
>
> Diffs
> -----
>
> src/master/master.cpp ddc8df0ea82241be6c733237feef1553c7669eb2
>
>
> Diff: https://reviews.apache.org/r/67817/diff/2/
>
>
> Testing
> -------
>
> Manual testing =).
>
>
> Thanks,
>
> Gastón Kleiman
>
>
Re: Review Request 67817: Improved logging for offers and inverse
offers.
Posted by Greg Mann <gr...@mesosphere.io>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67817/#review205874
-----------------------------------------------------------
Ship it!
Ship It!
- Greg Mann
On July 3, 2018, 8:56 p.m., Gastón Kleiman wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/67817/
> -----------------------------------------------------------
>
> (Updated July 3, 2018, 8:56 p.m.)
>
>
> Review request for mesos, Chun-Hung Hsiao, Greg Mann, and Vinod Kone.
>
>
> Repository: mesos
>
>
> Description
> -------
>
> Log offer IDs and inverse offer IDs when sending out offers and
> inverse offers so it is easier to match them to their ACCEPT or DECLINE
> calls and removals.
>
> Also log at `VLOG(2)` level which resources are offered.
>
> NOTE: It is possible to enable `VLOG(2)` logs just for `master.cpp` by
> setting the following env variable when starting the master:
> `GLOG_vmodule=master=2`.
>
>
> Diffs
> -----
>
> src/master/master.cpp ddc8df0ea82241be6c733237feef1553c7669eb2
>
>
> Diff: https://reviews.apache.org/r/67817/diff/2/
>
>
> Testing
> -------
>
> Manual testing =).
>
>
> Thanks,
>
> Gastón Kleiman
>
>
Re: Review Request 67817: Improved logging for offers and inverse
offers.
Posted by Gastón Kleiman <ga...@mesosphere.io>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67817/
-----------------------------------------------------------
(Updated July 3, 2018, 1:56 p.m.)
Review request for mesos, Chun-Hung Hsiao, Greg Mann, and Vinod Kone.
Changes
-------
Addressed feedback.
Repository: mesos
Description
-------
Log offer IDs and inverse offer IDs when sending out offers and
inverse offers so it is easier to match them to their ACCEPT or DECLINE
calls and removals.
Also log at `VLOG(2)` level which resources are offered.
NOTE: It is possible to enable `VLOG(2)` logs just for `master.cpp` by
setting the following env variable when starting the master:
`GLOG_vmodule=master=2`.
Diffs (updated)
-----
src/master/master.cpp ddc8df0ea82241be6c733237feef1553c7669eb2
Diff: https://reviews.apache.org/r/67817/diff/2/
Changes: https://reviews.apache.org/r/67817/diff/1-2/
Testing
-------
Manual testing =).
Thanks,
Gastón Kleiman
Re: Review Request 67817: Improved logging for offers and inverse
offers.
Posted by Vinod Kone <vi...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67817/#review205683
-----------------------------------------------------------
Fix it, then Ship it!
src/master/master.cpp
Lines 9573 (patched)
<https://reviews.apache.org/r/67817/#comment288583>
Can we log `*slave` here instead of just slave id? im assuming that pointer is still valid here.
- Vinod Kone
On July 3, 2018, 7:16 p.m., Gastón Kleiman wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/67817/
> -----------------------------------------------------------
>
> (Updated July 3, 2018, 7:16 p.m.)
>
>
> Review request for mesos, Chun-Hung Hsiao, Greg Mann, and Vinod Kone.
>
>
> Repository: mesos
>
>
> Description
> -------
>
> Log offer IDs and inverse offer IDs when sending out offers and
> inverse offers so it is easier to match them to their ACCEPT or DECLINE
> calls and removals.
>
> Also log at `VLOG(2)` level which resources are offered.
>
> NOTE: It is possible to enable `VLOG(2)` logs just for `master.cpp` by
> setting the following env variable when starting the master:
> `GLOG_vmodule=master=2`.
>
>
> Diffs
> -----
>
> src/master/master.cpp ddc8df0ea82241be6c733237feef1553c7669eb2
>
>
> Diff: https://reviews.apache.org/r/67817/diff/1/
>
>
> Testing
> -------
>
> Manual testing =).
>
>
> Thanks,
>
> Gastón Kleiman
>
>
Re: Review Request 67817: Improved logging for offers and inverse
offers.
Posted by Mesos Reviewbot Windows <re...@mesos.apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67817/#review205689
-----------------------------------------------------------
FAIL: Some of the unit tests failed. Please check the relevant logs.
Reviews applied: `['67817']`
Failed command: `Start-MesosCITesting`
All the build artifacts available at: http://dcos-win.westus.cloudapp.azure.com/mesos-build/review/67817
Relevant logs:
- [mesos-tests-stdout.log](http://dcos-win.westus.cloudapp.azure.com/mesos-build/review/67817/logs/mesos-tests-stdout.log):
```
[ RUN ] DockerContainerizerTest.ROOT_DOCKER_DestroyWhilePulling
[ OK ] DockerContainerizerTest.ROOT_DOCKER_DestroyWhilePulling (801 ms)
[ RUN ] DockerContainerizerTest.ROOT_DOCKER_DestroyUnknownContainer
[ OK ] DockerContainerizerTest.ROOT_DOCKER_DestroyUnknownContainer (600 ms)
[ RUN ] DockerContainerizerTest.ROOT_DOCKER_ExecutorCleanupWhenLaunchFailed
[ OK ] DockerContainerizerTest.ROOT_DOCKER_ExecutorCleanupWhenLaunchFailed (1221 ms)
[ RUN ] DockerContainerizerTest.ROOT_DOCKER_FetchFailure
[ OK ] DockerContainerizerTest.ROOT_DOCKER_FetchFailure (800 ms)
[ RUN ] DockerContainerizerTest.ROOT_DOCKER_DockerPullFailure
[ OK ] DockerContainerizerTest.ROOT_DOCKER_DockerPullFailure (802 ms)
[ RUN ] DockerContainerizerTest.ROOT_DOCKER_DockerInspectDiscard
[ OK ] DockerContainerizerTest.ROOT_DOCKER_DockerInspectDiscard (1004 ms)
[ RUN ] DockerContainerizerTest.ROOT_DOCKER_WaitUnknownContainer
[ OK ] DockerContainerizerTest.ROOT_DOCKER_WaitUnknownContainer (598 ms)
[ RUN ] DockerContainerizerTest.ROOT_DOCKER_NoTransitionFromKillingToRunning
[ OK ] DockerContainerizerTest.ROOT_DOCKER_NoTransitionFromKillingToRunning (5259 ms)
[ RUN ] DockerContainerizerTest.ROOT_DOCKER_DefaultDNS
[ OK ] DockerContainerizerTest.ROOT_DOCKER_DefaultDNS (4853 ms)
[----------] 24 tests from DockerContainerizerTest (91298 ms total)
[----------] 1 test from HungDockerTest
[ RUN ] HungDockerTest.ROOT_DOCKER_InspectHungDuringPull
d:\dcos\mesos\mesos\src\tests\mock_docker.hpp(155): ERROR: this mock object (used in test HungDockerTest.ROOT_DOCKER_InspectHungDuringPull) should be deleted but never is. Its address is @000000ABE7B3B870.
d:\dcos\mesos\mesos\src\tests\containerizer\docker_containerizer_tests.cpp(5187): ERROR: this mock object (used in test HungDockerTest.ROOT_DOCKER_InspectHungDuringPull) should be deleted but never is. Its address is @000000ABE7B3BAD0.
d:\dcos\mesos\mesos\src\tests\mock_docker.cpp(48): ERROR: this mock object (used in test HungDockerTest.ROOT_DOCKER_InspectHungDuringPull) should be deleted but never is. Its address is @00000227A82CFF40.
d:\dcos\mesos\mesos\src\tests\mock_registrar.cpp(54): ERROR: this mock object (used in test HungDockerTest.ROOT_DOCKER_InspectHungDuringPull) should be deleted but never is. Its address is @00000227A8AC09B0.
d:\dcos\mesos\mesos\3rdparty\libprocess\include\process\gmock.hpp(235): ERROR: this mock object (used in test HungDockerTest.ROOT_DOCKER_InspectHungDuringPull) should be deleted but never is. Its address is @00000227A8D39B28.
ERROR: 5 leaked mock objects found at program exit.
```
- [mesos-tests-stderr.log](http://dcos-win.westus.cloudapp.azure.com/mesos-build/review/67817/logs/mesos-tests-stderr.log):
```
I0703 20:50:21.466701 9152 authenticatee.cpp:299] Authentication success
I0703 20:50:21.466701 8332 master.cpp:9816] Successfully authenticated principal 'test-principal' at scheduler-bb414125-29b9-4549-8cb0-10f5f602e64f@192.10.1.6:54213
I0703 20:50:21.467741 8372 sched.cpp:501] Successfully authenticated with master master@192.10.1.6:54213
I0703 20:50:21.468708 9152 master.cpp:2927] Received SUBSCRIBE call for framework 'default' at scheduler-bb414125-29b9-4549-8cb0-10f5f602e64f@192.10.1.6:54213
I0703 20:50:21.468708 9152 master.cpp:2234] Authorizing framework principal 'test-principal' to receive offers for roles '{ * }'
I0703 20:50:21.469713 8332 master.cpp:3008] Subscribing framework default with checkpointing disabled and capabilities [ MULTI_ROLE, RESERVATION_REFINEMENT ]
I0703 20:50:21.469713 8332 master.cpp:10007] Adding framework 7e549775-1613-4460-8e5b-2cc2f9685bd9-0000 (default) at scheduler-bb414125-29b9-4549-8cb0-10f5f602e64f@192.10.1.6:54213 with roles { } suppressed
E0703 20:50:21.470713 5608 slave.cpp:7289] EXIT with status 1: Failed to perform recovery: Collect failed: Failed to run 'C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\VC\Auxiliary\Build\test-docker.bat -H npipe:////./pipe/docker_engine ps -a': exited with status 1; stderr=''C:\Program' is not recognized as an internal or external command,
operable program or batch file.
'
If recovery failed due to a change in configuration and you want to
keep the current agent id, you might want to change the
`--reconfiguration_policy` flag to a more permissive value.
To restart this agent with a new agent id instead, do as follows:
rm -f C:\Users\jenkins\AppData\Local\Temp\ssYmId\meta\slaves\latest
This ensures that the agent does not recover old live executors.
If you use the Docker containerizer and think that the Docker
daemon state is broken, you can try to clear it. But be careful:
these commands will erase all containers and images from this host,
not just those started by Mesos!
docker kill $(docker ps -q)
docker rm $(docker ps -a -q)
docker rmi $(docker images -q)
Finally, restart the agent.
I0703 20:50:21.471709 6796 sched.cpp:749] Framework registered with 7e549775-1613-4460-8e5b-2cc2f9685bd9-0000
I0703 20:50:21.471709 3960 hierarchical.cpp:299] Added framework 7e549775-1613-4460-8e5b-2cc2f9685bd9-0000
```
- Mesos Reviewbot Windows
On July 3, 2018, 7:16 p.m., Gastón Kleiman wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/67817/
> -----------------------------------------------------------
>
> (Updated July 3, 2018, 7:16 p.m.)
>
>
> Review request for mesos, Chun-Hung Hsiao, Greg Mann, and Vinod Kone.
>
>
> Repository: mesos
>
>
> Description
> -------
>
> Log offer IDs and inverse offer IDs when sending out offers and
> inverse offers so it is easier to match them to their ACCEPT or DECLINE
> calls and removals.
>
> Also log at `VLOG(2)` level which resources are offered.
>
> NOTE: It is possible to enable `VLOG(2)` logs just for `master.cpp` by
> setting the following env variable when starting the master:
> `GLOG_vmodule=master=2`.
>
>
> Diffs
> -----
>
> src/master/master.cpp ddc8df0ea82241be6c733237feef1553c7669eb2
>
>
> Diff: https://reviews.apache.org/r/67817/diff/1/
>
>
> Testing
> -------
>
> Manual testing =).
>
>
> Thanks,
>
> Gastón Kleiman
>
>