You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@mesos.apache.org by Michael Park <mc...@gmail.com> on 2015/10/14 01:56:54 UTC
Review Request 39300: Fixed flaky
ReservationEndpointsTest.AvailableResources test.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39300/
-----------------------------------------------------------
Review request for mesos and Ben Mahler.
Bugs: MESOS-3411
https://issues.apache.org/jira/browse/MESOS-3411
Repository: mesos
Description
-------
The following failure was encountered on buildbot: https://builds.apache.org/job/Mesos/COMPILER=gcc,CONFIGURATION=--verbose,OS=ubuntu:14.04,label_exp=docker%7C%7CHadoop/921/console
```
I1013 09:54:08.882694 29149 master.cpp:5559] Processing TEARDOWN call for framework 4b0845cd-7ce9-4e7a-b5d1-bcf1c413ca39-0000 (default) at scheduler-62c161d7-60e9-4361-aae1-6431e60035f6@172.17.5.161:57074
I1013 09:54:08.882822 29149 master.cpp:5571] Removing framework 4b0845cd-7ce9-4e7a-b5d1-bcf1c413ca39-0000 (default) at scheduler-62c161d7-60e9-4361-aae1-6431e60035f6@172.17.5.161:57074
../../src/tests/reservation_endpoints_tests.cpp:184: Failure
Mock function called more times than expected - taking default action specified at:
../../src/tests/mesos.hpp:1518:
Function call: recoverResources(@0x2b1be400cc10 4b0845cd-7ce9-4e7a-b5d1-bcf1c413ca39-0000, @0x2b1be4018f40 4b0845cd-7ce9-4e7a-b5d1-bcf1c413ca39-S0, @0x2b1bc22fc390 { cpus(*):2, mem(*):1024, disk(*):1024, ports(*):[31000-32000] }, @0x2b1bc22fc3d0 40-byte object <01-00 00-00 1C-2B 00-00 0B-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 80-3F 00-00 00-00>)
Expected: to be called once
Actual: called twice - over-saturated and active
I1013 09:54:08.884042 29149 hierarchical.hpp:599] Deactivated framework 4b0845cd-7ce9-4e7a-b5d1-bcf1c413ca39-0000
I1013 09:54:08.884371 29149 hierarchical.hpp:1103] Recovered cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000] (total: cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000], allocated: ) on slave 4b0845cd-7ce9-4e7a-b5d1-bcf1c413ca39-S0 from framework 4b0845cd-7ce9-4e7a-b5d1-bcf1c413ca39-0000
I1013 09:54:08.884469 29149 hierarchical.hpp:552] Removed framework 4b0845cd-7ce9-4e7a-b5d1-bcf1c413ca39-0000
```
In `ReservationEndpointsTest.AvailableResources`, the test ends with the framework holding an offer.
The issue is that there is only a `WillOnce` on `recoverResources`, but it should ignore subsequent calls with `WillRepeatedly`
since resources are expected to recover on framework teardown.
It seems that there is a race in the test teardown logic, but this fix is applicable regardless.
Diffs
-----
src/tests/reservation_endpoints_tests.cpp d41185241ae8f1f4ad4f9e88b7dea6fdac991ae7
Diff: https://reviews.apache.org/r/39300/diff/
Testing
-------
Thanks,
Michael Park
Re: Review Request 39300: Fixed flaky
ReservationEndpointsTest.AvailableResources test.
Posted by Mesos ReviewBot <re...@mesos.apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39300/#review102568
-----------------------------------------------------------
Patch looks great!
Reviews applied: [39300]
All tests passed.
- Mesos ReviewBot
On Oct. 13, 2015, 11:56 p.m., Michael Park wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/39300/
> -----------------------------------------------------------
>
> (Updated Oct. 13, 2015, 11:56 p.m.)
>
>
> Review request for mesos and Ben Mahler.
>
>
> Bugs: MESOS-3411
> https://issues.apache.org/jira/browse/MESOS-3411
>
>
> Repository: mesos
>
>
> Description
> -------
>
> The following failure was encountered on buildbot: https://builds.apache.org/job/Mesos/COMPILER=gcc,CONFIGURATION=--verbose,OS=ubuntu:14.04,label_exp=docker%7C%7CHadoop/921/console
>
> ```
> I1013 09:54:08.882694 29149 master.cpp:5559] Processing TEARDOWN call for framework 4b0845cd-7ce9-4e7a-b5d1-bcf1c413ca39-0000 (default) at scheduler-62c161d7-60e9-4361-aae1-6431e60035f6@172.17.5.161:57074
> I1013 09:54:08.882822 29149 master.cpp:5571] Removing framework 4b0845cd-7ce9-4e7a-b5d1-bcf1c413ca39-0000 (default) at scheduler-62c161d7-60e9-4361-aae1-6431e60035f6@172.17.5.161:57074
> ../../src/tests/reservation_endpoints_tests.cpp:184: Failure
> Mock function called more times than expected - taking default action specified at:
> ../../src/tests/mesos.hpp:1518:
> Function call: recoverResources(@0x2b1be400cc10 4b0845cd-7ce9-4e7a-b5d1-bcf1c413ca39-0000, @0x2b1be4018f40 4b0845cd-7ce9-4e7a-b5d1-bcf1c413ca39-S0, @0x2b1bc22fc390 { cpus(*):2, mem(*):1024, disk(*):1024, ports(*):[31000-32000] }, @0x2b1bc22fc3d0 40-byte object <01-00 00-00 1C-2B 00-00 0B-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 80-3F 00-00 00-00>)
> Expected: to be called once
> Actual: called twice - over-saturated and active
> I1013 09:54:08.884042 29149 hierarchical.hpp:599] Deactivated framework 4b0845cd-7ce9-4e7a-b5d1-bcf1c413ca39-0000
> I1013 09:54:08.884371 29149 hierarchical.hpp:1103] Recovered cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000] (total: cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000], allocated: ) on slave 4b0845cd-7ce9-4e7a-b5d1-bcf1c413ca39-S0 from framework 4b0845cd-7ce9-4e7a-b5d1-bcf1c413ca39-0000
> I1013 09:54:08.884469 29149 hierarchical.hpp:552] Removed framework 4b0845cd-7ce9-4e7a-b5d1-bcf1c413ca39-0000
> ```
>
> In `ReservationEndpointsTest.AvailableResources`, the test ends with the framework holding an offer.
> The issue is that there is only a `WillOnce` on `recoverResources`, but it should ignore subsequent calls with `WillRepeatedly`
> since resources are expected to recover on framework teardown.
>
> It seems that there is a race in the test teardown logic, but this fix is applicable regardless.
>
>
> Diffs
> -----
>
> src/tests/reservation_endpoints_tests.cpp d41185241ae8f1f4ad4f9e88b7dea6fdac991ae7
>
> Diff: https://reviews.apache.org/r/39300/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Michael Park
>
>
Re: Review Request 39300: Fixed flaky
ReservationEndpointsTest.AvailableResources test.
Posted by Mesos ReviewBot <re...@mesos.apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39300/#review102588
-----------------------------------------------------------
Patch looks great!
Reviews applied: [39300]
All tests passed.
- Mesos ReviewBot
On Oct. 14, 2015, 7:38 a.m., Michael Park wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/39300/
> -----------------------------------------------------------
>
> (Updated Oct. 14, 2015, 7:38 a.m.)
>
>
> Review request for mesos and Ben Mahler.
>
>
> Bugs: MESOS-3411
> https://issues.apache.org/jira/browse/MESOS-3411
>
>
> Repository: mesos
>
>
> Description
> -------
>
> The following failure was encountered on buildbot: https://builds.apache.org/job/Mesos/COMPILER=gcc,CONFIGURATION=--verbose,OS=ubuntu:14.04,label_exp=docker%7C%7CHadoop/921/console
>
> ```
> I1013 09:54:08.882694 29149 master.cpp:5559] Processing TEARDOWN call for framework 4b0845cd-7ce9-4e7a-b5d1-bcf1c413ca39-0000 (default) at scheduler-62c161d7-60e9-4361-aae1-6431e60035f6@172.17.5.161:57074
> I1013 09:54:08.882822 29149 master.cpp:5571] Removing framework 4b0845cd-7ce9-4e7a-b5d1-bcf1c413ca39-0000 (default) at scheduler-62c161d7-60e9-4361-aae1-6431e60035f6@172.17.5.161:57074
> ../../src/tests/reservation_endpoints_tests.cpp:184: Failure
> Mock function called more times than expected - taking default action specified at:
> ../../src/tests/mesos.hpp:1518:
> Function call: recoverResources(@0x2b1be400cc10 4b0845cd-7ce9-4e7a-b5d1-bcf1c413ca39-0000, @0x2b1be4018f40 4b0845cd-7ce9-4e7a-b5d1-bcf1c413ca39-S0, @0x2b1bc22fc390 { cpus(*):2, mem(*):1024, disk(*):1024, ports(*):[31000-32000] }, @0x2b1bc22fc3d0 40-byte object <01-00 00-00 1C-2B 00-00 0B-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 80-3F 00-00 00-00>)
> Expected: to be called once
> Actual: called twice - over-saturated and active
> I1013 09:54:08.884042 29149 hierarchical.hpp:599] Deactivated framework 4b0845cd-7ce9-4e7a-b5d1-bcf1c413ca39-0000
> I1013 09:54:08.884371 29149 hierarchical.hpp:1103] Recovered cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000] (total: cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000], allocated: ) on slave 4b0845cd-7ce9-4e7a-b5d1-bcf1c413ca39-S0 from framework 4b0845cd-7ce9-4e7a-b5d1-bcf1c413ca39-0000
> I1013 09:54:08.884469 29149 hierarchical.hpp:552] Removed framework 4b0845cd-7ce9-4e7a-b5d1-bcf1c413ca39-0000
> ```
>
> In `ReservationEndpointsTest.AvailableResources`, the test ends with the framework holding an offer.
> The issue is that there is only a `WillOnce` on `recoverResources`, but it should ignore subsequent calls with `WillRepeatedly`
> since resources are expected to recover on framework teardown.
>
> It seems that there is a race in the test teardown logic, but this fix is applicable regardless.
>
>
> Diffs
> -----
>
> src/tests/reservation_endpoints_tests.cpp d41185241ae8f1f4ad4f9e88b7dea6fdac991ae7
>
> Diff: https://reviews.apache.org/r/39300/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Michael Park
>
>
Re: Review Request 39300: Fixed flaky
ReservationEndpointsTest.AvailableResources test.
Posted by Michael Park <mc...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39300/
-----------------------------------------------------------
(Updated Oct. 14, 2015, 7:38 a.m.)
Review request for mesos and Ben Mahler.
Changes
-------
Fixed a missing but expected `recoverResources` in the middle of 2 other instances of this pattern.
Bugs: MESOS-3411
https://issues.apache.org/jira/browse/MESOS-3411
Repository: mesos
Description
-------
The following failure was encountered on buildbot: https://builds.apache.org/job/Mesos/COMPILER=gcc,CONFIGURATION=--verbose,OS=ubuntu:14.04,label_exp=docker%7C%7CHadoop/921/console
```
I1013 09:54:08.882694 29149 master.cpp:5559] Processing TEARDOWN call for framework 4b0845cd-7ce9-4e7a-b5d1-bcf1c413ca39-0000 (default) at scheduler-62c161d7-60e9-4361-aae1-6431e60035f6@172.17.5.161:57074
I1013 09:54:08.882822 29149 master.cpp:5571] Removing framework 4b0845cd-7ce9-4e7a-b5d1-bcf1c413ca39-0000 (default) at scheduler-62c161d7-60e9-4361-aae1-6431e60035f6@172.17.5.161:57074
../../src/tests/reservation_endpoints_tests.cpp:184: Failure
Mock function called more times than expected - taking default action specified at:
../../src/tests/mesos.hpp:1518:
Function call: recoverResources(@0x2b1be400cc10 4b0845cd-7ce9-4e7a-b5d1-bcf1c413ca39-0000, @0x2b1be4018f40 4b0845cd-7ce9-4e7a-b5d1-bcf1c413ca39-S0, @0x2b1bc22fc390 { cpus(*):2, mem(*):1024, disk(*):1024, ports(*):[31000-32000] }, @0x2b1bc22fc3d0 40-byte object <01-00 00-00 1C-2B 00-00 0B-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 80-3F 00-00 00-00>)
Expected: to be called once
Actual: called twice - over-saturated and active
I1013 09:54:08.884042 29149 hierarchical.hpp:599] Deactivated framework 4b0845cd-7ce9-4e7a-b5d1-bcf1c413ca39-0000
I1013 09:54:08.884371 29149 hierarchical.hpp:1103] Recovered cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000] (total: cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000], allocated: ) on slave 4b0845cd-7ce9-4e7a-b5d1-bcf1c413ca39-S0 from framework 4b0845cd-7ce9-4e7a-b5d1-bcf1c413ca39-0000
I1013 09:54:08.884469 29149 hierarchical.hpp:552] Removed framework 4b0845cd-7ce9-4e7a-b5d1-bcf1c413ca39-0000
```
In `ReservationEndpointsTest.AvailableResources`, the test ends with the framework holding an offer.
The issue is that there is only a `WillOnce` on `recoverResources`, but it should ignore subsequent calls with `WillRepeatedly`
since resources are expected to recover on framework teardown.
It seems that there is a race in the test teardown logic, but this fix is applicable regardless.
Diffs (updated)
-----
src/tests/reservation_endpoints_tests.cpp d41185241ae8f1f4ad4f9e88b7dea6fdac991ae7
Diff: https://reviews.apache.org/r/39300/diff/
Testing
-------
Thanks,
Michael Park
Re: Review Request 39300: Fixed flaky
ReservationEndpointsTest.AvailableResources test.
Posted by Michael Park <mc...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39300/
-----------------------------------------------------------
(Updated Oct. 14, 2015, 4:21 a.m.)
Review request for mesos and Ben Mahler.
Changes
-------
Moved the `WillRepeatedly(DoDefault())` to the end of the test accompanied with a comment.
Updated 2 other such instances in this file to follow the same pattern.
Bugs: MESOS-3411
https://issues.apache.org/jira/browse/MESOS-3411
Repository: mesos
Description
-------
The following failure was encountered on buildbot: https://builds.apache.org/job/Mesos/COMPILER=gcc,CONFIGURATION=--verbose,OS=ubuntu:14.04,label_exp=docker%7C%7CHadoop/921/console
```
I1013 09:54:08.882694 29149 master.cpp:5559] Processing TEARDOWN call for framework 4b0845cd-7ce9-4e7a-b5d1-bcf1c413ca39-0000 (default) at scheduler-62c161d7-60e9-4361-aae1-6431e60035f6@172.17.5.161:57074
I1013 09:54:08.882822 29149 master.cpp:5571] Removing framework 4b0845cd-7ce9-4e7a-b5d1-bcf1c413ca39-0000 (default) at scheduler-62c161d7-60e9-4361-aae1-6431e60035f6@172.17.5.161:57074
../../src/tests/reservation_endpoints_tests.cpp:184: Failure
Mock function called more times than expected - taking default action specified at:
../../src/tests/mesos.hpp:1518:
Function call: recoverResources(@0x2b1be400cc10 4b0845cd-7ce9-4e7a-b5d1-bcf1c413ca39-0000, @0x2b1be4018f40 4b0845cd-7ce9-4e7a-b5d1-bcf1c413ca39-S0, @0x2b1bc22fc390 { cpus(*):2, mem(*):1024, disk(*):1024, ports(*):[31000-32000] }, @0x2b1bc22fc3d0 40-byte object <01-00 00-00 1C-2B 00-00 0B-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 80-3F 00-00 00-00>)
Expected: to be called once
Actual: called twice - over-saturated and active
I1013 09:54:08.884042 29149 hierarchical.hpp:599] Deactivated framework 4b0845cd-7ce9-4e7a-b5d1-bcf1c413ca39-0000
I1013 09:54:08.884371 29149 hierarchical.hpp:1103] Recovered cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000] (total: cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000], allocated: ) on slave 4b0845cd-7ce9-4e7a-b5d1-bcf1c413ca39-S0 from framework 4b0845cd-7ce9-4e7a-b5d1-bcf1c413ca39-0000
I1013 09:54:08.884469 29149 hierarchical.hpp:552] Removed framework 4b0845cd-7ce9-4e7a-b5d1-bcf1c413ca39-0000
```
In `ReservationEndpointsTest.AvailableResources`, the test ends with the framework holding an offer.
The issue is that there is only a `WillOnce` on `recoverResources`, but it should ignore subsequent calls with `WillRepeatedly`
since resources are expected to recover on framework teardown.
It seems that there is a race in the test teardown logic, but this fix is applicable regardless.
Diffs (updated)
-----
src/tests/reservation_endpoints_tests.cpp d41185241ae8f1f4ad4f9e88b7dea6fdac991ae7
Diff: https://reviews.apache.org/r/39300/diff/
Testing
-------
Thanks,
Michael Park
Re: Review Request 39300: Fixed flaky
ReservationEndpointsTest.AvailableResources test.
Posted by Michael Park <mc...@gmail.com>.
> On Oct. 14, 2015, 12:44 a.m., Ben Mahler wrote:
> > src/tests/reservation_endpoints_tests.cpp, line 187
> > <https://reviews.apache.org/r/39300/diff/1/?file=1097896#file1097896line187>
> >
> > Shall we put this down before stopping the driver to be a bit clearer? A comment there would be helpful. :)
Great suggestion! I've updated the other 2 instances in this file as well.
- Michael
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39300/#review102565
-----------------------------------------------------------
On Oct. 14, 2015, 4:21 a.m., Michael Park wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/39300/
> -----------------------------------------------------------
>
> (Updated Oct. 14, 2015, 4:21 a.m.)
>
>
> Review request for mesos and Ben Mahler.
>
>
> Bugs: MESOS-3411
> https://issues.apache.org/jira/browse/MESOS-3411
>
>
> Repository: mesos
>
>
> Description
> -------
>
> The following failure was encountered on buildbot: https://builds.apache.org/job/Mesos/COMPILER=gcc,CONFIGURATION=--verbose,OS=ubuntu:14.04,label_exp=docker%7C%7CHadoop/921/console
>
> ```
> I1013 09:54:08.882694 29149 master.cpp:5559] Processing TEARDOWN call for framework 4b0845cd-7ce9-4e7a-b5d1-bcf1c413ca39-0000 (default) at scheduler-62c161d7-60e9-4361-aae1-6431e60035f6@172.17.5.161:57074
> I1013 09:54:08.882822 29149 master.cpp:5571] Removing framework 4b0845cd-7ce9-4e7a-b5d1-bcf1c413ca39-0000 (default) at scheduler-62c161d7-60e9-4361-aae1-6431e60035f6@172.17.5.161:57074
> ../../src/tests/reservation_endpoints_tests.cpp:184: Failure
> Mock function called more times than expected - taking default action specified at:
> ../../src/tests/mesos.hpp:1518:
> Function call: recoverResources(@0x2b1be400cc10 4b0845cd-7ce9-4e7a-b5d1-bcf1c413ca39-0000, @0x2b1be4018f40 4b0845cd-7ce9-4e7a-b5d1-bcf1c413ca39-S0, @0x2b1bc22fc390 { cpus(*):2, mem(*):1024, disk(*):1024, ports(*):[31000-32000] }, @0x2b1bc22fc3d0 40-byte object <01-00 00-00 1C-2B 00-00 0B-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 80-3F 00-00 00-00>)
> Expected: to be called once
> Actual: called twice - over-saturated and active
> I1013 09:54:08.884042 29149 hierarchical.hpp:599] Deactivated framework 4b0845cd-7ce9-4e7a-b5d1-bcf1c413ca39-0000
> I1013 09:54:08.884371 29149 hierarchical.hpp:1103] Recovered cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000] (total: cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000], allocated: ) on slave 4b0845cd-7ce9-4e7a-b5d1-bcf1c413ca39-S0 from framework 4b0845cd-7ce9-4e7a-b5d1-bcf1c413ca39-0000
> I1013 09:54:08.884469 29149 hierarchical.hpp:552] Removed framework 4b0845cd-7ce9-4e7a-b5d1-bcf1c413ca39-0000
> ```
>
> In `ReservationEndpointsTest.AvailableResources`, the test ends with the framework holding an offer.
> The issue is that there is only a `WillOnce` on `recoverResources`, but it should ignore subsequent calls with `WillRepeatedly`
> since resources are expected to recover on framework teardown.
>
> It seems that there is a race in the test teardown logic, but this fix is applicable regardless.
>
>
> Diffs
> -----
>
> src/tests/reservation_endpoints_tests.cpp d41185241ae8f1f4ad4f9e88b7dea6fdac991ae7
>
> Diff: https://reviews.apache.org/r/39300/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Michael Park
>
>
Re: Review Request 39300: Fixed flaky
ReservationEndpointsTest.AvailableResources test.
Posted by Ben Mahler <be...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39300/#review102565
-----------------------------------------------------------
Ship it!
Really appreciate the follow up, thanks Michael!
src/tests/reservation_endpoints_tests.cpp (line 187)
<https://reviews.apache.org/r/39300/#comment160270>
Shall we put this down before stopping the driver to be a bit clearer? A comment there would be helpful. :)
- Ben Mahler
On Oct. 13, 2015, 11:56 p.m., Michael Park wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/39300/
> -----------------------------------------------------------
>
> (Updated Oct. 13, 2015, 11:56 p.m.)
>
>
> Review request for mesos and Ben Mahler.
>
>
> Bugs: MESOS-3411
> https://issues.apache.org/jira/browse/MESOS-3411
>
>
> Repository: mesos
>
>
> Description
> -------
>
> The following failure was encountered on buildbot: https://builds.apache.org/job/Mesos/COMPILER=gcc,CONFIGURATION=--verbose,OS=ubuntu:14.04,label_exp=docker%7C%7CHadoop/921/console
>
> ```
> I1013 09:54:08.882694 29149 master.cpp:5559] Processing TEARDOWN call for framework 4b0845cd-7ce9-4e7a-b5d1-bcf1c413ca39-0000 (default) at scheduler-62c161d7-60e9-4361-aae1-6431e60035f6@172.17.5.161:57074
> I1013 09:54:08.882822 29149 master.cpp:5571] Removing framework 4b0845cd-7ce9-4e7a-b5d1-bcf1c413ca39-0000 (default) at scheduler-62c161d7-60e9-4361-aae1-6431e60035f6@172.17.5.161:57074
> ../../src/tests/reservation_endpoints_tests.cpp:184: Failure
> Mock function called more times than expected - taking default action specified at:
> ../../src/tests/mesos.hpp:1518:
> Function call: recoverResources(@0x2b1be400cc10 4b0845cd-7ce9-4e7a-b5d1-bcf1c413ca39-0000, @0x2b1be4018f40 4b0845cd-7ce9-4e7a-b5d1-bcf1c413ca39-S0, @0x2b1bc22fc390 { cpus(*):2, mem(*):1024, disk(*):1024, ports(*):[31000-32000] }, @0x2b1bc22fc3d0 40-byte object <01-00 00-00 1C-2B 00-00 0B-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 80-3F 00-00 00-00>)
> Expected: to be called once
> Actual: called twice - over-saturated and active
> I1013 09:54:08.884042 29149 hierarchical.hpp:599] Deactivated framework 4b0845cd-7ce9-4e7a-b5d1-bcf1c413ca39-0000
> I1013 09:54:08.884371 29149 hierarchical.hpp:1103] Recovered cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000] (total: cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000], allocated: ) on slave 4b0845cd-7ce9-4e7a-b5d1-bcf1c413ca39-S0 from framework 4b0845cd-7ce9-4e7a-b5d1-bcf1c413ca39-0000
> I1013 09:54:08.884469 29149 hierarchical.hpp:552] Removed framework 4b0845cd-7ce9-4e7a-b5d1-bcf1c413ca39-0000
> ```
>
> In `ReservationEndpointsTest.AvailableResources`, the test ends with the framework holding an offer.
> The issue is that there is only a `WillOnce` on `recoverResources`, but it should ignore subsequent calls with `WillRepeatedly`
> since resources are expected to recover on framework teardown.
>
> It seems that there is a race in the test teardown logic, but this fix is applicable regardless.
>
>
> Diffs
> -----
>
> src/tests/reservation_endpoints_tests.cpp d41185241ae8f1f4ad4f9e88b7dea6fdac991ae7
>
> Diff: https://reviews.apache.org/r/39300/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Michael Park
>
>