You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@mesos.apache.org by Guangya Liu <gy...@gmail.com> on 2015/11/24 10:39:50 UTC
Review Request 40632: WIP: Enabled oversubscribed resources for
reservations in allocator.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40632/
-----------------------------------------------------------
Review request for mesos and Klaus Ma.
Repository: mesos
Description
-------
WIP: Enabled oversubscribed resources for reservations in allocator.
Diffs
-----
src/master/allocator/mesos/hierarchical.cpp aee8ced1fbfec8cf30cb939ff67fadfc6b08f37a
src/tests/resources_tests.cpp dbd39cd5a6786682a7b528b6fea37ab78904cf12
Diff: https://reviews.apache.org/r/40632/diff/
Testing
-------
Thanks,
Guangya Liu
Re: Review Request 40632: WIP: Enabled oversubscribed resources for
reservations in allocator.
Posted by Mesos ReviewBot <re...@mesos.apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40632/#review107783
-----------------------------------------------------------
Patch looks great!
Reviews applied: [40339, 40524, 40529, 40532, 40632]
Passed command: export OS=ubuntu:14.04;export CONFIGURATION="--verbose";export COMPILER=gcc; ./support/docker_build.sh
- Mesos ReviewBot
On Nov. 24, 2015, 9:39 a.m., Guangya Liu wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40632/
> -----------------------------------------------------------
>
> (Updated Nov. 24, 2015, 9:39 a.m.)
>
>
> Review request for mesos and Klaus Ma.
>
>
> Repository: mesos
>
>
> Description
> -------
>
> WIP: Enabled oversubscribed resources for reservations in allocator.
>
>
> Diffs
> -----
>
> src/master/allocator/mesos/hierarchical.cpp aee8ced1fbfec8cf30cb939ff67fadfc6b08f37a
> src/tests/resources_tests.cpp dbd39cd5a6786682a7b528b6fea37ab78904cf12
>
> Diff: https://reviews.apache.org/r/40632/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Guangya Liu
>
>
Re: Review Request 40632: WIP: Enabled oversubscribed resources for
reservations in allocator.
Posted by Mesos ReviewBot <re...@mesos.apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40632/#review108783
-----------------------------------------------------------
Bad patch!
Reviews applied: [40632]
Failed command: ./support/apply-review.sh -n -r 40632
Error:
2015-12-03 05:57:49 URL:https://reviews.apache.org/r/40632/diff/raw/ [1068/1068] -> "40632.patch" [1]
error: patch failed: src/master/allocator/mesos/hierarchical.cpp:1031
error: src/master/allocator/mesos/hierarchical.cpp: patch does not apply
Failed to apply patch
- Mesos ReviewBot
On Dec. 3, 2015, 4:34 a.m., Guangya Liu wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40632/
> -----------------------------------------------------------
>
> (Updated Dec. 3, 2015, 4:34 a.m.)
>
>
> Review request for mesos and Klaus Ma.
>
>
> Repository: mesos
>
>
> Description
> -------
>
> WIP: Enabled oversubscribed resources for reservations in allocator.
>
>
> Diffs
> -----
>
> src/master/allocator/mesos/hierarchical.cpp aee8ced1fbfec8cf30cb939ff67fadfc6b08f37a
>
> Diff: https://reviews.apache.org/r/40632/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Guangya Liu
>
>
Re: Review Request 40632: WIP: Enabled oversubscribed resources for
reservations in allocator.
Posted by Mesos ReviewBot <re...@mesos.apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40632/#review109503
-----------------------------------------------------------
Bad patch!
Reviews applied: [40375, 40339]
Failed command: ./support/apply-review.sh -n -r 40339
Error:
2015-12-09 13:01:36 URL:https://reviews.apache.org/r/40339/diff/raw/ [26264/26264] -> "40339.patch" [1]
error: patch failed: src/tests/reservation_tests.cpp:406
error: src/tests/reservation_tests.cpp: patch does not apply
Failed to apply patch
- Mesos ReviewBot
On Dec. 9, 2015, 12:38 p.m., Guangya Liu wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40632/
> -----------------------------------------------------------
>
> (Updated Dec. 9, 2015, 12:38 p.m.)
>
>
> Review request for mesos and Klaus Ma.
>
>
> Repository: mesos
>
>
> Description
> -------
>
> WIP: Enabled oversubscribed resources for reservations in allocator.
>
>
> Diffs
> -----
>
> src/cli/execute.cpp ac933b6a8636af3356f3d6a99656c0185077c262
> src/master/allocator/mesos/hierarchical.cpp 5da825a1d578a9ee40b4985378fddb3c5fb3b416
>
> Diff: https://reviews.apache.org/r/40632/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Guangya Liu
>
>
Re: Review Request 40632: Enabled oversubscribed resources for
reservations in allocator.
Posted by Klaus Ma <kl...@gmail.com>.
> On Dec. 14, 2015, 10:14 a.m., Klaus Ma wrote:
> > src/master/allocator/mesos/hierarchical.cpp, line 431
> > <https://reviews.apache.org/r/40632/diff/6/?file=1161883#file1161883line431>
> >
> > Honestly, I'd like to separate optimistic resources into other counter in allocator; it's a litte different with oversubscription: here, one reserved resources are counting twice in total.
>
> Guangya Liu wrote:
> The allocation slack was decreased in 1304-1329
>
> Jian Qiu wrote:
> Just wondering the reason to set resources in allocaction slack when add/update slave. It seems to me that you just need to calculate available resources in allocate() and add allocation slack to the available there?
>
> Guangya Liu wrote:
> If added slave with optimistic offer phase 1 enabled, then I want to make sure that the log message in allocator is able to reflect that there are allocation slack resources. SO just initializt allocation slack resources in add/update slave and use it in allocate().
The result should be fine for this task; as we sync up online this morning, I'm confused by `slaves[slaveId].total` that included optimistic offer resources. I'd suggest to enhance our HLD on the counter in allocator; we also need to cover the code style in future :).
- Klaus
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40632/#review110154
-----------------------------------------------------------
On Dec. 14, 2015, 2:41 p.m., Guangya Liu wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40632/
> -----------------------------------------------------------
>
> (Updated Dec. 14, 2015, 2:41 p.m.)
>
>
> Review request for mesos, Artem Harutyunyan, Joris Van Remoortere, Joseph Wu, and Klaus Ma.
>
>
> Repository: mesos
>
>
> Description
> -------
>
> Enabled oversubscribed resources for reservations in allocator.
>
>
> Diffs
> -----
>
> src/master/allocator/mesos/hierarchical.cpp 5da825a1d578a9ee40b4985378fddb3c5fb3b416
>
> Diff: https://reviews.apache.org/r/40632/diff/
>
>
> Testing
> -------
>
> make
> make check
>
>
> Thanks,
>
> Guangya Liu
>
>
Re: Review Request 40632: WIP: Enabled oversubscribed resources for
reservations in allocator.
Posted by Jian Qiu <qi...@cn.ibm.com>.
> On Dec. 14, 2015, 2:14 a.m., Klaus Ma wrote:
> > src/master/allocator/mesos/hierarchical.cpp, line 431
> > <https://reviews.apache.org/r/40632/diff/6/?file=1161883#file1161883line431>
> >
> > Honestly, I'd like to separate optimistic resources into other counter in allocator; it's a litte different with oversubscription: here, one reserved resources are counting twice in total.
>
> Guangya Liu wrote:
> The allocation slack was decreased in 1304-1329
Just wondering the reason to set resources in allocaction slack when add/update slave. It seems to me that you just need to calculate available resources in allocate() and add allocation slack to the available there?
- Jian
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40632/#review110154
-----------------------------------------------------------
On Dec. 13, 2015, 11:35 a.m., Guangya Liu wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40632/
> -----------------------------------------------------------
>
> (Updated Dec. 13, 2015, 11:35 a.m.)
>
>
> Review request for mesos, Artem Harutyunyan, Joris Van Remoortere, Joseph Wu, and Klaus Ma.
>
>
> Repository: mesos
>
>
> Description
> -------
>
> WIP: Enabled oversubscribed resources for reservations in allocator.
>
>
> Diffs
> -----
>
> src/master/allocator/mesos/hierarchical.cpp 5da825a1d578a9ee40b4985378fddb3c5fb3b416
>
> Diff: https://reviews.apache.org/r/40632/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Guangya Liu
>
>
Re: Review Request 40632: WIP: Enabled oversubscribed resources for
reservations in allocator.
Posted by Guangya Liu <gy...@gmail.com>.
> On Dec. 14, 2015, 2:14 a.m., Klaus Ma wrote:
> > src/master/allocator/mesos/hierarchical.cpp, line 431
> > <https://reviews.apache.org/r/40632/diff/6/?file=1161883#file1161883line431>
> >
> > Honestly, I'd like to separate optimistic resources into other counter in allocator; it's a litte different with oversubscription: here, one reserved resources are counting twice in total.
>
> Guangya Liu wrote:
> The allocation slack was decreased in 1304-1329
>
> Jian Qiu wrote:
> Just wondering the reason to set resources in allocaction slack when add/update slave. It seems to me that you just need to calculate available resources in allocate() and add allocation slack to the available there?
If added slave with optimistic offer phase 1 enabled, then I want to make sure that the log message in allocator is able to reflect that there are allocation slack resources. SO just initializt allocation slack resources in add/update slave and use it in allocate().
- Guangya
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40632/#review110154
-----------------------------------------------------------
On Dec. 13, 2015, 11:35 a.m., Guangya Liu wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40632/
> -----------------------------------------------------------
>
> (Updated Dec. 13, 2015, 11:35 a.m.)
>
>
> Review request for mesos, Artem Harutyunyan, Joris Van Remoortere, Joseph Wu, and Klaus Ma.
>
>
> Repository: mesos
>
>
> Description
> -------
>
> WIP: Enabled oversubscribed resources for reservations in allocator.
>
>
> Diffs
> -----
>
> src/master/allocator/mesos/hierarchical.cpp 5da825a1d578a9ee40b4985378fddb3c5fb3b416
>
> Diff: https://reviews.apache.org/r/40632/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Guangya Liu
>
>
Re: Review Request 40632: WIP: Enabled oversubscribed resources for
reservations in allocator.
Posted by Guangya Liu <gy...@gmail.com>.
> On Dec. 14, 2015, 2:14 a.m., Klaus Ma wrote:
> > src/master/allocator/mesos/hierarchical.cpp, line 431
> > <https://reviews.apache.org/r/40632/diff/6/?file=1161883#file1161883line431>
> >
> > Honestly, I'd like to separate optimistic resources into other counter in allocator; it's a litte different with oversubscription: here, one reserved resources are counting twice in total.
The allocation slack was decreased in 1304-1329
- Guangya
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40632/#review110154
-----------------------------------------------------------
On Dec. 13, 2015, 11:35 a.m., Guangya Liu wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40632/
> -----------------------------------------------------------
>
> (Updated Dec. 13, 2015, 11:35 a.m.)
>
>
> Review request for mesos, Artem Harutyunyan, Joris Van Remoortere, Joseph Wu, and Klaus Ma.
>
>
> Repository: mesos
>
>
> Description
> -------
>
> WIP: Enabled oversubscribed resources for reservations in allocator.
>
>
> Diffs
> -----
>
> src/master/allocator/mesos/hierarchical.cpp 5da825a1d578a9ee40b4985378fddb3c5fb3b416
>
> Diff: https://reviews.apache.org/r/40632/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Guangya Liu
>
>
Re: Review Request 40632: WIP: Enabled oversubscribed resources for
reservations in allocator.
Posted by Klaus Ma <kl...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40632/#review110154
-----------------------------------------------------------
src/master/allocator/mesos/hierarchical.cpp (line 431)
<https://reviews.apache.org/r/40632/#comment169923>
Honestly, I'd like to separate optimistic resources into other counter in allocator; it's a litte different with oversubscription: here, one reserved resources are counting twice in total.
- Klaus Ma
On Dec. 13, 2015, 7:35 p.m., Guangya Liu wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40632/
> -----------------------------------------------------------
>
> (Updated Dec. 13, 2015, 7:35 p.m.)
>
>
> Review request for mesos, Artem Harutyunyan, Joris Van Remoortere, Joseph Wu, and Klaus Ma.
>
>
> Repository: mesos
>
>
> Description
> -------
>
> WIP: Enabled oversubscribed resources for reservations in allocator.
>
>
> Diffs
> -----
>
> src/master/allocator/mesos/hierarchical.cpp 5da825a1d578a9ee40b4985378fddb3c5fb3b416
>
> Diff: https://reviews.apache.org/r/40632/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Guangya Liu
>
>
Re: Review Request 40632: Enabled oversubscribed resources for
reservations in allocator.
Posted by Guangya Liu <gy...@gmail.com>.
> On 一月 7, 2016, 8:22 p.m., Joseph Wu wrote:
> > src/master/allocator/mesos/hierarchical.cpp, lines 438-443
> > <https://reviews.apache.org/r/40632/diff/10/?file=1180362#file1180362line438>
> >
> > Resources can be dynamically reserved, so ideally, you should recalculating the allocation slack in someplace like `updateAllocation`. The `RESERVE` operation specifically increases allocation slack.
Patch is here https://reviews.apache.org/r/41791/
> On 一月 7, 2016, 8:22 p.m., Joseph Wu wrote:
> > src/master/allocator/mesos/hierarchical.cpp, line 516
> > <https://reviews.apache.org/r/40632/diff/10/?file=1180362#file1180362line516>
> >
> > The Oversubscription logic here will overwrite anything you've done in `::addSlave`.
Patch is here https://reviews.apache.org/r/41847/
> On 一月 7, 2016, 8:22 p.m., Joseph Wu wrote:
> > src/master/allocator/mesos/hierarchical.cpp, lines 1305-1313
> > <https://reviews.apache.org/r/40632/diff/10/?file=1180362#file1180362line1305>
> >
> > If you make the changes above, this recalculation will be unnecessary. `resources.allocationSlack()` should be sufficient.
`resources.allocationSlack()` may not enough. If some task run on stateless reserved resources, the allocator should also remove those resources from allocation slack.
i.e.
total resources: `cpus(role1):100;mem(role1):100;cpus(*){ALLOCATION_SLACK}:100;mem(*){ALLOCATION_SLACK}:100`
allocated: `cpus(role1):30;mem(*){ALLOCATION_SLACK}:20`
Then the available resources should be: `cpus(role1):70;mem(role1):100;cpus(*){ALLOCATION_SLACK}:70;mem(*){ALLOCATION_SLACK}:80`
The allocation slack for cpus was also updated by decreasing 30 due to stateless reserved resources are used.
> On 一月 7, 2016, 8:22 p.m., Joseph Wu wrote:
> > src/master/allocator/mesos/hierarchical.cpp, lines 1343-1360
> > <https://reviews.apache.org/r/40632/diff/10/?file=1180362#file1180362line1343>
> >
> > There would be less code duplication if you did something like:
> > ```
> > if (!remainingAllocationSlack.empty()) {
> > resources = remainingAllocationSlack;
> > } else {
> > continue;
> > }
> > ```
Cool!
- Guangya
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40632/#review113303
-----------------------------------------------------------
On 一月 7, 2016, 11:27 a.m., Guangya Liu wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40632/
> -----------------------------------------------------------
>
> (Updated 一月 7, 2016, 11:27 a.m.)
>
>
> Review request for mesos, Ben Mahler, Artem Harutyunyan, Joris Van Remoortere, Joseph Wu, Klaus Ma, and Jian Qiu.
>
>
> Bugs: MESOS-4145
> https://issues.apache.org/jira/browse/MESOS-4145
>
>
> Repository: mesos
>
>
> Description
> -------
>
> Enabled oversubscribed resources for reservations in allocator.
>
>
> Diffs
> -----
>
> src/master/allocator/mesos/hierarchical.cpp df8bccaf2b8cfc0cb5ca18d4867371ae7a84c12f
> src/tests/hierarchical_allocator_tests.cpp e044f832c2c16e53e663c6ced5452649bb0dcb59
>
> Diff: https://reviews.apache.org/r/40632/diff/
>
>
> Testing
> -------
>
> make
> make check
>
>
> Thanks,
>
> Guangya Liu
>
>
Re: Review Request 40632: Enabled oversubscribed resources for
reservations in allocator.
Posted by Joseph Wu <jo...@mesosphere.io>.
> On Jan. 7, 2016, 12:22 p.m., Joseph Wu wrote:
> > I haven't taken a close look at the tests yet. However, it may benefit you to add tests (preferably in separate reviews) for:
> >
> > + Dynamic reservations + optimistic offers. Make sure the new reservations show up as allocation slack.
> > + Oversubscription + optimistic offers. Make sure neither overwrites the other.
> > + Quota + optimistic offers.
Oops, I didn't notice that the next few reviews do some of this already.
Can you add a note to the "Testing Done" and "Description" saying that the changes to allocator are spread out over several reviews?
- Joseph
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40632/#review113303
-----------------------------------------------------------
On Jan. 7, 2016, 3:27 a.m., Guangya Liu wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40632/
> -----------------------------------------------------------
>
> (Updated Jan. 7, 2016, 3:27 a.m.)
>
>
> Review request for mesos, Ben Mahler, Artem Harutyunyan, Joris Van Remoortere, Joseph Wu, Klaus Ma, and Jian Qiu.
>
>
> Bugs: MESOS-4145
> https://issues.apache.org/jira/browse/MESOS-4145
>
>
> Repository: mesos
>
>
> Description
> -------
>
> Enabled oversubscribed resources for reservations in allocator.
>
>
> Diffs
> -----
>
> src/master/allocator/mesos/hierarchical.cpp df8bccaf2b8cfc0cb5ca18d4867371ae7a84c12f
> src/tests/hierarchical_allocator_tests.cpp e044f832c2c16e53e663c6ced5452649bb0dcb59
>
> Diff: https://reviews.apache.org/r/40632/diff/
>
>
> Testing
> -------
>
> make
> make check
>
>
> Thanks,
>
> Guangya Liu
>
>
Re: Review Request 40632: Enabled oversubscribed resources for
reservations in allocator.
Posted by Joseph Wu <jo...@mesosphere.io>.
> On Jan. 7, 2016, 12:22 p.m., Joseph Wu wrote:
> > src/master/allocator/mesos/hierarchical.cpp, lines 1305-1313
> > <https://reviews.apache.org/r/40632/diff/10/?file=1180362#file1180362line1305>
> >
> > If you make the changes above, this recalculation will be unnecessary. `resources.allocationSlack()` should be sufficient.
>
> Guangya Liu wrote:
> `resources.allocationSlack()` may not enough. If some task run on stateless reserved resources, the allocator should also remove those resources from allocation slack.
>
> i.e.
> total resources: `cpus(role1):100;mem(role1):100;cpus(*){ALLOCATION_SLACK}:100;mem(*){ALLOCATION_SLACK}:100`
> allocated: `cpus(role1):30;mem(*){ALLOCATION_SLACK}:20`
> Then the available resources should be: `cpus(role1):70;mem(role1):100;cpus(*){ALLOCATION_SLACK}:70;mem(*){ALLOCATION_SLACK}:80`
>
> The allocation slack for cpus was also updated by decreasing 30 due to stateless reserved resources are used.
I'm going to drop this since the updated diffs make this comment somewhat obsolete.
- Joseph
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40632/#review113303
-----------------------------------------------------------
On Jan. 13, 2016, 4:39 a.m., Guangya Liu wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40632/
> -----------------------------------------------------------
>
> (Updated Jan. 13, 2016, 4:39 a.m.)
>
>
> Review request for mesos, Ben Mahler, Artem Harutyunyan, Joris Van Remoortere, Joseph Wu, Klaus Ma, and Jian Qiu.
>
>
> Bugs: MESOS-4145
> https://issues.apache.org/jira/browse/MESOS-4145
>
>
> Repository: mesos
>
>
> Description
> -------
>
> Enabled oversubscribed resources for reservations in allocator.
>
> There are five patches handling the allocator part:
> 1) This patch handles send offer, add slave and quota integration test.
> 2) https://reviews.apache.org/r/41847/ Handles the case of update slave.
> 3) https://reviews.apache.org/r/41791/ Handles the case of dynamic reservations (1/3).
> 4) https://reviews.apache.org/r/42113/ Handle unreserve logic for dynamic reservation (2/3)
> 5) https://reviews.apache.org/r/42194/ Handle unreserve logic for dynamic reservation (3/3)
>
>
> Diffs
> -----
>
> src/master/allocator/mesos/hierarchical.cpp d541bfa3f4190865c65d35c9d1ffdb8a3f194056
> src/tests/hierarchical_allocator_tests.cpp e044f832c2c16e53e663c6ced5452649bb0dcb59
>
> Diff: https://reviews.apache.org/r/40632/diff/
>
>
> Testing
> -------
>
> make
> make check
> GLOG_v=2 ./bin/mesos-tests.sh --gtest_filter="HierarchicalAllocatorTest.*" --verbose
>
>
> Thanks,
>
> Guangya Liu
>
>
Re: Review Request 40632: Enabled oversubscribed resources for
reservations in allocator.
Posted by Joseph Wu <jo...@mesosphere.io>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40632/#review113303
-----------------------------------------------------------
I haven't taken a close look at the tests yet. However, it may benefit you to add tests (preferably in separate reviews) for:
+ Dynamic reservations + optimistic offers. Make sure the new reservations show up as allocation slack.
+ Oversubscription + optimistic offers. Make sure neither overwrites the other.
+ Quota + optimistic offers.
src/master/allocator/mesos/hierarchical.cpp (lines 438 - 443)
<https://reviews.apache.org/r/40632/#comment173841>
Resources can be dynamically reserved, so ideally, you should recalculating the allocation slack in someplace like `updateAllocation`. The `RESERVE` operation specifically increases allocation slack.
src/master/allocator/mesos/hierarchical.cpp (line 516)
<https://reviews.apache.org/r/40632/#comment173835>
The Oversubscription logic here will overwrite anything you've done in `::addSlave`.
src/master/allocator/mesos/hierarchical.cpp (lines 1305 - 1313)
<https://reviews.apache.org/r/40632/#comment173846>
If you make the changes above, this recalculation will be unnecessary. `resources.allocationSlack()` should be sufficient.
src/master/allocator/mesos/hierarchical.cpp (lines 1343 - 1360)
<https://reviews.apache.org/r/40632/#comment173855>
There would be less code duplication if you did something like:
```
if (!remainingAllocationSlack.empty()) {
resources = remainingAllocationSlack;
} else {
continue;
}
```
- Joseph Wu
On Jan. 7, 2016, 3:27 a.m., Guangya Liu wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40632/
> -----------------------------------------------------------
>
> (Updated Jan. 7, 2016, 3:27 a.m.)
>
>
> Review request for mesos, Ben Mahler, Artem Harutyunyan, Joris Van Remoortere, Joseph Wu, Klaus Ma, and Jian Qiu.
>
>
> Bugs: MESOS-4145
> https://issues.apache.org/jira/browse/MESOS-4145
>
>
> Repository: mesos
>
>
> Description
> -------
>
> Enabled oversubscribed resources for reservations in allocator.
>
>
> Diffs
> -----
>
> src/master/allocator/mesos/hierarchical.cpp df8bccaf2b8cfc0cb5ca18d4867371ae7a84c12f
> src/tests/hierarchical_allocator_tests.cpp e044f832c2c16e53e663c6ced5452649bb0dcb59
>
> Diff: https://reviews.apache.org/r/40632/diff/
>
>
> Testing
> -------
>
> make
> make check
>
>
> Thanks,
>
> Guangya Liu
>
>
Re: Review Request 40632: Enabled oversubscribed resources for
reservations in allocator.
Posted by Guangya Liu <gy...@gmail.com>.
> On 一月 16, 2016, 2:11 a.m., Joseph Wu wrote:
> > src/master/allocator/mesos/hierarchical.cpp, lines 1291-1319
> > <https://reviews.apache.org/r/40632/diff/15/?file=1195627#file1195627line1291>
> >
> > None of this should be necessary:
> >
> > 1) You should have all the allocation slack inside `available`.
> > 2) We don't have allocation slack ACLs in the MVP, so all allocation slack should fall under `available.unreserved`. If we do have ACLs, some of the allocation slack will be accounted for in `available.reserved(role)`.
> > 3) You shouldn't need to recalculate this during `::allocate`.
> >
> > If any of the above are not true, you probably have a problem in the `Resource` helpers or in `::addSlave`.
>
> Guangya Liu wrote:
> We can take a look at the following cases:
> 1) total resources cpus(r1):100
> 2) register a framework f1, cycle 1: allocated cpus(r1):100;cpu(*){ALLOCATION_SLACK}:100 to f1
> 3) f1 recovered cpu(*){ALLOCATION_SLACK}:100 back
> 4) register another framwork f2, f2 get nothing here as f1 used up all reserved resoures and there is no allocation slack now.
>
> The above logic make sure that step 4) can always get the latest remaining allocation slack.
>
> Joseph Wu wrote:
> The step 3) in your case is presumably because you haven't implemented all the allocator methods yet (as of this review in the chain), like `recoverResources`, `updateAllocation`, and `updateAvailable`.
>
> If so, you should consider moving the test additions/changes before/after (this lets us know what your intended behavior is) the allocator changes.
>
> Guangya Liu wrote:
> I was updating the comments here to make it more clear.
>
> // Calculate the `remainingAllocationSlack` if the framework can
> // use revocable resources and reservation oversubscription also
> // enabled. The `remainingAllocationSlack` need to exclude the
> // stateless reserved resources allocated in previous allocation
> // cycle.
>
> Joseph Wu wrote:
> I would still **highly recommend** that you don't recalculate the allocation slack inside a doubly-nested for-loop.
>
> And I'll reinterate: if you can't find the allocation slack inside the `available` Resources object, then you're doing something wrong. You should probably double-check if this is the case.
The reason I need to recalcuate this is because for a role, let's say `r1`, the `r1`'s resources including both `reserved` and `allocation slack` would be send out as an offer for `r1` if a framework is using `r1`.
A case is as this:
1) agent1 with cpus(r1):100
2) add agent1, then agent1 reosurces will become cpus(r1):100;cpus(*){ALLOCATION_SLACK}:100
3) Register a framework with r1
4) The allocator will send out resources as cpus(r1):100;cpus(*){ALLOCATION_SLACK}:100. For here, as we want thr framework can also run revocable tasks, so here, we are sending both reserved and revocable resources to framework and let the framework select the resources.
5) The allcator recover the allocation slack
6) When allocate again, the allocator need to recalculate the allocation slack and found it was 0
But if we do not send out both one role's `reserved` and `allocation slack`, we can simply use the `allocation slack` in `available` resources.
This is a question that we need to discuss today: Do we need to send out the offer in one role include both `reserved` and `allocation slack` ?
- Guangya
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40632/#review114831
-----------------------------------------------------------
On 一月 20, 2016, 6:25 a.m., Guangya Liu wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40632/
> -----------------------------------------------------------
>
> (Updated 一月 20, 2016, 6:25 a.m.)
>
>
> Review request for mesos, Ben Mahler, Artem Harutyunyan, Joris Van Remoortere, Joseph Wu, Klaus Ma, and Jian Qiu.
>
>
> Bugs: MESOS-4145
> https://issues.apache.org/jira/browse/MESOS-4145
>
>
> Repository: mesos
>
>
> Description
> -------
>
> Enabled oversubscribed resources for reservations in allocator.
> The allocator part including 5 patches:
> 1) https://reviews.apache.org/r/40632 Enabled oversubscribed resources for reservations in allocator
> 2) https://reviews.apache.org/r/41847 Updated allocation slack when slave was updated.
> 3) https://reviews.apache.org/r/41791 Updated allocation slack for dynamic reserve (1/3).
> 4) https://reviews.apache.org/r/42113 Handle unreserve logic for dynamic reservation (2/3).
> 5) https://reviews.apache.org/r/42194 Handle unreserve logic for dynamic reservation (3/3).
>
>
> Diffs
> -----
>
> src/master/allocator/mesos/hierarchical.cpp e32ee4aa3ed9793bb5a99233e699e5cc2bdd796b
> src/tests/hierarchical_allocator_tests.cpp 953712149bd951789beb29c72779c4ac65aa48dc
>
> Diff: https://reviews.apache.org/r/40632/diff/
>
>
> Testing
> -------
>
> make
> make check
> GLOG_v=2 ./bin/mesos-tests.sh --gtest_filter="HierarchicalAllocatorTest.*" --verbose --gtest_repeat=100 --gtest_shuffle
>
>
> Thanks,
>
> Guangya Liu
>
>
Re: Review Request 40632: Enabled oversubscribed resources for
reservations in allocator.
Posted by Guangya Liu <gy...@gmail.com>.
> On 一月 16, 2016, 2:11 a.m., Joseph Wu wrote:
> > src/tests/hierarchical_allocator_tests.cpp, line 993
> > <https://reviews.apache.org/r/40632/diff/15/?file=1195628#file1195628line993>
> >
> > Can you pull this out into an entirely separate (not part of this review chain) review?
> >
> > We can get the test cleanup merged faster.
> > Ditto for below.
All of the related clean up are moved to https://reviews.apache.org/r/42386/ but still in this chain, as following patches related to hierarchical test highly depend on this patch. Hope it is OK.
> On 一月 16, 2016, 2:11 a.m., Joseph Wu wrote:
> > src/master/allocator/mesos/hierarchical.cpp, lines 1291-1319
> > <https://reviews.apache.org/r/40632/diff/15/?file=1195627#file1195627line1291>
> >
> > None of this should be necessary:
> >
> > 1) You should have all the allocation slack inside `available`.
> > 2) We don't have allocation slack ACLs in the MVP, so all allocation slack should fall under `available.unreserved`. If we do have ACLs, some of the allocation slack will be accounted for in `available.reserved(role)`.
> > 3) You shouldn't need to recalculate this during `::allocate`.
> >
> > If any of the above are not true, you probably have a problem in the `Resource` helpers or in `::addSlave`.
We can take a look at the following cases:
1) total resources cpus(r1):100
2) register a framework f1, cycle 1: allocated cpus(r1):100;cpu(*){ALLOCATION_SLACK}:100 to f1
3) f1 recovered cpu(*){ALLOCATION_SLACK}:100 back
4) register another framwork f2, f2 get nothing here as f1 used up all reserved resoures and there is no allocation slack now.
The above logic make sure that step 4) can always get the latest remaining allocation slack.
- Guangya
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40632/#review114831
-----------------------------------------------------------
On 一月 16, 2016, 7:51 a.m., Guangya Liu wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40632/
> -----------------------------------------------------------
>
> (Updated 一月 16, 2016, 7:51 a.m.)
>
>
> Review request for mesos, Ben Mahler, Artem Harutyunyan, Joris Van Remoortere, Joseph Wu, Klaus Ma, and Jian Qiu.
>
>
> Bugs: MESOS-4145
> https://issues.apache.org/jira/browse/MESOS-4145
>
>
> Repository: mesos
>
>
> Description
> -------
>
> Enabled oversubscribed resources for reservations in allocator.
> The allocator part including 5 patches:
> 1) https://reviews.apache.org/r/40632 Enabled oversubscribed resources for reservations in allocator
> 2) https://reviews.apache.org/r/41847 Updated allocation slack when slave was updated.
> 3) https://reviews.apache.org/r/41791 Updated allocation slack for dynamic reserve (1/3).
> 4) https://reviews.apache.org/r/42113 Handle unreserve logic for dynamic reservation (2/3).
> 5) https://reviews.apache.org/r/42194 Handle unreserve logic for dynamic reservation (3/3).
>
>
> Diffs
> -----
>
> src/master/allocator/mesos/hierarchical.cpp 72e69a0f42dd724713f2a7a75f1b92ef16eb5569
> src/tests/hierarchical_allocator_tests.cpp 9362dd306497ba01e0f387c3862456cdcac6f863
>
> Diff: https://reviews.apache.org/r/40632/diff/
>
>
> Testing
> -------
>
> make
> make check
> GLOG_v=2 ./bin/mesos-tests.sh --gtest_filter="HierarchicalAllocatorTest.*" --verbose
>
>
> Thanks,
>
> Guangya Liu
>
>
Re: Review Request 40632: Enabled oversubscribed resources for
reservations in allocator.
Posted by Joseph Wu <jo...@mesosphere.io>.
> On Jan. 15, 2016, 6:11 p.m., Joseph Wu wrote:
> > src/master/allocator/mesos/hierarchical.cpp, lines 1291-1319
> > <https://reviews.apache.org/r/40632/diff/15/?file=1195627#file1195627line1291>
> >
> > None of this should be necessary:
> >
> > 1) You should have all the allocation slack inside `available`.
> > 2) We don't have allocation slack ACLs in the MVP, so all allocation slack should fall under `available.unreserved`. If we do have ACLs, some of the allocation slack will be accounted for in `available.reserved(role)`.
> > 3) You shouldn't need to recalculate this during `::allocate`.
> >
> > If any of the above are not true, you probably have a problem in the `Resource` helpers or in `::addSlave`.
>
> Guangya Liu wrote:
> We can take a look at the following cases:
> 1) total resources cpus(r1):100
> 2) register a framework f1, cycle 1: allocated cpus(r1):100;cpu(*){ALLOCATION_SLACK}:100 to f1
> 3) f1 recovered cpu(*){ALLOCATION_SLACK}:100 back
> 4) register another framwork f2, f2 get nothing here as f1 used up all reserved resoures and there is no allocation slack now.
>
> The above logic make sure that step 4) can always get the latest remaining allocation slack.
The step 3) in your case is presumably because you haven't implemented all the allocator methods yet (as of this review in the chain), like `recoverResources`, `updateAllocation`, and `updateAvailable`.
If so, you should consider moving the test additions/changes before/after (this lets us know what your intended behavior is) the allocator changes.
- Joseph
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40632/#review114831
-----------------------------------------------------------
On Jan. 19, 2016, 12:03 a.m., Guangya Liu wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40632/
> -----------------------------------------------------------
>
> (Updated Jan. 19, 2016, 12:03 a.m.)
>
>
> Review request for mesos, Ben Mahler, Artem Harutyunyan, Joris Van Remoortere, Joseph Wu, Klaus Ma, and Jian Qiu.
>
>
> Bugs: MESOS-4145
> https://issues.apache.org/jira/browse/MESOS-4145
>
>
> Repository: mesos
>
>
> Description
> -------
>
> Enabled oversubscribed resources for reservations in allocator.
> The allocator part including 5 patches:
> 1) https://reviews.apache.org/r/40632 Enabled oversubscribed resources for reservations in allocator
> 2) https://reviews.apache.org/r/41847 Updated allocation slack when slave was updated.
> 3) https://reviews.apache.org/r/41791 Updated allocation slack for dynamic reserve (1/3).
> 4) https://reviews.apache.org/r/42113 Handle unreserve logic for dynamic reservation (2/3).
> 5) https://reviews.apache.org/r/42194 Handle unreserve logic for dynamic reservation (3/3).
>
>
> Diffs
> -----
>
> src/master/allocator/mesos/hierarchical.cpp 48acde69b1a2f305b568a7e322a58708063dd30a
> src/tests/hierarchical_allocator_tests.cpp 9362dd306497ba01e0f387c3862456cdcac6f863
>
> Diff: https://reviews.apache.org/r/40632/diff/
>
>
> Testing
> -------
>
> make
> make check
> GLOG_v=2 ./bin/mesos-tests.sh --gtest_filter="HierarchicalAllocatorTest.*" --verbose
>
>
> Thanks,
>
> Guangya Liu
>
>
Re: Review Request 40632: Enabled oversubscribed resources for
reservations in allocator.
Posted by Guangya Liu <gy...@gmail.com>.
> On 一月 16, 2016, 2:11 a.m., Joseph Wu wrote:
> > src/master/allocator/mesos/hierarchical.cpp, lines 1291-1319
> > <https://reviews.apache.org/r/40632/diff/15/?file=1195627#file1195627line1291>
> >
> > None of this should be necessary:
> >
> > 1) You should have all the allocation slack inside `available`.
> > 2) We don't have allocation slack ACLs in the MVP, so all allocation slack should fall under `available.unreserved`. If we do have ACLs, some of the allocation slack will be accounted for in `available.reserved(role)`.
> > 3) You shouldn't need to recalculate this during `::allocate`.
> >
> > If any of the above are not true, you probably have a problem in the `Resource` helpers or in `::addSlave`.
>
> Guangya Liu wrote:
> We can take a look at the following cases:
> 1) total resources cpus(r1):100
> 2) register a framework f1, cycle 1: allocated cpus(r1):100;cpu(*){ALLOCATION_SLACK}:100 to f1
> 3) f1 recovered cpu(*){ALLOCATION_SLACK}:100 back
> 4) register another framwork f2, f2 get nothing here as f1 used up all reserved resoures and there is no allocation slack now.
>
> The above logic make sure that step 4) can always get the latest remaining allocation slack.
>
> Joseph Wu wrote:
> The step 3) in your case is presumably because you haven't implemented all the allocator methods yet (as of this review in the chain), like `recoverResources`, `updateAllocation`, and `updateAvailable`.
>
> If so, you should consider moving the test additions/changes before/after (this lets us know what your intended behavior is) the allocator changes.
I was updating the comments here to make it more clear.
// Calculate the `remainingAllocationSlack` if the framework can
// use revocable resources and reservation oversubscription also
// enabled. The `remainingAllocationSlack` need to exclude the
// stateless reserved resources allocated in previous allocation
// cycle.
- Guangya
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40632/#review114831
-----------------------------------------------------------
On 一月 19, 2016, 8:03 a.m., Guangya Liu wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40632/
> -----------------------------------------------------------
>
> (Updated 一月 19, 2016, 8:03 a.m.)
>
>
> Review request for mesos, Ben Mahler, Artem Harutyunyan, Joris Van Remoortere, Joseph Wu, Klaus Ma, and Jian Qiu.
>
>
> Bugs: MESOS-4145
> https://issues.apache.org/jira/browse/MESOS-4145
>
>
> Repository: mesos
>
>
> Description
> -------
>
> Enabled oversubscribed resources for reservations in allocator.
> The allocator part including 5 patches:
> 1) https://reviews.apache.org/r/40632 Enabled oversubscribed resources for reservations in allocator
> 2) https://reviews.apache.org/r/41847 Updated allocation slack when slave was updated.
> 3) https://reviews.apache.org/r/41791 Updated allocation slack for dynamic reserve (1/3).
> 4) https://reviews.apache.org/r/42113 Handle unreserve logic for dynamic reservation (2/3).
> 5) https://reviews.apache.org/r/42194 Handle unreserve logic for dynamic reservation (3/3).
>
>
> Diffs
> -----
>
> src/master/allocator/mesos/hierarchical.cpp 48acde69b1a2f305b568a7e322a58708063dd30a
> src/tests/hierarchical_allocator_tests.cpp 9362dd306497ba01e0f387c3862456cdcac6f863
>
> Diff: https://reviews.apache.org/r/40632/diff/
>
>
> Testing
> -------
>
> make
> make check
> GLOG_v=2 ./bin/mesos-tests.sh --gtest_filter="HierarchicalAllocatorTest.*" --verbose
>
>
> Thanks,
>
> Guangya Liu
>
>
Re: Review Request 40632: Enabled oversubscribed resources for
reservations in allocator.
Posted by Joseph Wu <jo...@mesosphere.io>.
> On Jan. 15, 2016, 6:11 p.m., Joseph Wu wrote:
> > src/master/allocator/mesos/hierarchical.cpp, lines 1291-1319
> > <https://reviews.apache.org/r/40632/diff/15/?file=1195627#file1195627line1291>
> >
> > None of this should be necessary:
> >
> > 1) You should have all the allocation slack inside `available`.
> > 2) We don't have allocation slack ACLs in the MVP, so all allocation slack should fall under `available.unreserved`. If we do have ACLs, some of the allocation slack will be accounted for in `available.reserved(role)`.
> > 3) You shouldn't need to recalculate this during `::allocate`.
> >
> > If any of the above are not true, you probably have a problem in the `Resource` helpers or in `::addSlave`.
>
> Guangya Liu wrote:
> We can take a look at the following cases:
> 1) total resources cpus(r1):100
> 2) register a framework f1, cycle 1: allocated cpus(r1):100;cpu(*){ALLOCATION_SLACK}:100 to f1
> 3) f1 recovered cpu(*){ALLOCATION_SLACK}:100 back
> 4) register another framwork f2, f2 get nothing here as f1 used up all reserved resoures and there is no allocation slack now.
>
> The above logic make sure that step 4) can always get the latest remaining allocation slack.
>
> Joseph Wu wrote:
> The step 3) in your case is presumably because you haven't implemented all the allocator methods yet (as of this review in the chain), like `recoverResources`, `updateAllocation`, and `updateAvailable`.
>
> If so, you should consider moving the test additions/changes before/after (this lets us know what your intended behavior is) the allocator changes.
>
> Guangya Liu wrote:
> I was updating the comments here to make it more clear.
>
> // Calculate the `remainingAllocationSlack` if the framework can
> // use revocable resources and reservation oversubscription also
> // enabled. The `remainingAllocationSlack` need to exclude the
> // stateless reserved resources allocated in previous allocation
> // cycle.
I would still **highly recommend** that you don't recalculate the allocation slack inside a doubly-nested for-loop.
And I'll reinterate: if you can't find the allocation slack inside the `available` Resources object, then you're doing something wrong. You should probably double-check if this is the case.
- Joseph
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40632/#review114831
-----------------------------------------------------------
On Jan. 19, 2016, 10:25 p.m., Guangya Liu wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40632/
> -----------------------------------------------------------
>
> (Updated Jan. 19, 2016, 10:25 p.m.)
>
>
> Review request for mesos, Ben Mahler, Artem Harutyunyan, Joris Van Remoortere, Joseph Wu, Klaus Ma, and Jian Qiu.
>
>
> Bugs: MESOS-4145
> https://issues.apache.org/jira/browse/MESOS-4145
>
>
> Repository: mesos
>
>
> Description
> -------
>
> Enabled oversubscribed resources for reservations in allocator.
> The allocator part including 5 patches:
> 1) https://reviews.apache.org/r/40632 Enabled oversubscribed resources for reservations in allocator
> 2) https://reviews.apache.org/r/41847 Updated allocation slack when slave was updated.
> 3) https://reviews.apache.org/r/41791 Updated allocation slack for dynamic reserve (1/3).
> 4) https://reviews.apache.org/r/42113 Handle unreserve logic for dynamic reservation (2/3).
> 5) https://reviews.apache.org/r/42194 Handle unreserve logic for dynamic reservation (3/3).
>
>
> Diffs
> -----
>
> src/master/allocator/mesos/hierarchical.cpp e32ee4aa3ed9793bb5a99233e699e5cc2bdd796b
> src/tests/hierarchical_allocator_tests.cpp 953712149bd951789beb29c72779c4ac65aa48dc
>
> Diff: https://reviews.apache.org/r/40632/diff/
>
>
> Testing
> -------
>
> make
> make check
> GLOG_v=2 ./bin/mesos-tests.sh --gtest_filter="HierarchicalAllocatorTest.*" --verbose --gtest_repeat=100 --gtest_shuffle
>
>
> Thanks,
>
> Guangya Liu
>
>
Re: Review Request 40632: Enabled oversubscribed resources for
reservations in allocator.
Posted by Joseph Wu <jo...@mesosphere.io>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40632/#review114831
-----------------------------------------------------------
src/master/allocator/mesos/hierarchical.cpp (lines 1291 - 1319)
<https://reviews.apache.org/r/40632/#comment175675>
None of this should be necessary:
1) You should have all the allocation slack inside `available`.
2) We don't have allocation slack ACLs in the MVP, so all allocation slack should fall under `available.unreserved`. If we do have ACLs, some of the allocation slack will be accounted for in `available.reserved(role)`.
3) You shouldn't need to recalculate this during `::allocate`.
If any of the above are not true, you probably have a problem in the `Resource` helpers or in `::addSlave`.
src/master/allocator/mesos/hierarchical.cpp (lines 1337 - 1340)
<https://reviews.apache.org/r/40632/#comment175676>
Suggestion:
Quota does not affect revocable resources. If the non-revocable component of this offer would exceed quota, the revocable portion can still be offered.
src/master/allocator/mesos/hierarchical.cpp (lines 1341 - 1346)
<https://reviews.apache.org/r/40632/#comment175677>
This should be something resembling:
```
Resources revocable = resources.revocable();
if (revocable.empty()) {
continue;
} else {
// Offer just the revocable resources.
resources = revocable;
}
```
src/tests/hierarchical_allocator_tests.cpp (lines 146 - 155)
<https://reviews.apache.org/r/40632/#comment175678>
This change should probably go in the previous review.
src/tests/hierarchical_allocator_tests.cpp (line 993)
<https://reviews.apache.org/r/40632/#comment175679>
Can you pull this out into an entirely separate (not part of this review chain) review?
We can get the test cleanup merged faster.
Ditto for below.
src/tests/hierarchical_allocator_tests.cpp (line 1009)
<https://reviews.apache.org/r/40632/#comment175680>
Test cleanup -> separate review.
src/tests/hierarchical_allocator_tests.cpp (line 1101)
<https://reviews.apache.org/r/40632/#comment175681>
Test cleanup -> separate review.
src/tests/hierarchical_allocator_tests.cpp (line 2067)
<https://reviews.apache.org/r/40632/#comment175682>
Test cleanup -> separate review.
- Joseph Wu
On Jan. 13, 2016, 4:39 a.m., Guangya Liu wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40632/
> -----------------------------------------------------------
>
> (Updated Jan. 13, 2016, 4:39 a.m.)
>
>
> Review request for mesos, Ben Mahler, Artem Harutyunyan, Joris Van Remoortere, Joseph Wu, Klaus Ma, and Jian Qiu.
>
>
> Bugs: MESOS-4145
> https://issues.apache.org/jira/browse/MESOS-4145
>
>
> Repository: mesos
>
>
> Description
> -------
>
> Enabled oversubscribed resources for reservations in allocator.
>
> There are five patches handling the allocator part:
> 1) This patch handles send offer, add slave and quota integration test.
> 2) https://reviews.apache.org/r/41847/ Handles the case of update slave.
> 3) https://reviews.apache.org/r/41791/ Handles the case of dynamic reservations (1/3).
> 4) https://reviews.apache.org/r/42113/ Handle unreserve logic for dynamic reservation (2/3)
> 5) https://reviews.apache.org/r/42194/ Handle unreserve logic for dynamic reservation (3/3)
>
>
> Diffs
> -----
>
> src/master/allocator/mesos/hierarchical.cpp d541bfa3f4190865c65d35c9d1ffdb8a3f194056
> src/tests/hierarchical_allocator_tests.cpp e044f832c2c16e53e663c6ced5452649bb0dcb59
>
> Diff: https://reviews.apache.org/r/40632/diff/
>
>
> Testing
> -------
>
> make
> make check
> GLOG_v=2 ./bin/mesos-tests.sh --gtest_filter="HierarchicalAllocatorTest.*" --verbose
>
>
> Thanks,
>
> Guangya Liu
>
>
Re: Review Request 40632: Enabled oversubscribed resources for
reservations in allocator.
Posted by Guangya Liu <gy...@gmail.com>.
> On 一月 19, 2016, 11:09 p.m., Joseph Wu wrote:
> > src/tests/hierarchical_allocator_tests.cpp, lines 1995-1997
> > <https://reviews.apache.org/r/40632/diff/18/?file=1201213#file1201213line1995>
> >
> > Consider moving this test to after you've implemented optimistic offers for `Allocator::recoverResources`.
I think that we can keep this test cases which focusing on recovering allocation slack. The update for `Allocator::recoverResources` is only used when shrinking dynamic reservations, and the allocation slack based on those shrinked dynamic reservations are being used by some revocable tasks.
Please refer to https://reviews.apache.org/r/42113/ and https://reviews.apache.org/r/42194/ for detail.
> On 一月 19, 2016, 11:09 p.m., Joseph Wu wrote:
> > src/tests/hierarchical_allocator_tests.cpp, lines 2045-2047
> > <https://reviews.apache.org/r/40632/diff/18/?file=1201213#file1201213line2045>
> >
> > Consider moving this test to after you've implemented optimistic offers for `Allocator::recoverResources`.
Same as aobove.
- Guangya
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40632/#review115263
-----------------------------------------------------------
On 一月 19, 2016, 8:03 a.m., Guangya Liu wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40632/
> -----------------------------------------------------------
>
> (Updated 一月 19, 2016, 8:03 a.m.)
>
>
> Review request for mesos, Ben Mahler, Artem Harutyunyan, Joris Van Remoortere, Joseph Wu, Klaus Ma, and Jian Qiu.
>
>
> Bugs: MESOS-4145
> https://issues.apache.org/jira/browse/MESOS-4145
>
>
> Repository: mesos
>
>
> Description
> -------
>
> Enabled oversubscribed resources for reservations in allocator.
> The allocator part including 5 patches:
> 1) https://reviews.apache.org/r/40632 Enabled oversubscribed resources for reservations in allocator
> 2) https://reviews.apache.org/r/41847 Updated allocation slack when slave was updated.
> 3) https://reviews.apache.org/r/41791 Updated allocation slack for dynamic reserve (1/3).
> 4) https://reviews.apache.org/r/42113 Handle unreserve logic for dynamic reservation (2/3).
> 5) https://reviews.apache.org/r/42194 Handle unreserve logic for dynamic reservation (3/3).
>
>
> Diffs
> -----
>
> src/master/allocator/mesos/hierarchical.cpp 48acde69b1a2f305b568a7e322a58708063dd30a
> src/tests/hierarchical_allocator_tests.cpp 9362dd306497ba01e0f387c3862456cdcac6f863
>
> Diff: https://reviews.apache.org/r/40632/diff/
>
>
> Testing
> -------
>
> make
> make check
> GLOG_v=2 ./bin/mesos-tests.sh --gtest_filter="HierarchicalAllocatorTest.*" --verbose
>
>
> Thanks,
>
> Guangya Liu
>
>
Re: Review Request 40632: Enabled oversubscribed resources for
reservations in allocator.
Posted by Joseph Wu <jo...@mesosphere.io>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40632/#review115263
-----------------------------------------------------------
src/tests/hierarchical_allocator_tests.cpp (lines 1918 - 1919)
<https://reviews.apache.org/r/40632/#comment176206>
Note: This separation comment is probably unnecessary. As long as the new tests are named properly (i.e. AllocationSlack...) we can tell them apart quite easily.
src/tests/hierarchical_allocator_tests.cpp (line 1922)
<https://reviews.apache.org/r/40632/#comment176207>
s/resources on a slave that are statically reserved/statically reserved resources/
src/tests/hierarchical_allocator_tests.cpp (lines 1924 - 1925)
<https://reviews.apache.org/r/40632/#comment176211>
s/to any role and/to any role, and/
s/for a role are only offered to frameworks in that role/are still only allocated to the reserved role/
src/tests/hierarchical_allocator_tests.cpp (lines 1952 - 1960)
<https://reviews.apache.org/r/40632/#comment176212>
It would be great to have a comment above this :)
src/tests/hierarchical_allocator_tests.cpp (lines 1995 - 1997)
<https://reviews.apache.org/r/40632/#comment176198>
Consider moving this test to after you've implemented optimistic offers for `Allocator::recoverResources`.
src/tests/hierarchical_allocator_tests.cpp (lines 2045 - 2047)
<https://reviews.apache.org/r/40632/#comment176199>
Consider moving this test to after you've implemented optimistic offers for `Allocator::recoverResources`.
src/tests/hierarchical_allocator_tests.cpp (line 2089)
<https://reviews.apache.org/r/40632/#comment176213>
s/was not count in Quota/can be allocated regardless of Quota/
src/tests/hierarchical_allocator_tests.cpp (line 2107)
<https://reviews.apache.org/r/40632/#comment176215>
It may improve readability to split up that big comment. Suggestion:
Set quota for the quota'd role. In the first allocation cycle, this effectively prevents `agent1` from being offered to any role except the quota'd role (as doing so would put the quota'd role under quota).
src/tests/hierarchical_allocator_tests.cpp (line 2111)
<https://reviews.apache.org/r/40632/#comment176218>
Continued suggestion:
Add a framework that can only be offered non-revocable resources from `agent2`.
src/tests/hierarchical_allocator_tests.cpp (lines 2116 - 2123)
<https://reviews.apache.org/r/40632/#comment176221>
Continued suggestion:
Check that `framework1` gets offered resources from `agent2` as well as allocation slack from `agent1`.
- Joseph Wu
On Jan. 19, 2016, 12:03 a.m., Guangya Liu wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40632/
> -----------------------------------------------------------
>
> (Updated Jan. 19, 2016, 12:03 a.m.)
>
>
> Review request for mesos, Ben Mahler, Artem Harutyunyan, Joris Van Remoortere, Joseph Wu, Klaus Ma, and Jian Qiu.
>
>
> Bugs: MESOS-4145
> https://issues.apache.org/jira/browse/MESOS-4145
>
>
> Repository: mesos
>
>
> Description
> -------
>
> Enabled oversubscribed resources for reservations in allocator.
> The allocator part including 5 patches:
> 1) https://reviews.apache.org/r/40632 Enabled oversubscribed resources for reservations in allocator
> 2) https://reviews.apache.org/r/41847 Updated allocation slack when slave was updated.
> 3) https://reviews.apache.org/r/41791 Updated allocation slack for dynamic reserve (1/3).
> 4) https://reviews.apache.org/r/42113 Handle unreserve logic for dynamic reservation (2/3).
> 5) https://reviews.apache.org/r/42194 Handle unreserve logic for dynamic reservation (3/3).
>
>
> Diffs
> -----
>
> src/master/allocator/mesos/hierarchical.cpp 48acde69b1a2f305b568a7e322a58708063dd30a
> src/tests/hierarchical_allocator_tests.cpp 9362dd306497ba01e0f387c3862456cdcac6f863
>
> Diff: https://reviews.apache.org/r/40632/diff/
>
>
> Testing
> -------
>
> make
> make check
> GLOG_v=2 ./bin/mesos-tests.sh --gtest_filter="HierarchicalAllocatorTest.*" --verbose
>
>
> Thanks,
>
> Guangya Liu
>
>
Re: Review Request 40632: Enabled oversubscribed resources for
reservations in allocator.
Posted by Klaus Ma <kl...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40632/#review116776
-----------------------------------------------------------
src/master/allocator/mesos/hierarchical.cpp (lines 1333 - 1335)
<https://reviews.apache.org/r/40632/#comment177869>
The result is un-stable here: if framework accept reserved resource come firstly, the others can not get those revocable resources; but if tenant framework come fristly, it will get revocable resources. It all depedent on sorter, and the offer maybe wrong in about cases.
- Klaus Ma
On Jan. 23, 2016, 11:18 a.m., Guangya Liu wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40632/
> -----------------------------------------------------------
>
> (Updated Jan. 23, 2016, 11:18 a.m.)
>
>
> Review request for mesos, Ben Mahler, Artem Harutyunyan, Joris Van Remoortere, Joseph Wu, Klaus Ma, and Jian Qiu.
>
>
> Bugs: MESOS-4145
> https://issues.apache.org/jira/browse/MESOS-4145
>
>
> Repository: mesos
>
>
> Description
> -------
>
> Enabled oversubscribed resources for reservations in allocator.
>
>
> Diffs
> -----
>
> src/master/allocator/mesos/hierarchical.cpp 65c7e6b15c5308c0910667e1b12f39b21293a316
> src/tests/hierarchical_allocator_tests.cpp b1cb955b7eb1213c7ba4a9c5181545bb49154f06
>
> Diff: https://reviews.apache.org/r/40632/diff/
>
>
> Testing
> -------
>
> make
> make check
> GLOG_v=2 ./bin/mesos-tests.sh --gtest_filter="HierarchicalAllocatorTest.*" --verbose --gtest_repeat=100 --gtest_shuffle
>
>
> Thanks,
>
> Guangya Liu
>
>
Re: Review Request 40632: Enabled oversubscribed resources for
reservations in allocator.
Posted by Guangya Liu <gy...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40632/
-----------------------------------------------------------
(Updated 三月 14, 2016, 4:03 a.m.)
Review request for mesos, Ben Mahler, Artem Harutyunyan, Joris Van Remoortere, Joseph Wu, Klaus Ma, and Jian Qiu.
Bugs: MESOS-4145
https://issues.apache.org/jira/browse/MESOS-4145
Repository: mesos
Description
-------
Enabled oversubscribed resources for reservations in allocator.
Diffs
-----
src/master/allocator/mesos/hierarchical.cpp 70291075c00a9a557529c2562dedcfc6c6c3ec32
src/tests/hierarchical_allocator_tests.cpp 459e02576f6d05abbbcc83ae5cabac5c66e93f05
Diff: https://reviews.apache.org/r/40632/diff/
Testing
-------
make
make check
GLOG_v=2 ./bin/mesos-tests.sh --gtest_filter="HierarchicalAllocatorTest.*" --verbose --gtest_repeat=100 --gtest_shuffle
Thanks,
Guangya Liu
Re: Review Request 40632: Enabled oversubscribed resources for
reservations in allocator.
Posted by Guangya Liu <gy...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40632/
-----------------------------------------------------------
(Updated 三月 11, 2016, 9:10 a.m.)
Review request for mesos, Ben Mahler, Artem Harutyunyan, Joris Van Remoortere, Joseph Wu, Klaus Ma, and Jian Qiu.
Bugs: MESOS-4145
https://issues.apache.org/jira/browse/MESOS-4145
Repository: mesos
Description
-------
Enabled oversubscribed resources for reservations in allocator.
Diffs (updated)
-----
src/master/allocator/mesos/hierarchical.cpp 70291075c00a9a557529c2562dedcfc6c6c3ec32
src/tests/hierarchical_allocator_tests.cpp 459e02576f6d05abbbcc83ae5cabac5c66e93f05
Diff: https://reviews.apache.org/r/40632/diff/
Testing
-------
make
make check
GLOG_v=2 ./bin/mesos-tests.sh --gtest_filter="HierarchicalAllocatorTest.*" --verbose --gtest_repeat=100 --gtest_shuffle
Thanks,
Guangya Liu
Re: Review Request 40632: Enabled oversubscribed resources for
reservations in allocator.
Posted by Guangya Liu <gy...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40632/
-----------------------------------------------------------
(Updated 二月 29, 2016, 7:04 a.m.)
Review request for mesos, Ben Mahler, Artem Harutyunyan, Joris Van Remoortere, Joseph Wu, Klaus Ma, and Jian Qiu.
Changes
-------
Remove dependency to make some patches get merged first.
Bugs: MESOS-4145
https://issues.apache.org/jira/browse/MESOS-4145
Repository: mesos
Description
-------
Enabled oversubscribed resources for reservations in allocator.
Diffs
-----
src/master/allocator/mesos/hierarchical.cpp 65c7e6b15c5308c0910667e1b12f39b21293a316
src/tests/hierarchical_allocator_tests.cpp b1cb955b7eb1213c7ba4a9c5181545bb49154f06
Diff: https://reviews.apache.org/r/40632/diff/
Testing
-------
make
make check
GLOG_v=2 ./bin/mesos-tests.sh --gtest_filter="HierarchicalAllocatorTest.*" --verbose --gtest_repeat=100 --gtest_shuffle
Thanks,
Guangya Liu
Re: Review Request 40632: Enabled oversubscribed resources for
reservations in allocator.
Posted by Guangya Liu <gy...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40632/
-----------------------------------------------------------
(Updated 一月 23, 2016, 3:18 a.m.)
Review request for mesos, Ben Mahler, Artem Harutyunyan, Joris Van Remoortere, Joseph Wu, Klaus Ma, and Jian Qiu.
Bugs: MESOS-4145
https://issues.apache.org/jira/browse/MESOS-4145
Repository: mesos
Description (updated)
-------
Enabled oversubscribed resources for reservations in allocator.
Diffs (updated)
-----
src/master/allocator/mesos/hierarchical.cpp 65c7e6b15c5308c0910667e1b12f39b21293a316
src/tests/hierarchical_allocator_tests.cpp b1cb955b7eb1213c7ba4a9c5181545bb49154f06
Diff: https://reviews.apache.org/r/40632/diff/
Testing
-------
make
make check
GLOG_v=2 ./bin/mesos-tests.sh --gtest_filter="HierarchicalAllocatorTest.*" --verbose --gtest_repeat=100 --gtest_shuffle
Thanks,
Guangya Liu
Re: Review Request 40632: Enabled oversubscribed resources for
reservations in allocator.
Posted by Guangya Liu <gy...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40632/
-----------------------------------------------------------
(Updated 一月 20, 2016, 6:25 a.m.)
Review request for mesos, Ben Mahler, Artem Harutyunyan, Joris Van Remoortere, Joseph Wu, Klaus Ma, and Jian Qiu.
Bugs: MESOS-4145
https://issues.apache.org/jira/browse/MESOS-4145
Repository: mesos
Description
-------
Enabled oversubscribed resources for reservations in allocator.
The allocator part including 5 patches:
1) https://reviews.apache.org/r/40632 Enabled oversubscribed resources for reservations in allocator
2) https://reviews.apache.org/r/41847 Updated allocation slack when slave was updated.
3) https://reviews.apache.org/r/41791 Updated allocation slack for dynamic reserve (1/3).
4) https://reviews.apache.org/r/42113 Handle unreserve logic for dynamic reservation (2/3).
5) https://reviews.apache.org/r/42194 Handle unreserve logic for dynamic reservation (3/3).
Diffs (updated)
-----
src/master/allocator/mesos/hierarchical.cpp e32ee4aa3ed9793bb5a99233e699e5cc2bdd796b
src/tests/hierarchical_allocator_tests.cpp 953712149bd951789beb29c72779c4ac65aa48dc
Diff: https://reviews.apache.org/r/40632/diff/
Testing (updated)
-------
make
make check
GLOG_v=2 ./bin/mesos-tests.sh --gtest_filter="HierarchicalAllocatorTest.*" --verbose --gtest_repeat=100 --gtest_shuffle
Thanks,
Guangya Liu
Re: Review Request 40632: Enabled oversubscribed resources for
reservations in allocator.
Posted by Guangya Liu <gy...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40632/
-----------------------------------------------------------
(Updated 一月 19, 2016, 8:03 a.m.)
Review request for mesos, Ben Mahler, Artem Harutyunyan, Joris Van Remoortere, Joseph Wu, Klaus Ma, and Jian Qiu.
Bugs: MESOS-4145
https://issues.apache.org/jira/browse/MESOS-4145
Repository: mesos
Description
-------
Enabled oversubscribed resources for reservations in allocator.
The allocator part including 5 patches:
1) https://reviews.apache.org/r/40632 Enabled oversubscribed resources for reservations in allocator
2) https://reviews.apache.org/r/41847 Updated allocation slack when slave was updated.
3) https://reviews.apache.org/r/41791 Updated allocation slack for dynamic reserve (1/3).
4) https://reviews.apache.org/r/42113 Handle unreserve logic for dynamic reservation (2/3).
5) https://reviews.apache.org/r/42194 Handle unreserve logic for dynamic reservation (3/3).
Diffs (updated)
-----
src/master/allocator/mesos/hierarchical.cpp 48acde69b1a2f305b568a7e322a58708063dd30a
src/tests/hierarchical_allocator_tests.cpp 9362dd306497ba01e0f387c3862456cdcac6f863
Diff: https://reviews.apache.org/r/40632/diff/
Testing
-------
make
make check
GLOG_v=2 ./bin/mesos-tests.sh --gtest_filter="HierarchicalAllocatorTest.*" --verbose
Thanks,
Guangya Liu
Re: Review Request 40632: Enabled oversubscribed resources for
reservations in allocator.
Posted by Guangya Liu <gy...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40632/
-----------------------------------------------------------
(Updated 一月 16, 2016, 7:51 a.m.)
Review request for mesos, Ben Mahler, Artem Harutyunyan, Joris Van Remoortere, Joseph Wu, Klaus Ma, and Jian Qiu.
Bugs: MESOS-4145
https://issues.apache.org/jira/browse/MESOS-4145
Repository: mesos
Description
-------
Enabled oversubscribed resources for reservations in allocator.
The allocator part including 5 patches:
1) https://reviews.apache.org/r/40632 Enabled oversubscribed resources for reservations in allocator
2) https://reviews.apache.org/r/41847 Updated allocation slack when slave was updated.
3) https://reviews.apache.org/r/41791 Updated allocation slack for dynamic reserve (1/3).
4) https://reviews.apache.org/r/42113 Handle unreserve logic for dynamic reservation (2/3).
5) https://reviews.apache.org/r/42194 Handle unreserve logic for dynamic reservation (3/3).
Diffs (updated)
-----
src/master/allocator/mesos/hierarchical.cpp 72e69a0f42dd724713f2a7a75f1b92ef16eb5569
src/tests/hierarchical_allocator_tests.cpp 9362dd306497ba01e0f387c3862456cdcac6f863
Diff: https://reviews.apache.org/r/40632/diff/
Testing
-------
make
make check
GLOG_v=2 ./bin/mesos-tests.sh --gtest_filter="HierarchicalAllocatorTest.*" --verbose
Thanks,
Guangya Liu
Re: Review Request 40632: Enabled oversubscribed resources for
reservations in allocator.
Posted by Guangya Liu <gy...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40632/
-----------------------------------------------------------
(Updated 一月 16, 2016, 7:38 a.m.)
Review request for mesos, Ben Mahler, Artem Harutyunyan, Joris Van Remoortere, Joseph Wu, Klaus Ma, and Jian Qiu.
Bugs: MESOS-4145
https://issues.apache.org/jira/browse/MESOS-4145
Repository: mesos
Description (updated)
-------
Enabled oversubscribed resources for reservations in allocator.
The allocator part including 5 patches:
1) https://reviews.apache.org/r/40632 Enabled oversubscribed resources for reservations in allocator
2) https://reviews.apache.org/r/41847 Updated allocation slack when slave was updated.
3) https://reviews.apache.org/r/41791 Updated allocation slack for dynamic reserve (1/3).
4) https://reviews.apache.org/r/42113 Handle unreserve logic for dynamic reservation (2/3).
5) https://reviews.apache.org/r/42194 Handle unreserve logic for dynamic reservation (3/3).
Diffs (updated)
-----
src/master/allocator/mesos/hierarchical.cpp 72e69a0f42dd724713f2a7a75f1b92ef16eb5569
src/tests/hierarchical_allocator_tests.cpp 9362dd306497ba01e0f387c3862456cdcac6f863
Diff: https://reviews.apache.org/r/40632/diff/
Testing
-------
make
make check
GLOG_v=2 ./bin/mesos-tests.sh --gtest_filter="HierarchicalAllocatorTest.*" --verbose
Thanks,
Guangya Liu
Re: Review Request 40632: Enabled oversubscribed resources for
reservations in allocator.
Posted by Guangya Liu <gy...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40632/
-----------------------------------------------------------
(Updated 一月 13, 2016, 12:39 p.m.)
Review request for mesos, Ben Mahler, Artem Harutyunyan, Joris Van Remoortere, Joseph Wu, Klaus Ma, and Jian Qiu.
Bugs: MESOS-4145
https://issues.apache.org/jira/browse/MESOS-4145
Repository: mesos
Description (updated)
-------
Enabled oversubscribed resources for reservations in allocator.
There are five patches handling the allocator part:
1) This patch handles send offer, add slave and quota integration test.
2) https://reviews.apache.org/r/41847/ Handles the case of update slave.
3) https://reviews.apache.org/r/41791/ Handles the case of dynamic reservations (1/3).
4) https://reviews.apache.org/r/42113/ Handle unreserve logic for dynamic reservation (2/3)
5) https://reviews.apache.org/r/42194/ Handle unreserve logic for dynamic reservation (3/3)
Diffs (updated)
-----
src/master/allocator/mesos/hierarchical.cpp d541bfa3f4190865c65d35c9d1ffdb8a3f194056
src/tests/hierarchical_allocator_tests.cpp e044f832c2c16e53e663c6ced5452649bb0dcb59
Diff: https://reviews.apache.org/r/40632/diff/
Testing
-------
make
make check
GLOG_v=2 ./bin/mesos-tests.sh --gtest_filter="HierarchicalAllocatorTest.*" --verbose
Thanks,
Guangya Liu
Re: Review Request 40632: Enabled oversubscribed resources for
reservations in allocator.
Posted by Guangya Liu <gy...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40632/
-----------------------------------------------------------
(Updated 一月 13, 2016, 2:24 a.m.)
Review request for mesos, Ben Mahler, Artem Harutyunyan, Joris Van Remoortere, Joseph Wu, Klaus Ma, and Jian Qiu.
Bugs: MESOS-4145
https://issues.apache.org/jira/browse/MESOS-4145
Repository: mesos
Description (updated)
-------
Enabled oversubscribed resources for reservations in allocator.
There are three patches handling the allocator part:
1) This patch handles send offer, add slave and quota integration test.
2) https://reviews.apache.org/r/41847/ Handles the case of update slave.
3) https://reviews.apache.org/r/41791/ Handles the case of dynamic reservations.
4?https://reviews.apache.org/r/42113/ Handle unreserve logic for dynamic reservation (1/2)
5?https://reviews.apache.org/r/42194/ Handle unreserve logic for dynamic reservation (1/2)
Diffs (updated)
-----
src/master/allocator/mesos/hierarchical.cpp df8bccaf2b8cfc0cb5ca18d4867371ae7a84c12f
src/tests/hierarchical_allocator_tests.cpp e044f832c2c16e53e663c6ced5452649bb0dcb59
Diff: https://reviews.apache.org/r/40632/diff/
Testing (updated)
-------
make
make check
GLOG_v=2 ./bin/mesos-tests.sh --gtest_filter="HierarchicalAllocatorTest.*" --verbose
Thanks,
Guangya Liu
Re: Review Request 40632: Enabled oversubscribed resources for
reservations in allocator.
Posted by Guangya Liu <gy...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40632/
-----------------------------------------------------------
(Updated 一月 8, 2016, 6:55 a.m.)
Review request for mesos, Ben Mahler, Artem Harutyunyan, Joris Van Remoortere, Joseph Wu, Klaus Ma, and Jian Qiu.
Bugs: MESOS-4145
https://issues.apache.org/jira/browse/MESOS-4145
Repository: mesos
Description
-------
Enabled oversubscribed resources for reservations in allocator.
There are three patches handling the allocator part:
1) This patch handles send offer, add slave and quota integration test.
2) https://reviews.apache.org/r/41847/ Handles the case of update slave.
3) https://reviews.apache.org/r/41791/ Handles the case of dynamic reservations.
Diffs (updated)
-----
src/master/allocator/mesos/hierarchical.cpp df8bccaf2b8cfc0cb5ca18d4867371ae7a84c12f
src/tests/hierarchical_allocator_tests.cpp e044f832c2c16e53e663c6ced5452649bb0dcb59
Diff: https://reviews.apache.org/r/40632/diff/
Testing
-------
make
make check
Thanks,
Guangya Liu
Re: Review Request 40632: Enabled oversubscribed resources for
reservations in allocator.
Posted by Guangya Liu <gy...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40632/
-----------------------------------------------------------
(Updated 一月 8, 2016, 2:34 a.m.)
Review request for mesos, Ben Mahler, Artem Harutyunyan, Joris Van Remoortere, Joseph Wu, Klaus Ma, and Jian Qiu.
Bugs: MESOS-4145
https://issues.apache.org/jira/browse/MESOS-4145
Repository: mesos
Description (updated)
-------
Enabled oversubscribed resources for reservations in allocator.
There are three patches handling the allocator part:
1) This patch handles send offer, add slave and quota integration test.
2) https://reviews.apache.org/r/41847/ Handles the case of update slave.
3) https://reviews.apache.org/r/41791/ Handles the case of dynamic reservations.
Diffs (updated)
-----
src/master/allocator/mesos/hierarchical.cpp df8bccaf2b8cfc0cb5ca18d4867371ae7a84c12f
src/tests/hierarchical_allocator_tests.cpp e044f832c2c16e53e663c6ced5452649bb0dcb59
Diff: https://reviews.apache.org/r/40632/diff/
Testing
-------
make
make check
Thanks,
Guangya Liu
Re: Review Request 40632: Enabled oversubscribed resources for
reservations in allocator.
Posted by Guangya Liu <gy...@gmail.com>.
> On 一月 7, 2016, 11:51 a.m., Klaus Ma wrote:
> > src/master/allocator/mesos/hierarchical.cpp, line 1379
> > <https://reviews.apache.org/r/40632/diff/11/?file=1186198#file1186198line1379>
> >
> > Add `allocation slack resource` if any
`resources.unreserved()` already include allocation slack.
- Guangya
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40632/#review113240
-----------------------------------------------------------
On 一月 7, 2016, 11:27 a.m., Guangya Liu wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40632/
> -----------------------------------------------------------
>
> (Updated 一月 7, 2016, 11:27 a.m.)
>
>
> Review request for mesos, Ben Mahler, Artem Harutyunyan, Joris Van Remoortere, Joseph Wu, Klaus Ma, and Jian Qiu.
>
>
> Bugs: MESOS-4145
> https://issues.apache.org/jira/browse/MESOS-4145
>
>
> Repository: mesos
>
>
> Description
> -------
>
> Enabled oversubscribed resources for reservations in allocator.
>
>
> Diffs
> -----
>
> src/master/allocator/mesos/hierarchical.cpp df8bccaf2b8cfc0cb5ca18d4867371ae7a84c12f
> src/tests/hierarchical_allocator_tests.cpp e044f832c2c16e53e663c6ced5452649bb0dcb59
>
> Diff: https://reviews.apache.org/r/40632/diff/
>
>
> Testing
> -------
>
> make
> make check
>
>
> Thanks,
>
> Guangya Liu
>
>
Re: Review Request 40632: Enabled oversubscribed resources for
reservations in allocator.
Posted by Klaus Ma <kl...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40632/#review113240
-----------------------------------------------------------
src/master/allocator/mesos/hierarchical.cpp (line 1379)
<https://reviews.apache.org/r/40632/#comment173738>
Add `allocation slack resource` if any
- Klaus Ma
On Jan. 7, 2016, 7:27 p.m., Guangya Liu wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40632/
> -----------------------------------------------------------
>
> (Updated Jan. 7, 2016, 7:27 p.m.)
>
>
> Review request for mesos, Ben Mahler, Artem Harutyunyan, Joris Van Remoortere, Joseph Wu, Klaus Ma, and Jian Qiu.
>
>
> Bugs: MESOS-4145
> https://issues.apache.org/jira/browse/MESOS-4145
>
>
> Repository: mesos
>
>
> Description
> -------
>
> Enabled oversubscribed resources for reservations in allocator.
>
>
> Diffs
> -----
>
> src/master/allocator/mesos/hierarchical.cpp df8bccaf2b8cfc0cb5ca18d4867371ae7a84c12f
> src/tests/hierarchical_allocator_tests.cpp e044f832c2c16e53e663c6ced5452649bb0dcb59
>
> Diff: https://reviews.apache.org/r/40632/diff/
>
>
> Testing
> -------
>
> make
> make check
>
>
> Thanks,
>
> Guangya Liu
>
>
Re: Review Request 40632: Enabled oversubscribed resources for
reservations in allocator.
Posted by Guangya Liu <gy...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40632/
-----------------------------------------------------------
(Updated 一月 7, 2016, 11:27 a.m.)
Review request for mesos, Ben Mahler, Artem Harutyunyan, Joris Van Remoortere, Joseph Wu, Klaus Ma, and Jian Qiu.
Bugs: MESOS-4145
https://issues.apache.org/jira/browse/MESOS-4145
Repository: mesos
Description
-------
Enabled oversubscribed resources for reservations in allocator.
Diffs (updated)
-----
src/master/allocator/mesos/hierarchical.cpp df8bccaf2b8cfc0cb5ca18d4867371ae7a84c12f
src/tests/hierarchical_allocator_tests.cpp e044f832c2c16e53e663c6ced5452649bb0dcb59
Diff: https://reviews.apache.org/r/40632/diff/
Testing
-------
make
make check
Thanks,
Guangya Liu
Re: Review Request 40632: Enabled oversubscribed resources for
reservations in allocator.
Posted by Guangya Liu <gy...@gmail.com>.
> On 一月 7, 2016, 6:55 a.m., Klaus Ma wrote:
> > src/master/allocator/mesos/hierarchical.cpp, line 1350
> > <https://reviews.apache.org/r/40632/diff/10/?file=1180362#file1180362line1350>
> >
> > It does not include `ALLOCATIONS_SLACK` in `remainingClusterResources`; so if account `remainingAllocationSlack` in `allocatedForWDRF`, the check of `remainingClusterResources.contains(allocatedForWDRF + resources)` will be always false.
The `remainingClusterResources` DOES include all revocable resoruces including both usgae slack and allocation slack. Please refer to https://github.com/apache/mesos/blob/master/src/master/allocator/mesos/hierarchical.cpp#L1220-L1224
- Guangya
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40632/#review113193
-----------------------------------------------------------
On 一月 3, 2016, 9:21 a.m., Guangya Liu wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40632/
> -----------------------------------------------------------
>
> (Updated 一月 3, 2016, 9:21 a.m.)
>
>
> Review request for mesos, Ben Mahler, Artem Harutyunyan, Joris Van Remoortere, Joseph Wu, Klaus Ma, and Jian Qiu.
>
>
> Bugs: MESOS-4145
> https://issues.apache.org/jira/browse/MESOS-4145
>
>
> Repository: mesos
>
>
> Description
> -------
>
> Enabled oversubscribed resources for reservations in allocator.
>
>
> Diffs
> -----
>
> src/master/allocator/mesos/hierarchical.cpp 7f900c4e024485704d79e57ae22407557598fe6c
> src/tests/hierarchical_allocator_tests.cpp e044f832c2c16e53e663c6ced5452649bb0dcb59
>
> Diff: https://reviews.apache.org/r/40632/diff/
>
>
> Testing
> -------
>
> make
> make check
>
>
> Thanks,
>
> Guangya Liu
>
>
Re: Review Request 40632: Enabled oversubscribed resources for
reservations in allocator.
Posted by Klaus Ma <kl...@gmail.com>.
> On Jan. 7, 2016, 2:55 p.m., Klaus Ma wrote:
> > src/master/allocator/mesos/hierarchical.cpp, line 1350
> > <https://reviews.apache.org/r/40632/diff/10/?file=1180362#file1180362line1350>
> >
> > It does not include `ALLOCATIONS_SLACK` in `remainingClusterResources`; so if account `remainingAllocationSlack` in `allocatedForWDRF`, the check of `remainingClusterResources.contains(allocatedForWDRF + resources)` will be always false.
>
> Guangya Liu wrote:
> The `remainingClusterResources` DOES include all revocable resoruces including both usgae slack and allocation slack. Please refer to https://github.com/apache/mesos/blob/master/src/master/allocator/mesos/hierarchical.cpp#L1220-L1224
Yes, that's OK :)
- Klaus
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40632/#review113193
-----------------------------------------------------------
On Jan. 7, 2016, 7:27 p.m., Guangya Liu wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40632/
> -----------------------------------------------------------
>
> (Updated Jan. 7, 2016, 7:27 p.m.)
>
>
> Review request for mesos, Ben Mahler, Artem Harutyunyan, Joris Van Remoortere, Joseph Wu, Klaus Ma, and Jian Qiu.
>
>
> Bugs: MESOS-4145
> https://issues.apache.org/jira/browse/MESOS-4145
>
>
> Repository: mesos
>
>
> Description
> -------
>
> Enabled oversubscribed resources for reservations in allocator.
>
>
> Diffs
> -----
>
> src/master/allocator/mesos/hierarchical.cpp df8bccaf2b8cfc0cb5ca18d4867371ae7a84c12f
> src/tests/hierarchical_allocator_tests.cpp e044f832c2c16e53e663c6ced5452649bb0dcb59
>
> Diff: https://reviews.apache.org/r/40632/diff/
>
>
> Testing
> -------
>
> make
> make check
>
>
> Thanks,
>
> Guangya Liu
>
>
Re: Review Request 40632: Enabled oversubscribed resources for
reservations in allocator.
Posted by Klaus Ma <kl...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40632/#review113193
-----------------------------------------------------------
src/master/allocator/mesos/hierarchical.cpp (line 1350)
<https://reviews.apache.org/r/40632/#comment173699>
It does not include `ALLOCATIONS_SLACK` in `remainingClusterResources`; so if account `remainingAllocationSlack` in `allocatedForWDRF`, the check of `remainingClusterResources.contains(allocatedForWDRF + resources)` will be always false.
src/tests/hierarchical_allocator_tests.cpp (line 210)
<https://reviews.apache.org/r/40632/#comment173681>
add blank line.
- Klaus Ma
On Jan. 3, 2016, 5:21 p.m., Guangya Liu wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40632/
> -----------------------------------------------------------
>
> (Updated Jan. 3, 2016, 5:21 p.m.)
>
>
> Review request for mesos, Ben Mahler, Artem Harutyunyan, Joris Van Remoortere, Joseph Wu, Klaus Ma, and Jian Qiu.
>
>
> Bugs: MESOS-4145
> https://issues.apache.org/jira/browse/MESOS-4145
>
>
> Repository: mesos
>
>
> Description
> -------
>
> Enabled oversubscribed resources for reservations in allocator.
>
>
> Diffs
> -----
>
> src/master/allocator/mesos/hierarchical.cpp 7f900c4e024485704d79e57ae22407557598fe6c
> src/tests/hierarchical_allocator_tests.cpp e044f832c2c16e53e663c6ced5452649bb0dcb59
>
> Diff: https://reviews.apache.org/r/40632/diff/
>
>
> Testing
> -------
>
> make
> make check
>
>
> Thanks,
>
> Guangya Liu
>
>
Re: Review Request 40632: Enabled oversubscribed resources for
reservations in allocator.
Posted by Guangya Liu <gy...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40632/
-----------------------------------------------------------
(Updated 一月 3, 2016, 9:21 a.m.)
Review request for mesos, Ben Mahler, Artem Harutyunyan, Joris Van Remoortere, Joseph Wu, Klaus Ma, and Jian Qiu.
Summary (updated)
-----------------
Enabled oversubscribed resources for reservations in allocator.
Bugs: MESOS-4145
https://issues.apache.org/jira/browse/MESOS-4145
Repository: mesos
Description
-------
Enabled oversubscribed resources for reservations in allocator.
Diffs
-----
src/master/allocator/mesos/hierarchical.cpp 7f900c4e024485704d79e57ae22407557598fe6c
src/tests/hierarchical_allocator_tests.cpp e044f832c2c16e53e663c6ced5452649bb0dcb59
Diff: https://reviews.apache.org/r/40632/diff/
Testing
-------
make
make check
Thanks,
Guangya Liu
Re: Review Request 40632: WIP: Enabled oversubscribed resources for
reservations in allocator.
Posted by Guangya Liu <gy...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40632/
-----------------------------------------------------------
(Updated 一月 3, 2016, 3:03 a.m.)
Review request for mesos, Klaus Ma and Jian Qiu.
Repository: mesos
Description
-------
Enabled oversubscribed resources for reservations in allocator.
Diffs (updated)
-----
src/master/allocator/mesos/hierarchical.cpp 7f900c4e024485704d79e57ae22407557598fe6c
src/tests/hierarchical_allocator_tests.cpp e044f832c2c16e53e663c6ced5452649bb0dcb59
Diff: https://reviews.apache.org/r/40632/diff/
Testing
-------
make
make check
Thanks,
Guangya Liu
Re: Review Request 40632: WIP: Enabled oversubscribed resources for
reservations in allocator.
Posted by Guangya Liu <gy...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40632/
-----------------------------------------------------------
(Updated 一月 2, 2016, 1:18 a.m.)
Review request for mesos, Klaus Ma and Jian Qiu.
Repository: mesos
Description
-------
Enabled oversubscribed resources for reservations in allocator.
Diffs (updated)
-----
src/master/allocator/mesos/hierarchical.cpp 7f900c4e024485704d79e57ae22407557598fe6c
Diff: https://reviews.apache.org/r/40632/diff/
Testing
-------
make
make check
Thanks,
Guangya Liu
Re: Review Request 40632: Enabled oversubscribed resources for
reservations in allocator.
Posted by Guangya Liu <gy...@gmail.com>.
> On 十二月 30, 2015, 12:42 p.m., Klaus Ma wrote:
> > src/master/allocator/mesos/hierarchical.cpp, line 1321
> > <https://reviews.apache.org/r/40632/diff/8/?file=1178313#file1178313line1321>
> >
> > As framework can not use ALLOCATION_SLACK mixed with other resources, allocator need to check seperatly.
>
> Guangya Liu wrote:
> Yes, currently the validation logic in master is https://github.com/apache/mesos/blob/master/src/master/validation.cpp#L479-L488 which can make sure a task and its executor can either use non-revocable resources or revocable resources of a given name but not both. After we introduced the allocation slack, we may want to guard that a task and its executor can either use allocation slack or usage slack or non-revocable resources but not all. I will upload a patch to handle this part.
>
> Guangya Liu wrote:
> Fixed in https://reviews.apache.org/r/41848/
>
> Joseph Wu wrote:
> Note: mixing resources is allowed. Everything in a container is evicted at once, including non-revocable resources mixed into a revocable one.
>
> Klaus Ma wrote:
> re 'Note: mixing resources is allowed.', my understanding is that: different tasks can use different resources, but one task can NOT use mixing resources; for example, one task can not use both `USAGE_SLACK` and `ALLOCATION_SLACK`.
It is an old issue, this problem also exist with usage slack. If the usage slack plus non revocable resources, the offer will also be send. Will enhance `allocatable` to update it.
> On 十二月 30, 2015, 12:42 p.m., Klaus Ma wrote:
> > src/master/allocator/mesos/hierarchical.cpp, line 1302
> > <https://reviews.apache.org/r/40632/diff/8/?file=1178313#file1178313line1302>
> >
> > If framework has reserved.statless resources, allocator should not dispatch it as ALLOCATION_SLACK.
>
> Klaus Ma wrote:
> @Joseph, I can not remenber where did you append your comments about this topic; but it seems you're OK to offer both `stateless.reserved` & `ALLOCATION_SLACK` of the same role, right? But how framework developer distinguish `stateless.reserved` & `ALLOCATION_SLACK` of its? If it launch task in `ALLOCATION_SLACK` and `stateless.reserved`, the task in `ALLOCATION_SLACK` was terminated immediately; it just increases overhead of cluster instead of resource utilization.
>
> Joseph Wu wrote:
> There's something about this in the design doc.
>
> The summary is: Meta-frameworks may benefit from receiving allocation slack from itself.
>
> Klaus Ma wrote:
> OK, would you help to update the design doc? There's comments that:
>
> NOTE: We need to be careful not to offer the same resources (as reserved resources and as revocable resources) to the same framework. This could presumably happen if a Lender Framework also accepts revocable resources.
>
> Guangya Liu wrote:
> I've already handled this case here: When sending offers, the allocation slack resources in the offer need to exclude all allocated stateless reserved resources and allocated allocation slack resources.
>
> Klaus Ma wrote:
> I think current code did not handle this case; if a Lender framekwork accepts revocable resources, the framework's `stateless.reserved` resource & allocation slack from that `stateless.reserved` will send to that famework; so "offer the same resources (as reserved resources and as revocable resources) to the same framework".
Filed a bug https://issues.apache.org/jira/browse/MESOS-4426
- Guangya
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40632/#review112309
-----------------------------------------------------------
On 一月 16, 2016, 7:51 a.m., Guangya Liu wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40632/
> -----------------------------------------------------------
>
> (Updated 一月 16, 2016, 7:51 a.m.)
>
>
> Review request for mesos, Ben Mahler, Artem Harutyunyan, Joris Van Remoortere, Joseph Wu, Klaus Ma, and Jian Qiu.
>
>
> Bugs: MESOS-4145
> https://issues.apache.org/jira/browse/MESOS-4145
>
>
> Repository: mesos
>
>
> Description
> -------
>
> Enabled oversubscribed resources for reservations in allocator.
> The allocator part including 5 patches:
> 1) https://reviews.apache.org/r/40632 Enabled oversubscribed resources for reservations in allocator
> 2) https://reviews.apache.org/r/41847 Updated allocation slack when slave was updated.
> 3) https://reviews.apache.org/r/41791 Updated allocation slack for dynamic reserve (1/3).
> 4) https://reviews.apache.org/r/42113 Handle unreserve logic for dynamic reservation (2/3).
> 5) https://reviews.apache.org/r/42194 Handle unreserve logic for dynamic reservation (3/3).
>
>
> Diffs
> -----
>
> src/master/allocator/mesos/hierarchical.cpp 72e69a0f42dd724713f2a7a75f1b92ef16eb5569
> src/tests/hierarchical_allocator_tests.cpp 9362dd306497ba01e0f387c3862456cdcac6f863
>
> Diff: https://reviews.apache.org/r/40632/diff/
>
>
> Testing
> -------
>
> make
> make check
> GLOG_v=2 ./bin/mesos-tests.sh --gtest_filter="HierarchicalAllocatorTest.*" --verbose
>
>
> Thanks,
>
> Guangya Liu
>
>
Re: Review Request 40632: Enabled oversubscribed resources for
reservations in allocator.
Posted by Joseph Wu <jo...@mesosphere.io>.
> On Dec. 30, 2015, 4:42 a.m., Klaus Ma wrote:
> > src/master/allocator/mesos/hierarchical.cpp, line 1334
> > <https://reviews.apache.org/r/40632/diff/8/?file=1178313#file1178313line1334>
> >
> > In this check, it seems allocator send offer when both !allocation & allocation are enough.
>
> Joseph Wu wrote:
> Looks like quota might have stepped on the toes of oversubscription. I'll double-check with Joris/AlexR.
>
> Guangya Liu wrote:
> Not quite clear about the problem, can you elaborate? Joseph?
>
> Klaus Ma wrote:
> @Joseph, the Quota only handle `unreserved.nonRevocable` resources; so `USAGE_SLACK` are not included in Quota.
Right. The quota code may prevent the allocation of USAGE_SLACK. When quota is met, USAGE_SLACK will not be allocated (if the agent has non-revocable available resources).
With your change, ALLOCATION_SLACK is allocated, even if quota is met. This isn't your problem originally, but we'd want to address both types of slack (or leave a TODO and address neither of them).
- Joseph
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40632/#review112309
-----------------------------------------------------------
On Jan. 7, 2016, 10:55 p.m., Guangya Liu wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40632/
> -----------------------------------------------------------
>
> (Updated Jan. 7, 2016, 10:55 p.m.)
>
>
> Review request for mesos, Ben Mahler, Artem Harutyunyan, Joris Van Remoortere, Joseph Wu, Klaus Ma, and Jian Qiu.
>
>
> Bugs: MESOS-4145
> https://issues.apache.org/jira/browse/MESOS-4145
>
>
> Repository: mesos
>
>
> Description
> -------
>
> Enabled oversubscribed resources for reservations in allocator.
>
> There are three patches handling the allocator part:
> 1) This patch handles send offer, add slave and quota integration test.
> 2) https://reviews.apache.org/r/41847/ Handles the case of update slave.
> 3) https://reviews.apache.org/r/41791/ Handles the case of dynamic reservations.
>
>
> Diffs
> -----
>
> src/master/allocator/mesos/hierarchical.cpp df8bccaf2b8cfc0cb5ca18d4867371ae7a84c12f
> src/tests/hierarchical_allocator_tests.cpp e044f832c2c16e53e663c6ced5452649bb0dcb59
>
> Diff: https://reviews.apache.org/r/40632/diff/
>
>
> Testing
> -------
>
> make
> make check
>
>
> Thanks,
>
> Guangya Liu
>
>
Re: Review Request 40632: Enabled oversubscribed resources for
reservations in allocator.
Posted by Joseph Wu <jo...@mesosphere.io>.
> On Dec. 30, 2015, 4:42 a.m., Klaus Ma wrote:
> > src/master/allocator/mesos/hierarchical.cpp, line 1302
> > <https://reviews.apache.org/r/40632/diff/8/?file=1178313#file1178313line1302>
> >
> > If framework has reserved.statless resources, allocator should not dispatch it as ALLOCATION_SLACK.
>
> Klaus Ma wrote:
> @Joseph, I can not remenber where did you append your comments about this topic; but it seems you're OK to offer both `stateless.reserved` & `ALLOCATION_SLACK` of the same role, right? But how framework developer distinguish `stateless.reserved` & `ALLOCATION_SLACK` of its? If it launch task in `ALLOCATION_SLACK` and `stateless.reserved`, the task in `ALLOCATION_SLACK` was terminated immediately; it just increases overhead of cluster instead of resource utilization.
There's something about this in the design doc.
The summary is: Meta-frameworks may benefit from receiving allocation slack from itself.
- Joseph
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40632/#review112309
-----------------------------------------------------------
On Jan. 7, 2016, 10:55 p.m., Guangya Liu wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40632/
> -----------------------------------------------------------
>
> (Updated Jan. 7, 2016, 10:55 p.m.)
>
>
> Review request for mesos, Ben Mahler, Artem Harutyunyan, Joris Van Remoortere, Joseph Wu, Klaus Ma, and Jian Qiu.
>
>
> Bugs: MESOS-4145
> https://issues.apache.org/jira/browse/MESOS-4145
>
>
> Repository: mesos
>
>
> Description
> -------
>
> Enabled oversubscribed resources for reservations in allocator.
>
> There are three patches handling the allocator part:
> 1) This patch handles send offer, add slave and quota integration test.
> 2) https://reviews.apache.org/r/41847/ Handles the case of update slave.
> 3) https://reviews.apache.org/r/41791/ Handles the case of dynamic reservations.
>
>
> Diffs
> -----
>
> src/master/allocator/mesos/hierarchical.cpp df8bccaf2b8cfc0cb5ca18d4867371ae7a84c12f
> src/tests/hierarchical_allocator_tests.cpp e044f832c2c16e53e663c6ced5452649bb0dcb59
>
> Diff: https://reviews.apache.org/r/40632/diff/
>
>
> Testing
> -------
>
> make
> make check
>
>
> Thanks,
>
> Guangya Liu
>
>
Re: Review Request 40632: Enabled oversubscribed resources for
reservations in allocator.
Posted by Klaus Ma <kl...@gmail.com>.
> On Dec. 30, 2015, 8:42 p.m., Klaus Ma wrote:
> > src/master/allocator/mesos/hierarchical.cpp, line 1302
> > <https://reviews.apache.org/r/40632/diff/8/?file=1178313#file1178313line1302>
> >
> > If framework has reserved.statless resources, allocator should not dispatch it as ALLOCATION_SLACK.
@Joseph, I can not remenber where did you append your comments about this topic; but it seems you're OK to offer both `stateless.reserved` & `ALLOCATION_SLACK` of the same role, right? But how framework developer distinguish `stateless.reserved` & `ALLOCATION_SLACK` of its? If it launch task in `ALLOCATION_SLACK` and `stateless.reserved`, the task in `ALLOCATION_SLACK` was terminated immediately; it just increases overhead of cluster instead of resource utilization.
- Klaus
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40632/#review112309
-----------------------------------------------------------
On Jan. 8, 2016, 2:55 p.m., Guangya Liu wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40632/
> -----------------------------------------------------------
>
> (Updated Jan. 8, 2016, 2:55 p.m.)
>
>
> Review request for mesos, Ben Mahler, Artem Harutyunyan, Joris Van Remoortere, Joseph Wu, Klaus Ma, and Jian Qiu.
>
>
> Bugs: MESOS-4145
> https://issues.apache.org/jira/browse/MESOS-4145
>
>
> Repository: mesos
>
>
> Description
> -------
>
> Enabled oversubscribed resources for reservations in allocator.
>
> There are three patches handling the allocator part:
> 1) This patch handles send offer, add slave and quota integration test.
> 2) https://reviews.apache.org/r/41847/ Handles the case of update slave.
> 3) https://reviews.apache.org/r/41791/ Handles the case of dynamic reservations.
>
>
> Diffs
> -----
>
> src/master/allocator/mesos/hierarchical.cpp df8bccaf2b8cfc0cb5ca18d4867371ae7a84c12f
> src/tests/hierarchical_allocator_tests.cpp e044f832c2c16e53e663c6ced5452649bb0dcb59
>
> Diff: https://reviews.apache.org/r/40632/diff/
>
>
> Testing
> -------
>
> make
> make check
>
>
> Thanks,
>
> Guangya Liu
>
>
Re: Review Request 40632: Enabled oversubscribed resources for
reservations in allocator.
Posted by Guangya Liu <gy...@gmail.com>.
> On 十二月 30, 2015, 12:42 p.m., Klaus Ma wrote:
> > src/master/allocator/mesos/hierarchical.cpp, line 1302
> > <https://reviews.apache.org/r/40632/diff/8/?file=1178313#file1178313line1302>
> >
> > If framework has reserved.statless resources, allocator should not dispatch it as ALLOCATION_SLACK.
>
> Klaus Ma wrote:
> @Joseph, I can not remenber where did you append your comments about this topic; but it seems you're OK to offer both `stateless.reserved` & `ALLOCATION_SLACK` of the same role, right? But how framework developer distinguish `stateless.reserved` & `ALLOCATION_SLACK` of its? If it launch task in `ALLOCATION_SLACK` and `stateless.reserved`, the task in `ALLOCATION_SLACK` was terminated immediately; it just increases overhead of cluster instead of resource utilization.
>
> Joseph Wu wrote:
> There's something about this in the design doc.
>
> The summary is: Meta-frameworks may benefit from receiving allocation slack from itself.
>
> Klaus Ma wrote:
> OK, would you help to update the design doc? There's comments that:
>
> NOTE: We need to be careful not to offer the same resources (as reserved resources and as revocable resources) to the same framework. This could presumably happen if a Lender Framework also accepts revocable resources.
I've already handled this case here: When sending offers, the allocation slack resources in the offer need to exclude all allocated stateless reserved resources and allocated allocation slack resources.
- Guangya
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40632/#review112309
-----------------------------------------------------------
On 一月 8, 2016, 6:55 a.m., Guangya Liu wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40632/
> -----------------------------------------------------------
>
> (Updated 一月 8, 2016, 6:55 a.m.)
>
>
> Review request for mesos, Ben Mahler, Artem Harutyunyan, Joris Van Remoortere, Joseph Wu, Klaus Ma, and Jian Qiu.
>
>
> Bugs: MESOS-4145
> https://issues.apache.org/jira/browse/MESOS-4145
>
>
> Repository: mesos
>
>
> Description
> -------
>
> Enabled oversubscribed resources for reservations in allocator.
>
> There are three patches handling the allocator part:
> 1) This patch handles send offer, add slave and quota integration test.
> 2) https://reviews.apache.org/r/41847/ Handles the case of update slave.
> 3) https://reviews.apache.org/r/41791/ Handles the case of dynamic reservations.
>
>
> Diffs
> -----
>
> src/master/allocator/mesos/hierarchical.cpp df8bccaf2b8cfc0cb5ca18d4867371ae7a84c12f
> src/tests/hierarchical_allocator_tests.cpp e044f832c2c16e53e663c6ced5452649bb0dcb59
>
> Diff: https://reviews.apache.org/r/40632/diff/
>
>
> Testing
> -------
>
> make
> make check
>
>
> Thanks,
>
> Guangya Liu
>
>
Re: Review Request 40632: Enabled oversubscribed resources for
reservations in allocator.
Posted by Guangya Liu <gy...@gmail.com>.
> On 十二月 30, 2015, 12:42 p.m., Klaus Ma wrote:
> > src/master/allocator/mesos/hierarchical.cpp, line 1321
> > <https://reviews.apache.org/r/40632/diff/8/?file=1178313#file1178313line1321>
> >
> > As framework can not use ALLOCATION_SLACK mixed with other resources, allocator need to check seperatly.
>
> Guangya Liu wrote:
> Yes, currently the validation logic in master is https://github.com/apache/mesos/blob/master/src/master/validation.cpp#L479-L488 which can make sure a task and its executor can either use non-revocable resources or revocable resources of a given name but not both. After we introduced the allocation slack, we may want to guard that a task and its executor can either use allocation slack or usage slack or non-revocable resources but not all. I will upload a patch to handle this part.
>
> Guangya Liu wrote:
> Fixed in https://reviews.apache.org/r/41848/
>
> Joseph Wu wrote:
> Note: mixing resources is allowed. Everything in a container is evicted at once, including non-revocable resources mixed into a revocable one.
>
> Klaus Ma wrote:
> re 'Note: mixing resources is allowed.', my understanding is that: different tasks can use different resources, but one task can NOT use mixing resources; for example, one task can not use both `USAGE_SLACK` and `ALLOCATION_SLACK`.
>
> Guangya Liu wrote:
> It is an old issue, this problem also exist with usage slack. If the usage slack plus non revocable resources, the offer will also be send. Will enhance `allocatable` to update it.
Fixed in https://issues.apache.org/jira/browse/MESOS-4426
- Guangya
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40632/#review112309
-----------------------------------------------------------
On 一月 19, 2016, 8:03 a.m., Guangya Liu wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40632/
> -----------------------------------------------------------
>
> (Updated 一月 19, 2016, 8:03 a.m.)
>
>
> Review request for mesos, Ben Mahler, Artem Harutyunyan, Joris Van Remoortere, Joseph Wu, Klaus Ma, and Jian Qiu.
>
>
> Bugs: MESOS-4145
> https://issues.apache.org/jira/browse/MESOS-4145
>
>
> Repository: mesos
>
>
> Description
> -------
>
> Enabled oversubscribed resources for reservations in allocator.
> The allocator part including 5 patches:
> 1) https://reviews.apache.org/r/40632 Enabled oversubscribed resources for reservations in allocator
> 2) https://reviews.apache.org/r/41847 Updated allocation slack when slave was updated.
> 3) https://reviews.apache.org/r/41791 Updated allocation slack for dynamic reserve (1/3).
> 4) https://reviews.apache.org/r/42113 Handle unreserve logic for dynamic reservation (2/3).
> 5) https://reviews.apache.org/r/42194 Handle unreserve logic for dynamic reservation (3/3).
>
>
> Diffs
> -----
>
> src/master/allocator/mesos/hierarchical.cpp 48acde69b1a2f305b568a7e322a58708063dd30a
> src/tests/hierarchical_allocator_tests.cpp 9362dd306497ba01e0f387c3862456cdcac6f863
>
> Diff: https://reviews.apache.org/r/40632/diff/
>
>
> Testing
> -------
>
> make
> make check
> GLOG_v=2 ./bin/mesos-tests.sh --gtest_filter="HierarchicalAllocatorTest.*" --verbose
>
>
> Thanks,
>
> Guangya Liu
>
>
Re: Review Request 40632: Enabled oversubscribed resources for
reservations in allocator.
Posted by Guangya Liu <gy...@gmail.com>.
> On 十二月 30, 2015, 12:42 p.m., Klaus Ma wrote:
> > src/master/allocator/mesos/hierarchical.cpp, line 1334
> > <https://reviews.apache.org/r/40632/diff/8/?file=1178313#file1178313line1334>
> >
> > In this check, it seems allocator send offer when both !allocation & allocation are enough.
>
> Joseph Wu wrote:
> Looks like quota might have stepped on the toes of oversubscription. I'll double-check with Joris/AlexR.
>
> Guangya Liu wrote:
> Not quite clear about the problem, can you elaborate? Joseph?
>
> Klaus Ma wrote:
> @Joseph, the Quota only handle `unreserved.nonRevocable` resources; so `USAGE_SLACK` are not included in Quota.
>
> Joseph Wu wrote:
> Right. The quota code may prevent the allocation of USAGE_SLACK. When quota is met, USAGE_SLACK will not be allocated (if the agent has non-revocable available resources).
>
> With your change, ALLOCATION_SLACK is allocated, even if quota is met. This isn't your problem originally, but we'd want to address both types of slack (or leave a TODO and address neither of them).
As non revocable resources are not counted for quota, so I think that we should allocate them in DRF stage without considering quota. The current patch is only handling allocation slack, I will add some TODO comments here to clarify that usage slack also needs to be considered here.
- Guangya
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40632/#review112309
-----------------------------------------------------------
On 一月 8, 2016, 6:55 a.m., Guangya Liu wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40632/
> -----------------------------------------------------------
>
> (Updated 一月 8, 2016, 6:55 a.m.)
>
>
> Review request for mesos, Ben Mahler, Artem Harutyunyan, Joris Van Remoortere, Joseph Wu, Klaus Ma, and Jian Qiu.
>
>
> Bugs: MESOS-4145
> https://issues.apache.org/jira/browse/MESOS-4145
>
>
> Repository: mesos
>
>
> Description
> -------
>
> Enabled oversubscribed resources for reservations in allocator.
>
> There are three patches handling the allocator part:
> 1) This patch handles send offer, add slave and quota integration test.
> 2) https://reviews.apache.org/r/41847/ Handles the case of update slave.
> 3) https://reviews.apache.org/r/41791/ Handles the case of dynamic reservations.
>
>
> Diffs
> -----
>
> src/master/allocator/mesos/hierarchical.cpp df8bccaf2b8cfc0cb5ca18d4867371ae7a84c12f
> src/tests/hierarchical_allocator_tests.cpp e044f832c2c16e53e663c6ced5452649bb0dcb59
>
> Diff: https://reviews.apache.org/r/40632/diff/
>
>
> Testing
> -------
>
> make
> make check
>
>
> Thanks,
>
> Guangya Liu
>
>
Re: Review Request 40632: Enabled oversubscribed resources for
reservations in allocator.
Posted by Guangya Liu <gy...@gmail.com>.
> On 十二月 30, 2015, 12:42 p.m., Klaus Ma wrote:
> > src/master/allocator/mesos/hierarchical.cpp, line 1334
> > <https://reviews.apache.org/r/40632/diff/8/?file=1178313#file1178313line1334>
> >
> > In this check, it seems allocator send offer when both !allocation & allocation are enough.
>
> Joseph Wu wrote:
> Looks like quota might have stepped on the toes of oversubscription. I'll double-check with Joris/AlexR.
Not quite clear about the problem, can you elaborate? Joseph?
- Guangya
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40632/#review112309
-----------------------------------------------------------
On 一月 7, 2016, 11:27 a.m., Guangya Liu wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40632/
> -----------------------------------------------------------
>
> (Updated 一月 7, 2016, 11:27 a.m.)
>
>
> Review request for mesos, Ben Mahler, Artem Harutyunyan, Joris Van Remoortere, Joseph Wu, Klaus Ma, and Jian Qiu.
>
>
> Bugs: MESOS-4145
> https://issues.apache.org/jira/browse/MESOS-4145
>
>
> Repository: mesos
>
>
> Description
> -------
>
> Enabled oversubscribed resources for reservations in allocator.
>
>
> Diffs
> -----
>
> src/master/allocator/mesos/hierarchical.cpp df8bccaf2b8cfc0cb5ca18d4867371ae7a84c12f
> src/tests/hierarchical_allocator_tests.cpp e044f832c2c16e53e663c6ced5452649bb0dcb59
>
> Diff: https://reviews.apache.org/r/40632/diff/
>
>
> Testing
> -------
>
> make
> make check
>
>
> Thanks,
>
> Guangya Liu
>
>
Re: Review Request 40632: WIP: Enabled oversubscribed resources for
reservations in allocator.
Posted by Guangya Liu <gy...@gmail.com>.
> On 十二月 30, 2015, 12:42 p.m., Klaus Ma wrote:
> > src/master/allocator/mesos/hierarchical.cpp, line 1321
> > <https://reviews.apache.org/r/40632/diff/8/?file=1178313#file1178313line1321>
> >
> > As framework can not use ALLOCATION_SLACK mixed with other resources, allocator need to check seperatly.
Yes, currently the validation logic in master is https://github.com/apache/mesos/blob/master/src/master/validation.cpp#L479-L488 which can make sure a task and its executor can either use non-revocable resources or revocable resources of a given name but not both. After we introduced the allocation slack, we may want to guard that a task and its executor can either use allocation slack or usage slack or non-revocable resources but not all. I will upload a patch to handle this part.
- Guangya
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40632/#review112309
-----------------------------------------------------------
On 十二月 30, 2015, 7:04 a.m., Guangya Liu wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40632/
> -----------------------------------------------------------
>
> (Updated 十二月 30, 2015, 7:04 a.m.)
>
>
> Review request for mesos, Klaus Ma and Jian Qiu.
>
>
> Repository: mesos
>
>
> Description
> -------
>
> Enabled oversubscribed resources for reservations in allocator.
>
>
> Diffs
> -----
>
> src/master/allocator/mesos/hierarchical.cpp 7f900c4e024485704d79e57ae22407557598fe6c
>
> Diff: https://reviews.apache.org/r/40632/diff/
>
>
> Testing
> -------
>
> make
> make check
>
>
> Thanks,
>
> Guangya Liu
>
>
Re: Review Request 40632: Enabled oversubscribed resources for
reservations in allocator.
Posted by Klaus Ma <kl...@gmail.com>.
> On Dec. 30, 2015, 8:42 p.m., Klaus Ma wrote:
> > src/master/allocator/mesos/hierarchical.cpp, line 1321
> > <https://reviews.apache.org/r/40632/diff/8/?file=1178313#file1178313line1321>
> >
> > As framework can not use ALLOCATION_SLACK mixed with other resources, allocator need to check seperatly.
>
> Guangya Liu wrote:
> Yes, currently the validation logic in master is https://github.com/apache/mesos/blob/master/src/master/validation.cpp#L479-L488 which can make sure a task and its executor can either use non-revocable resources or revocable resources of a given name but not both. After we introduced the allocation slack, we may want to guard that a task and its executor can either use allocation slack or usage slack or non-revocable resources but not all. I will upload a patch to handle this part.
>
> Guangya Liu wrote:
> Fixed in https://reviews.apache.org/r/41848/
>
> Joseph Wu wrote:
> Note: mixing resources is allowed. Everything in a container is evicted at once, including non-revocable resources mixed into a revocable one.
re 'Note: mixing resources is allowed.', my understanding is that: different tasks can use different resources, but one task can NOT use mixing resources; for example, one task can not use both `USAGE_SLACK` and `ALLOCATION_SLACK`.
> On Dec. 30, 2015, 8:42 p.m., Klaus Ma wrote:
> > src/master/allocator/mesos/hierarchical.cpp, line 1334
> > <https://reviews.apache.org/r/40632/diff/8/?file=1178313#file1178313line1334>
> >
> > In this check, it seems allocator send offer when both !allocation & allocation are enough.
>
> Joseph Wu wrote:
> Looks like quota might have stepped on the toes of oversubscription. I'll double-check with Joris/AlexR.
>
> Guangya Liu wrote:
> Not quite clear about the problem, can you elaborate? Joseph?
@Joseph, the Quota only handle `unreserved.nonRevocable` resources; so `USAGE_SLACK` are not included in Quota.
- Klaus
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40632/#review112309
-----------------------------------------------------------
On Jan. 8, 2016, 2:55 p.m., Guangya Liu wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40632/
> -----------------------------------------------------------
>
> (Updated Jan. 8, 2016, 2:55 p.m.)
>
>
> Review request for mesos, Ben Mahler, Artem Harutyunyan, Joris Van Remoortere, Joseph Wu, Klaus Ma, and Jian Qiu.
>
>
> Bugs: MESOS-4145
> https://issues.apache.org/jira/browse/MESOS-4145
>
>
> Repository: mesos
>
>
> Description
> -------
>
> Enabled oversubscribed resources for reservations in allocator.
>
> There are three patches handling the allocator part:
> 1) This patch handles send offer, add slave and quota integration test.
> 2) https://reviews.apache.org/r/41847/ Handles the case of update slave.
> 3) https://reviews.apache.org/r/41791/ Handles the case of dynamic reservations.
>
>
> Diffs
> -----
>
> src/master/allocator/mesos/hierarchical.cpp df8bccaf2b8cfc0cb5ca18d4867371ae7a84c12f
> src/tests/hierarchical_allocator_tests.cpp e044f832c2c16e53e663c6ced5452649bb0dcb59
>
> Diff: https://reviews.apache.org/r/40632/diff/
>
>
> Testing
> -------
>
> make
> make check
>
>
> Thanks,
>
> Guangya Liu
>
>
Re: Review Request 40632: Enabled oversubscribed resources for
reservations in allocator.
Posted by Guangya Liu <gy...@gmail.com>.
> On 十二月 30, 2015, 12:42 p.m., Klaus Ma wrote:
> > src/master/allocator/mesos/hierarchical.cpp, line 1321
> > <https://reviews.apache.org/r/40632/diff/8/?file=1178313#file1178313line1321>
> >
> > As framework can not use ALLOCATION_SLACK mixed with other resources, allocator need to check seperatly.
>
> Guangya Liu wrote:
> Yes, currently the validation logic in master is https://github.com/apache/mesos/blob/master/src/master/validation.cpp#L479-L488 which can make sure a task and its executor can either use non-revocable resources or revocable resources of a given name but not both. After we introduced the allocation slack, we may want to guard that a task and its executor can either use allocation slack or usage slack or non-revocable resources but not all. I will upload a patch to handle this part.
Fixed in https://reviews.apache.org/r/41848/
- Guangya
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40632/#review112309
-----------------------------------------------------------
On 一月 3, 2016, 9:21 a.m., Guangya Liu wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40632/
> -----------------------------------------------------------
>
> (Updated 一月 3, 2016, 9:21 a.m.)
>
>
> Review request for mesos, Ben Mahler, Artem Harutyunyan, Joris Van Remoortere, Joseph Wu, Klaus Ma, and Jian Qiu.
>
>
> Bugs: MESOS-4145
> https://issues.apache.org/jira/browse/MESOS-4145
>
>
> Repository: mesos
>
>
> Description
> -------
>
> Enabled oversubscribed resources for reservations in allocator.
>
>
> Diffs
> -----
>
> src/master/allocator/mesos/hierarchical.cpp 7f900c4e024485704d79e57ae22407557598fe6c
> src/tests/hierarchical_allocator_tests.cpp e044f832c2c16e53e663c6ced5452649bb0dcb59
>
> Diff: https://reviews.apache.org/r/40632/diff/
>
>
> Testing
> -------
>
> make
> make check
>
>
> Thanks,
>
> Guangya Liu
>
>
Re: Review Request 40632: Enabled oversubscribed resources for
reservations in allocator.
Posted by Klaus Ma <kl...@gmail.com>.
> On Dec. 30, 2015, 8:42 p.m., Klaus Ma wrote:
> > src/master/allocator/mesos/hierarchical.cpp, line 1302
> > <https://reviews.apache.org/r/40632/diff/8/?file=1178313#file1178313line1302>
> >
> > If framework has reserved.statless resources, allocator should not dispatch it as ALLOCATION_SLACK.
>
> Klaus Ma wrote:
> @Joseph, I can not remenber where did you append your comments about this topic; but it seems you're OK to offer both `stateless.reserved` & `ALLOCATION_SLACK` of the same role, right? But how framework developer distinguish `stateless.reserved` & `ALLOCATION_SLACK` of its? If it launch task in `ALLOCATION_SLACK` and `stateless.reserved`, the task in `ALLOCATION_SLACK` was terminated immediately; it just increases overhead of cluster instead of resource utilization.
>
> Joseph Wu wrote:
> There's something about this in the design doc.
>
> The summary is: Meta-frameworks may benefit from receiving allocation slack from itself.
>
> Klaus Ma wrote:
> OK, would you help to update the design doc? There's comments that:
>
> NOTE: We need to be careful not to offer the same resources (as reserved resources and as revocable resources) to the same framework. This could presumably happen if a Lender Framework also accepts revocable resources.
>
> Guangya Liu wrote:
> I've already handled this case here: When sending offers, the allocation slack resources in the offer need to exclude all allocated stateless reserved resources and allocated allocation slack resources.
I think current code did not handle this case; if a Lender framekwork accepts revocable resources, the framework's `stateless.reserved` resource & allocation slack from that `stateless.reserved` will send to that famework; so "offer the same resources (as reserved resources and as revocable resources) to the same framework".
- Klaus
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40632/#review112309
-----------------------------------------------------------
On Jan. 8, 2016, 2:55 p.m., Guangya Liu wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40632/
> -----------------------------------------------------------
>
> (Updated Jan. 8, 2016, 2:55 p.m.)
>
>
> Review request for mesos, Ben Mahler, Artem Harutyunyan, Joris Van Remoortere, Joseph Wu, Klaus Ma, and Jian Qiu.
>
>
> Bugs: MESOS-4145
> https://issues.apache.org/jira/browse/MESOS-4145
>
>
> Repository: mesos
>
>
> Description
> -------
>
> Enabled oversubscribed resources for reservations in allocator.
>
> There are three patches handling the allocator part:
> 1) This patch handles send offer, add slave and quota integration test.
> 2) https://reviews.apache.org/r/41847/ Handles the case of update slave.
> 3) https://reviews.apache.org/r/41791/ Handles the case of dynamic reservations.
>
>
> Diffs
> -----
>
> src/master/allocator/mesos/hierarchical.cpp df8bccaf2b8cfc0cb5ca18d4867371ae7a84c12f
> src/tests/hierarchical_allocator_tests.cpp e044f832c2c16e53e663c6ced5452649bb0dcb59
>
> Diff: https://reviews.apache.org/r/40632/diff/
>
>
> Testing
> -------
>
> make
> make check
>
>
> Thanks,
>
> Guangya Liu
>
>
Re: Review Request 40632: Enabled oversubscribed resources for
reservations in allocator.
Posted by Klaus Ma <kl...@gmail.com>.
> On Dec. 30, 2015, 8:42 p.m., Klaus Ma wrote:
> > src/master/allocator/mesos/hierarchical.cpp, line 1302
> > <https://reviews.apache.org/r/40632/diff/8/?file=1178313#file1178313line1302>
> >
> > If framework has reserved.statless resources, allocator should not dispatch it as ALLOCATION_SLACK.
>
> Klaus Ma wrote:
> @Joseph, I can not remenber where did you append your comments about this topic; but it seems you're OK to offer both `stateless.reserved` & `ALLOCATION_SLACK` of the same role, right? But how framework developer distinguish `stateless.reserved` & `ALLOCATION_SLACK` of its? If it launch task in `ALLOCATION_SLACK` and `stateless.reserved`, the task in `ALLOCATION_SLACK` was terminated immediately; it just increases overhead of cluster instead of resource utilization.
>
> Joseph Wu wrote:
> There's something about this in the design doc.
>
> The summary is: Meta-frameworks may benefit from receiving allocation slack from itself.
OK, would you help to update the design doc? There's comments that:
NOTE: We need to be careful not to offer the same resources (as reserved resources and as revocable resources) to the same framework. This could presumably happen if a Lender Framework also accepts revocable resources.
- Klaus
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40632/#review112309
-----------------------------------------------------------
On Jan. 8, 2016, 2:55 p.m., Guangya Liu wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40632/
> -----------------------------------------------------------
>
> (Updated Jan. 8, 2016, 2:55 p.m.)
>
>
> Review request for mesos, Ben Mahler, Artem Harutyunyan, Joris Van Remoortere, Joseph Wu, Klaus Ma, and Jian Qiu.
>
>
> Bugs: MESOS-4145
> https://issues.apache.org/jira/browse/MESOS-4145
>
>
> Repository: mesos
>
>
> Description
> -------
>
> Enabled oversubscribed resources for reservations in allocator.
>
> There are three patches handling the allocator part:
> 1) This patch handles send offer, add slave and quota integration test.
> 2) https://reviews.apache.org/r/41847/ Handles the case of update slave.
> 3) https://reviews.apache.org/r/41791/ Handles the case of dynamic reservations.
>
>
> Diffs
> -----
>
> src/master/allocator/mesos/hierarchical.cpp df8bccaf2b8cfc0cb5ca18d4867371ae7a84c12f
> src/tests/hierarchical_allocator_tests.cpp e044f832c2c16e53e663c6ced5452649bb0dcb59
>
> Diff: https://reviews.apache.org/r/40632/diff/
>
>
> Testing
> -------
>
> make
> make check
>
>
> Thanks,
>
> Guangya Liu
>
>
Re: Review Request 40632: Enabled oversubscribed resources for
reservations in allocator.
Posted by Joseph Wu <jo...@mesosphere.io>.
> On Dec. 30, 2015, 4:42 a.m., Klaus Ma wrote:
> > src/master/allocator/mesos/hierarchical.cpp, line 1321
> > <https://reviews.apache.org/r/40632/diff/8/?file=1178313#file1178313line1321>
> >
> > As framework can not use ALLOCATION_SLACK mixed with other resources, allocator need to check seperatly.
>
> Guangya Liu wrote:
> Yes, currently the validation logic in master is https://github.com/apache/mesos/blob/master/src/master/validation.cpp#L479-L488 which can make sure a task and its executor can either use non-revocable resources or revocable resources of a given name but not both. After we introduced the allocation slack, we may want to guard that a task and its executor can either use allocation slack or usage slack or non-revocable resources but not all. I will upload a patch to handle this part.
>
> Guangya Liu wrote:
> Fixed in https://reviews.apache.org/r/41848/
Note: mixing resources is allowed. Everything in a container is evicted at once, including non-revocable resources mixed into a revocable one.
> On Dec. 30, 2015, 4:42 a.m., Klaus Ma wrote:
> > src/master/allocator/mesos/hierarchical.cpp, line 1334
> > <https://reviews.apache.org/r/40632/diff/8/?file=1178313#file1178313line1334>
> >
> > In this check, it seems allocator send offer when both !allocation & allocation are enough.
Looks like quota might have stepped on the toes of oversubscription. I'll double-check with Joris/AlexR.
- Joseph
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40632/#review112309
-----------------------------------------------------------
On Jan. 7, 2016, 3:27 a.m., Guangya Liu wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40632/
> -----------------------------------------------------------
>
> (Updated Jan. 7, 2016, 3:27 a.m.)
>
>
> Review request for mesos, Ben Mahler, Artem Harutyunyan, Joris Van Remoortere, Joseph Wu, Klaus Ma, and Jian Qiu.
>
>
> Bugs: MESOS-4145
> https://issues.apache.org/jira/browse/MESOS-4145
>
>
> Repository: mesos
>
>
> Description
> -------
>
> Enabled oversubscribed resources for reservations in allocator.
>
>
> Diffs
> -----
>
> src/master/allocator/mesos/hierarchical.cpp df8bccaf2b8cfc0cb5ca18d4867371ae7a84c12f
> src/tests/hierarchical_allocator_tests.cpp e044f832c2c16e53e663c6ced5452649bb0dcb59
>
> Diff: https://reviews.apache.org/r/40632/diff/
>
>
> Testing
> -------
>
> make
> make check
>
>
> Thanks,
>
> Guangya Liu
>
>
Re: Review Request 40632: WIP: Enabled oversubscribed resources for
reservations in allocator.
Posted by Klaus Ma <kl...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40632/#review112309
-----------------------------------------------------------
src/master/allocator/mesos/hierarchical.cpp (line 1302)
<https://reviews.apache.org/r/40632/#comment172746>
If framework has reserved.statless resources, allocator should not dispatch it as ALLOCATION_SLACK.
src/master/allocator/mesos/hierarchical.cpp (line 1321)
<https://reviews.apache.org/r/40632/#comment172745>
As framework can not use ALLOCATION_SLACK mixed with other resources, allocator need to check seperatly.
src/master/allocator/mesos/hierarchical.cpp (line 1334)
<https://reviews.apache.org/r/40632/#comment172747>
In this check, it seems allocator send offer when both !allocation & allocation are enough.
- Klaus Ma
On Dec. 30, 2015, 3:04 p.m., Guangya Liu wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40632/
> -----------------------------------------------------------
>
> (Updated Dec. 30, 2015, 3:04 p.m.)
>
>
> Review request for mesos, Klaus Ma and Jian Qiu.
>
>
> Repository: mesos
>
>
> Description
> -------
>
> Enabled oversubscribed resources for reservations in allocator.
>
>
> Diffs
> -----
>
> src/master/allocator/mesos/hierarchical.cpp 7f900c4e024485704d79e57ae22407557598fe6c
>
> Diff: https://reviews.apache.org/r/40632/diff/
>
>
> Testing
> -------
>
> make
> make check
>
>
> Thanks,
>
> Guangya Liu
>
>
Re: Review Request 40632: WIP: Enabled oversubscribed resources for
reservations in allocator.
Posted by Guangya Liu <gy...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40632/
-----------------------------------------------------------
(Updated 十二月 30, 2015, 7:04 a.m.)
Review request for mesos, Klaus Ma and Jian Qiu.
Summary (updated)
-----------------
WIP: Enabled oversubscribed resources for reservations in allocator.
Repository: mesos
Description
-------
Enabled oversubscribed resources for reservations in allocator.
Diffs (updated)
-----
src/master/allocator/mesos/hierarchical.cpp 7f900c4e024485704d79e57ae22407557598fe6c
Diff: https://reviews.apache.org/r/40632/diff/
Testing
-------
make
make check
Thanks,
Guangya Liu
Re: Review Request 40632: Enabled oversubscribed resources for
reservations in allocator.
Posted by Guangya Liu <gy...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40632/
-----------------------------------------------------------
(Updated Dec. 14, 2015, 6:41 a.m.)
Review request for mesos, Artem Harutyunyan, Joris Van Remoortere, Joseph Wu, and Klaus Ma.
Summary (updated)
-----------------
Enabled oversubscribed resources for reservations in allocator.
Repository: mesos
Description (updated)
-------
Enabled oversubscribed resources for reservations in allocator.
Diffs
-----
src/master/allocator/mesos/hierarchical.cpp 5da825a1d578a9ee40b4985378fddb3c5fb3b416
Diff: https://reviews.apache.org/r/40632/diff/
Testing (updated)
-------
make
make check
Thanks,
Guangya Liu
Re: Review Request 40632: WIP: Enabled oversubscribed resources for
reservations in allocator.
Posted by Guangya Liu <gy...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40632/
-----------------------------------------------------------
(Updated Dec. 14, 2015, 6:31 a.m.)
Review request for mesos, Artem Harutyunyan, Joris Van Remoortere, Joseph Wu, and Klaus Ma.
Repository: mesos
Description
-------
WIP: Enabled oversubscribed resources for reservations in allocator.
Diffs (updated)
-----
src/master/allocator/mesos/hierarchical.cpp 5da825a1d578a9ee40b4985378fddb3c5fb3b416
Diff: https://reviews.apache.org/r/40632/diff/
Testing
-------
Thanks,
Guangya Liu
Re: Review Request 40632: WIP: Enabled oversubscribed resources for
reservations in allocator.
Posted by Guangya Liu <gy...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40632/
-----------------------------------------------------------
(Updated Dec. 13, 2015, 11:35 a.m.)
Review request for mesos, Artem Harutyunyan, Joris Van Remoortere, Joseph Wu, and Klaus Ma.
Repository: mesos
Description
-------
WIP: Enabled oversubscribed resources for reservations in allocator.
Diffs (updated)
-----
src/master/allocator/mesos/hierarchical.cpp 5da825a1d578a9ee40b4985378fddb3c5fb3b416
Diff: https://reviews.apache.org/r/40632/diff/
Testing
-------
Thanks,
Guangya Liu
Re: Review Request 40632: WIP: Enabled oversubscribed resources for
reservations in allocator.
Posted by Guangya Liu <gy...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40632/
-----------------------------------------------------------
(Updated 十二月 10, 2015, 8:31 a.m.)
Review request for mesos, Artem Harutyunyan, Joris Van Remoortere, Joseph Wu, and Klaus Ma.
Repository: mesos
Description
-------
WIP: Enabled oversubscribed resources for reservations in allocator.
Diffs (updated)
-----
src/master/allocator/mesos/hierarchical.cpp 5da825a1d578a9ee40b4985378fddb3c5fb3b416
Diff: https://reviews.apache.org/r/40632/diff/
Testing
-------
Thanks,
Guangya Liu
Re: Review Request 40632: WIP: Enabled oversubscribed resources for
reservations in allocator.
Posted by Mesos ReviewBot <re...@mesos.apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40632/#review109643
-----------------------------------------------------------
Bad patch!
Reviews applied: [40375, 40339, 40529, 40532, 40632]
Failed command: export OS=ubuntu:14.04;export CONFIGURATION="--verbose";export COMPILER=gcc; ./support/docker_build.sh
Error:
+ : ubuntu:14.04
+ : gcc
+ : --verbose
+++ dirname ./support/docker_build.sh
++ cd ./support/..
++ pwd
+ MESOS_DIRECTORY=/home/jenkins/jenkins-slave/workspace/mesos-reviewbot
+ cd /home/jenkins/jenkins-slave/workspace/mesos-reviewbot
+ DOCKERFILE=Dockerfile
+ rm -f Dockerfile
+ case $OS in
+ append_dockerfile 'FROM ubuntu:14.04'
+ echo FROM ubuntu:14.04
+ append_dockerfile 'RUN rm -rf /var/lib/apt/lists/*'
+ echo RUN rm -rf /var/lib/apt/lists/get.docker.io_ubuntu_dists_docker_main_binary-amd64_Packages /var/lib/apt/lists/get.docker.io_ubuntu_dists_docker_main_binary-i386_Packages /var/lib/apt/lists/get.docker.io_ubuntu_dists_docker_Release /var/lib/apt/lists/get.docker.io_ubuntu_dists_docker_Release.gpg /var/lib/apt/lists/lock /var/lib/apt/lists/packages.apache.org_asf%5finternal_dists_trusty_main_binary-amd64_Packages /var/lib/apt/lists/packages.apache.org_asf%5finternal_dists_trusty_main_binary-i386_Packages /var/lib/apt/lists/packages.apache.org_asf%5finternal_dists_trusty_Release /var/lib/apt/lists/packages.apache.org_asf%5finternal_dists_trusty_Release.gpg /var/lib/apt/lists/partial /var/lib/apt/lists/ppa.launchpad.net_webupd8team_java_ubuntu_dists_trusty_main_binary-amd64_Packages /var/lib/apt/lists/ppa.launchpad.net_webupd8team_java_ubuntu_dists_trusty_main_binary-i386_Packages /var/lib/apt/lists/ppa.launchpad.net_webupd8team_java_ubuntu_dists_trusty_main_i18n_Translation-en /va
r/lib/apt/lists/ppa.launchpad.net_webupd8team_java_ubuntu_dists_trusty_main_source_Sources /var/lib/apt/lists/ppa.launchpad.net_webupd8team_java_ubuntu_dists_trusty_Release /var/lib/apt/lists/security.ubuntu.com_ubuntu_dists_trusty-security_InRelease /var/lib/apt/lists/security.ubuntu.com_ubuntu_dists_trusty-security_main_binary-amd64_Packages /var/lib/apt/lists/security.ubuntu.com_ubuntu_dists_trusty-security_main_binary-i386_Packages /var/lib/apt/lists/security.ubuntu.com_ubuntu_dists_trusty-security_main_i18n_Translation-en /var/lib/apt/lists/security.ubuntu.com_ubuntu_dists_trusty-security_main_source_Sources /var/lib/apt/lists/security.ubuntu.com_ubuntu_dists_trusty-security_multiverse_binary-amd64_Packages /var/lib/apt/lists/security.ubuntu.com_ubuntu_dists_trusty-security_multiverse_binary-i386_Packages /var/lib/apt/lists/security.ubuntu.com_ubuntu_dists_trusty-security_multiverse_i18n_Translation-en /var/lib/apt/lists/security.ubuntu.com_ubuntu_dists_trusty-security_multiver
se_source_Sources /var/lib/apt/lists/security.ubuntu.com_ubuntu_dists_trusty-security_restricted_binary-amd64_Packages /var/lib/apt/lists/security.ubuntu.com_ubuntu_dists_trusty-security_restricted_binary-i386_Packages /var/lib/apt/lists/security.ubuntu.com_ubuntu_dists_trusty-security_restricted_i18n_Translation-en /var/lib/apt/lists/security.ubuntu.com_ubuntu_dists_trusty-security_restricted_source_Sources /var/lib/apt/lists/security.ubuntu.com_ubuntu_dists_trusty-security_universe_binary-amd64_Packages /var/lib/apt/lists/security.ubuntu.com_ubuntu_dists_trusty-security_universe_binary-i386_Packages /var/lib/apt/lists/security.ubuntu.com_ubuntu_dists_trusty-security_universe_i18n_Translation-en /var/lib/apt/lists/security.ubuntu.com_ubuntu_dists_trusty-security_universe_source_Sources /var/lib/apt/lists/us.archive.ubuntu.com_ubuntu_dists_trusty-backports_InRelease /var/lib/apt/lists/us.archive.ubuntu.com_ubuntu_dists_trusty-backports_main_binary-amd64_Packages /var/lib/apt/lists/u
s.archive.ubuntu.com_ubuntu_dists_trusty-backports_main_binary-i386_Packages /var/lib/apt/lists/us.archive.ubuntu.com_ubuntu_dists_trusty-backports_main_i18n_Translation-en /var/lib/apt/lists/us.archive.ubuntu.com_ubuntu_dists_trusty-backports_main_source_Sources /var/lib/apt/lists/us.archive.ubuntu.com_ubuntu_dists_trusty-backports_multiverse_binary-amd64_Packages /var/lib/apt/lists/us.archive.ubuntu.com_ubuntu_dists_trusty-backports_multiverse_binary-i386_Packages /var/lib/apt/lists/us.archive.ubuntu.com_ubuntu_dists_trusty-backports_multiverse_i18n_Translation-en /var/lib/apt/lists/us.archive.ubuntu.com_ubuntu_dists_trusty-backports_multiverse_source_Sources /var/lib/apt/lists/us.archive.ubuntu.com_ubuntu_dists_trusty-backports_restricted_binary-amd64_Packages /var/lib/apt/lists/us.archive.ubuntu.com_ubuntu_dists_trusty-backports_restricted_binary-i386_Packages /var/lib/apt/lists/us.archive.ubuntu.com_ubuntu_dists_trusty-backports_restricted_i18n_Translation-en /var/lib/apt/lists
/us.archive.ubuntu.com_ubuntu_dists_trusty-backports_restricted_source_Sources /var/lib/apt/lists/us.archive.ubuntu.com_ubuntu_dists_trusty-backports_universe_binary-amd64_Packages /var/lib/apt/lists/us.archive.ubuntu.com_ubuntu_dists_trusty-backports_universe_binary-i386_Packages /var/lib/apt/lists/us.archive.ubuntu.com_ubuntu_dists_trusty-backports_universe_i18n_Translation-en /var/lib/apt/lists/us.archive.ubuntu.com_ubuntu_dists_trusty-backports_universe_source_Sources /var/lib/apt/lists/us.archive.ubuntu.com_ubuntu_dists_trusty_main_binary-amd64_Packages /var/lib/apt/lists/us.archive.ubuntu.com_ubuntu_dists_trusty_main_binary-i386_Packages /var/lib/apt/lists/us.archive.ubuntu.com_ubuntu_dists_trusty_main_i18n_Translation-en /var/lib/apt/lists/us.archive.ubuntu.com_ubuntu_dists_trusty_main_source_Sources /var/lib/apt/lists/us.archive.ubuntu.com_ubuntu_dists_trusty_multiverse_binary-amd64_Packages /var/lib/apt/lists/us.archive.ubuntu.com_ubuntu_dists_trusty_multiverse_binary-i386_
Packages /var/lib/apt/lists/us.archive.ubuntu.com_ubuntu_dists_trusty_multiverse_i18n_Translation-en /var/lib/apt/lists/us.archive.ubuntu.com_ubuntu_dists_trusty_multiverse_source_Sources /var/lib/apt/lists/us.archive.ubuntu.com_ubuntu_dists_trusty_Release /var/lib/apt/lists/us.archive.ubuntu.com_ubuntu_dists_trusty_Release.gpg /var/lib/apt/lists/us.archive.ubuntu.com_ubuntu_dists_trusty_restricted_binary-amd64_Packages /var/lib/apt/lists/us.archive.ubuntu.com_ubuntu_dists_trusty_restricted_binary-i386_Packages /var/lib/apt/lists/us.archive.ubuntu.com_ubuntu_dists_trusty_restricted_i18n_Translation-en /var/lib/apt/lists/us.archive.ubuntu.com_ubuntu_dists_trusty_restricted_source_Sources /var/lib/apt/lists/us.archive.ubuntu.com_ubuntu_dists_trusty_universe_binary-amd64_Packages /var/lib/apt/lists/us.archive.ubuntu.com_ubuntu_dists_trusty_universe_binary-i386_Packages /var/lib/apt/lists/us.archive.ubuntu.com_ubuntu_dists_trusty_universe_i18n_Translation-en /var/lib/apt/lists/us.archiv
e.ubuntu.com_ubuntu_dists_trusty_universe_source_Sources /var/lib/apt/lists/us.archive.ubuntu.com_ubuntu_dists_trusty-updates_InRelease /var/lib/apt/lists/us.archive.ubuntu.com_ubuntu_dists_trusty-updates_main_binary-amd64_Packages /var/lib/apt/lists/us.archive.ubuntu.com_ubuntu_dists_trusty-updates_main_binary-i386_Packages /var/lib/apt/lists/us.archive.ubuntu.com_ubuntu_dists_trusty-updates_main_i18n_Translation-en /var/lib/apt/lists/us.archive.ubuntu.com_ubuntu_dists_trusty-updates_main_source_Sources /var/lib/apt/lists/us.archive.ubuntu.com_ubuntu_dists_trusty-updates_multiverse_binary-amd64_Packages /var/lib/apt/lists/us.archive.ubuntu.com_ubuntu_dists_trusty-updates_multiverse_binary-i386_Packages /var/lib/apt/lists/us.archive.ubuntu.com_ubuntu_dists_trusty-updates_multiverse_i18n_Translation-en /var/lib/apt/lists/us.archive.ubuntu.com_ubuntu_dists_trusty-updates_multiverse_source_Sources /var/lib/apt/lists/us.archive.ubuntu.com_ubuntu_dists_trusty-updates_restricted_binary-am
d64_Packages /var/lib/apt/lists/us.archive.ubuntu.com_ubuntu_dists_trusty-updates_restricted_binary-i386_Packages /var/lib/apt/lists/us.archive.ubuntu.com_ubuntu_dists_trusty-updates_restricted_i18n_Translation-en /var/lib/apt/lists/us.archive.ubuntu.com_ubuntu_dists_trusty-updates_restricted_source_Sources /var/lib/apt/lists/us.archive.ubuntu.com_ubuntu_dists_trusty-updates_universe_binary-amd64_Packages /var/lib/apt/lists/us.archive.ubuntu.com_ubuntu_dists_trusty-updates_universe_binary-i386_Packages /var/lib/apt/lists/us.archive.ubuntu.com_ubuntu_dists_trusty-updates_universe_i18n_Translation-en /var/lib/apt/lists/us.archive.ubuntu.com_ubuntu_dists_trusty-updates_universe_source_Sources
+ append_dockerfile 'RUN apt-get update'
+ echo RUN apt-get update
+ append_dockerfile 'RUN apt-get -y install build-essential clang git maven autoconf libtool'
+ echo RUN apt-get -y install build-essential clang git maven autoconf libtool
+ append_dockerfile 'RUN apt-get -y install openjdk-7-jdk python-dev python-boto libcurl4-nss-dev libsasl2-dev libapr1-dev libsvn-dev libevent-dev libev-dev'
+ echo RUN apt-get -y install openjdk-7-jdk python-dev python-boto libcurl4-nss-dev libsasl2-dev libapr1-dev libsvn-dev libevent-dev libev-dev
+ append_dockerfile 'RUN adduser --disabled-password --gecos '\'''\'' mesos'
+ echo RUN adduser --disabled-password --gecos ''\'''\''' mesos
+ append_dockerfile 'ENV GTEST_FILTER -FsTest.FileSystemTableRead'
+ echo ENV GTEST_FILTER -FsTest.FileSystemTableRead
+ case $COMPILER in
+ append_dockerfile 'ENV CC gcc'
+ echo ENV CC gcc
+ append_dockerfile 'ENV CXX g++'
+ echo ENV CXX g++
+ append_dockerfile 'WORKDIR mesos'
+ echo WORKDIR mesos
+ append_dockerfile 'COPY . /mesos/'
+ echo COPY . /mesos/
+ append_dockerfile 'RUN chown -R mesos /mesos'
+ echo RUN chown -R mesos /mesos
+ append_dockerfile 'USER mesos'
+ echo USER mesos
+ append_dockerfile 'CMD ./bootstrap && ./configure --verbose && DISTCHECK_CONFIGURE_FLAGS="--verbose" GLOG_v=1 MESOS_VERBOSE=1 make -j8 distcheck'
+ echo CMD ./bootstrap '&&' ./configure --verbose '&&' 'DISTCHECK_CONFIGURE_FLAGS="--verbose"' GLOG_v=1 MESOS_VERBOSE=1 make -j8 distcheck
++ date +%s
+ TAG=mesos-1449707084-6291
+ docker build --no-cache=true -t mesos-1449707084-6291 .
Sending build context to Docker daemon 552.6 kB
Sending build context to Docker daemon 1.079 MB
Sending build context to Docker daemon 1.606 MB
Sending build context to Docker daemon 2.163 MB
Sending build context to Docker daemon 2.72 MB
Sending build context to Docker daemon 3.277 MB
Sending build context to Docker daemon 3.834 MB
Sending build context to Docker daemon 4.391 MB
Sending build context to Docker daemon 4.948 MB
Sending build context to Docker daemon 5.505 MB
Sending build context to Docker daemon 6.062 MB
Sending build context to Docker daemon 6.619 MB
Sending build context to Docker daemon 7.176 MB
Sending build context to Docker daemon 7.733 MB
Sending build context to Docker daemon 8.29 MB
Sending build context to Docker daemon 8.847 MB
Sending build context to Docker daemon 9.404 MB
Sending build context to Docker daemon 9.961 MB
Sending build context to Docker daemon 10.52 MB
Sending build context to Docker daemon 11.08 MB
Sending build context to Docker daemon 11.63 MB
Sending build context to Docker daemon 12.19 MB
Sending build context to Docker daemon 12.75 MB
Sending build context to Docker daemon 13.3 MB
Sending build context to Docker daemon 13.86 MB
Sending build context to Docker daemon 14.42 MB
Sending build context to Docker daemon 14.97 MB
Sending build context to Docker daemon 15.53 MB
Sending build context to Docker daemon 16.09 MB
Sending build context to Docker daemon 16.65 MB
Sending build context to Docker daemon 17.2 MB
Sending build context to Docker daemon 17.76 MB
Sending build context to Docker daemon 18.32 MB
Sending build context to Docker daemon 18.87 MB
Sending build context to Docker daemon 19.43 MB
Sending build context to Docker daemon 19.99 MB
Sending build context to Docker daemon 20.55 MB
Sending build context to Docker daemon 21.1 MB
Sending build context to Docker daemon 21.66 MB
Sending build context to Docker daemon 22.22 MB
Sending build context to Docker daemon 22.77 MB
Sending build context to Docker daemon 23.33 MB
Sending build context to Docker daemon 23.89 MB
Sending build context to Docker daemon 24.44 MB
Sending build context to Docker daemon 25 MB
Sending build context to Docker daemon 25.56 MB
Sending build context to Docker daemon 26.12 MB
Sending build context to Docker daemon 26.67 MB
Sending build context to Docker daemon 27.23 MB
Sending build context to Docker daemon 27.79 MB
Sending build context to Docker daemon 28.34 MB
Sending build context to Docker daemon 28.9 MB
Sending build context to Docker daemon 29.46 MB
Sending build context to Docker daemon 30.02 MB
Sending build context to Docker daemon 30.57 MB
Sending build context to Docker daemon 31.13 MB
Sending build context to Docker daemon 31.69 MB
Sending build context to Docker daemon 32.24 MB
Sending build context to Docker daemon 32.8 MB
Sending build context to Docker daemon 33.36 MB
Sending build context to Docker daemon 33.91 MB
Sending build context to Docker daemon 34.47 MB
Sending build context to Docker daemon 35.03 MB
Sending build context to Docker daemon 35.59 MB
Sending build context to Docker daemon 36.14 MB
Sending build context to Docker daemon 36.7 MB
Sending build context to Docker daemon 37.26 MB
Sending build context to Docker daemon 37.81 MB
Sending build context to Docker daemon 38.37 MB
Sending build context to Docker daemon 38.93 MB
Sending build context to Docker daemon 39.49 MB
Sending build context to Docker daemon 40.04 MB
Sending build context to Docker daemon 40.6 MB
Sending build context to Docker daemon 41.16 MB
Sending build context to Docker daemon 41.71 MB
Sending build context to Docker daemon 42.27 MB
Sending build context to Docker daemon 42.83 MB
Sending build context to Docker daemon 43.38 MB
Sending build context to Docker daemon 43.94 MB
Sending build context to Docker daemon 44.5 MB
Sending build context to Docker daemon 45.06 MB
Sending build context to Docker daemon 45.61 MB
Sending build context to Docker daemon 46.17 MB
Sending build context to Docker daemon 46.73 MB
Sending build context to Docker daemon 47.28 MB
Sending build context to Docker daemon 47.84 MB
Sending build context to Docker daemon 48.4 MB
Sending build context to Docker daemon 48.96 MB
Sending build context to Docker daemon 49.51 MB
Sending build context to Docker daemon 50.07 MB
Sending build context to Docker daemon 50.63 MB
Sending build context to Docker daemon 51.18 MB
Sending build context to Docker daemon 51.74 MB
Sending build context to Docker daemon 52.3 MB
Sending build context to Docker daemon 52.85 MB
Sending build context to Docker daemon 53.41 MB
Sending build context to Docker daemon 53.97 MB
Sending build context to Docker daemon 54.53 MB
Sending build context to Docker daemon 55.08 MB
Sending build context to Docker daemon 55.64 MB
Sending build context to Docker daemon 56.2 MB
Sending build context to Docker daemon 56.75 MB
Sending build context to Docker daemon 57.31 MB
Sending build context to Docker daemon 57.87 MB
Sending build context to Docker daemon 58.43 MB
Sending build context to Docker daemon 58.98 MB
Sending build context to Docker daemon 59.54 MB
Sending build context to Docker daemon 60.1 MB
Sending build context to Docker daemon 60.65 MB
Sending build context to Docker daemon 61.21 MB
Sending build context to Docker daemon 61.77 MB
Sending build context to Docker daemon 62.32 MB
Sending build context to Docker daemon 62.88 MB
Sending build context to Docker daemon 63.44 MB
Sending build context to Docker daemon 64 MB
Sending build context to Docker daemon 64.55 MB
Sending build context to Docker daemon 65.11 MB
Sending build context to Docker daemon 65.67 MB
Sending build context to Docker daemon 66.22 MB
Sending build context to Docker daemon 66.78 MB
Sending build context to Docker daemon 67.34 MB
Sending build context to Docker daemon 67.9 MB
Sending build context to Docker daemon 68.45 MB
Sending build context to Docker daemon 69.01 MB
Sending build context to Docker daemon 69.57 MB
Sending build context to Docker daemon 70.12 MB
Sending build context to Docker daemon 70.68 MB
Sending build context to Docker daemon 71.24 MB
Sending build context to Docker daemon 71.79 MB
Sending build context to Docker daemon 72.35 MB
Sending build context to Docker daemon 72.91 MB
Sending build context to Docker daemon 73.47 MB
Sending build context to Docker daemon 74.02 MB
Sending build context to Docker daemon 74.58 MB
Sending build context to Docker daemon 75.14 MB
Sending build context to Docker daemon 75.69 MB
Sending build context to Docker daemon 76.25 MB
Sending build context to Docker daemon 76.81 MB
Sending build context to Docker daemon 77.37 MB
Sending build context to Docker daemon 77.92 MB
Sending build context to Docker daemon 78.48 MB
Sending build context to Docker daemon 79.04 MB
Sending build context to Docker daemon 79.59 MB
Sending build context to Docker daemon 80.15 MB
Sending build context to Docker daemon 80.71 MB
Sending build context to Docker daemon 81.26 MB
Sending build context to Docker daemon 81.82 MB
Sending build context to Docker daemon 82.38 MB
Sending build context to Docker daemon 82.94 MB
Sending build context to Docker daemon 83.49 MB
Sending build context to Docker daemon 84.05 MB
Sending build context to Docker daemon 84.61 MB
Sending build context to Docker daemon 85.16 MB
Sending build context to Docker daemon 85.72 MB
Sending build context to Docker daemon 86.28 MB
Sending build context to Docker daemon 86.84 MB
Sending build context to Docker daemon 87.39 MB
Sending build context to Docker daemon 87.95 MB
Sending build context to Docker daemon 88.51 MB
Sending build context to Docker daemon 89.06 MB
Sending build context to Docker daemon 89.62 MB
Sending build context to Docker daemon 90.18 MB
Sending build context to Docker daemon 90.73 MB
Sending build context to Docker daemon 91.29 MB
Sending build context to Docker daemon 91.85 MB
Sending build context to Docker daemon 92.41 MB
Sending build context to Docker daemon 92.96 MB
Sending build context to Docker daemon 93.52 MB
Sending build context to Docker daemon 94.08 MB
Sending build context to Docker daemon 94.63 MB
Sending build context to Docker daemon 95.19 MB
Sending build context to Docker daemon 95.75 MB
Sending build context to Docker daemon 96.31 MB
Sending build context to Docker daemon 96.86 MB
Sending build context to Docker daemon 97.42 MB
Sending build context to Docker daemon 97.98 MB
Sending build context to Docker daemon 98.53 MB
Sending build context to Docker daemon 99.09 MB
Sending build context to Docker daemon 99.65 MB
Sending build context to Docker daemon 100.2 MB
Sending build context to Docker daemon 100.8 MB
Sending build context to Docker daemon 101.3 MB
Sending build context to Docker daemon 101.9 MB
Sending build context to Docker daemon 102.4 MB
Sending build context to Docker daemon 103 MB
Sending build context to Docker daemon 103.5 MB
Sending build context to Docker daemon 104.1 MB
Sending build context to Docker daemon 104.7 MB
Sending build context to Docker daemon 105.2 MB
Sending build context to Docker daemon 105.8 MB
Sending build context to Docker daemon 106.3 MB
Sending build context to Docker daemon 106.9 MB
Sending build context to Docker daemon 107.4 MB
Sending build context to Docker daemon 108 MB
Sending build context to Docker daemon 108.6 MB
Sending build context to Docker daemon 109.1 MB
Sending build context to Docker daemon 109.7 MB
Sending build context to Docker daemon 110.2 MB
Sending build context to Docker daemon 110.8 MB
Sending build context to Docker daemon 111.3 MB
Sending build context to Docker daemon 111.9 MB
Sending build context to Docker daemon 112.5 MB
Sending build context to Docker daemon 113 MB
Sending build context to Docker daemon 113.6 MB
Sending build context to Docker daemon 114.1 MB
Sending build context to Docker daemon 114.7 MB
Sending build context to Docker daemon 115.2 MB
Sending build context to Docker daemon 115.8 MB
Sending build context to Docker daemon 116.4 MB
Sending build context to Docker daemon 116.9 MB
Sending build context to Docker daemon 117.5 MB
Sending build context to Docker daemon 118 MB
Sending build context to Docker daemon 118.6 MB
Sending build context to Docker daemon 119.1 MB
Sending build context to Docker daemon 119.7 MB
Sending build context to Docker daemon 120.3 MB
Sending build context to Docker daemon 120.8 MB
Sending build context to Docker daemon 121.4 MB
Sending build context to Docker daemon 121.9 MB
Sending build context to Docker daemon 122.5 MB
Sending build context to Docker daemon 123 MB
Sending build context to Docker daemon 123.6 MB
Sending build context to Docker daemon 124.2 MB
Sending build context to Docker daemon 124.7 MB
Sending build context to Docker daemon 125.3 MB
Sending build context to Docker daemon 125.8 MB
Sending build context to Docker daemon 126.4 MB
Sending build context to Docker daemon 126.9 MB
Sending build context to Docker daemon 127.5 MB
Sending build context to Docker daemon 128.1 MB
Sending build context to Docker daemon 128.6 MB
Sending build context to Docker daemon 129.2 MB
Sending build context to Docker daemon 129.7 MB
Sending build context to Docker daemon 130.3 MB
Sending build context to Docker daemon 130.8 MB
Sending build context to Docker daemon 131.4 MB
Sending build context to Docker daemon 132 MB
Sending build context to Docker daemon 132.5 MB
Sending build context to Docker daemon 133.1 MB
Sending build context to Docker daemon 133.6 MB
Sending build context to Docker daemon 134.2 MB
Sending build context to Docker daemon 134.7 MB
Sending build context to Docker daemon 135.3 MB
Sending build context to Docker daemon 135.9 MB
Sending build context to Docker daemon 136.4 MB
Sending build context to Docker daemon 137 MB
Sending build context to Docker daemon 137.5 MB
Sending build context to Docker daemon 138.1 MB
Sending build context to Docker daemon 138.6 MB
Sending build context to Docker daemon 139.2 MB
Sending build context to Docker daemon 139.8 MB
Sending build context to Docker daemon 140.3 MB
Sending build context to Docker daemon 140.9 MB
Sending build context to Docker daemon 141.4 MB
Sending build context to Docker daemon 142 MB
Sending build context to Docker daemon 142.5 MB
Sending build context to Docker daemon 143.1 MB
Sending build context to Docker daemon 143.7 MB
Sending build context to Docker daemon 144.2 MB
Sending build context to Docker daemon 144.8 MB
Sending build context to Docker daemon 145.3 MB
Sending build context to Docker daemon 145.9 MB
Sending build context to Docker daemon 146.4 MB
Sending build context to Docker daemon 147 MB
Sending build context to Docker daemon 147.6 MB
Sending build context to Docker daemon 148.1 MB
Sending build context to Docker daemon 148.7 MB
Sending build context to Docker daemon 149.2 MB
Sending build context to Docker daemon 149.8 MB
Sending build context to Docker daemon 150.3 MB
Sending build context to Docker daemon 150.9 MB
Sending build context to Docker daemon 151.5 MB
Sending build context to Docker daemon 152 MB
Sending build context to Docker daemon 152.6 MB
Sending build context to Docker daemon 153.1 MB
Sending build context to Docker daemon 153.7 MB
Sending build context to Docker daemon 154.2 MB
Sending build context to Docker daemon 154.8 MB
Sending build context to Docker daemon 155.4 MB
Sending build context to Docker daemon 155.9 MB
Sending build context to Docker daemon 156.5 MB
Sending build context to Docker daemon 157 MB
Sending build context to Docker daemon 157.6 MB
Sending build context to Docker daemon 158.1 MB
Sending build context to Docker daemon 158.7 MB
Sending build context to Docker daemon 159.3 MB
Sending build context to Docker daemon 159.8 MB
Sending build context to Docker daemon 160.4 MB
Sending build context to Docker daemon 160.9 MB
Sending build context to Docker daemon 161.5 MB
Sending build context to Docker daemon 162 MB
Sending build context to Docker daemon 162.6 MB
Sending build context to Docker daemon 163.2 MB
Sending build context to Docker daemon 163.7 MB
Sending build context to Docker daemon 164.3 MB
Sending build context to Docker daemon 164.8 MB
Sending build context to Docker daemon 165.4 MB
Sending build context to Docker daemon 165.9 MB
Sending build context to Docker daemon 166.5 MB
Sending build context to Docker daemon 167.1 MB
Sending build context to Docker daemon 167.6 MB
Sending build context to Docker daemon 168.2 MB
Sending build context to Docker daemon 168.7 MB
Sending build context to Docker daemon 169.3 MB
Sending build context to Docker daemon 169.8 MB
Sending build context to Docker daemon 170.4 MB
Sending build context to Docker daemon 171 MB
Sending build context to Docker daemon 171.5 MB
Sending build context to Docker daemon 172.1 MB
Sending build context to Docker daemon 172.6 MB
Sending build context to Docker daemon 173.2 MB
Sending build context to Docker daemon 173.7 MB
Sending build context to Docker daemon 174.3 MB
Sending build context to Docker daemon 174.9 MB
Sending build context to Docker daemon 175.4 MB
Sending build context to Docker daemon 176 MB
Sending build context to Docker daemon 176.5 MB
Sending build context to Docker daemon 177.1 MB
Sending build context to Docker daemon 177.6 MB
Sending build context to Docker daemon 178.2 MB
Sending build context to Docker daemon 178.7 MB
Sending build context to Docker daemon 179.3 MB
Sending build context to Docker daemon 179.9 MB
Sending build context to Docker daemon 180.4 MB
Sending build context to Docker daemon 181 MB
Sending build context to Docker daemon 181.5 MB
Sending build context to Docker daemon 182.1 MB
Sending build context to Docker daemon 182.6 MB
Sending build context to Docker daemon 183.2 MB
Sending build context to Docker daemon 183.8 MB
Sending build context to Docker daemon 184.3 MB
Sending build context to Docker daemon 184.9 MB
Sending build context to Docker daemon 185.4 MB
Sending build context to Docker daemon 186 MB
Sending build context to Docker daemon 186.5 MB
Sending build context to Docker daemon 187.1 MB
Sending build context to Docker daemon 187.7 MB
Sending build context to Docker daemon 188.2 MB
Sending build context to Docker daemon 188.8 MB
Sending build context to Docker daemon 189.3 MB
Sending build context to Docker daemon 189.9 MB
Sending build context to Docker daemon 190.4 MB
Sending build context to Docker daemon 191 MB
Sending build context to Docker daemon 191.6 MB
Sending build context to Docker daemon 192.1 MB
Sending build context to Docker daemon 192.7 MB
Sending build context to Docker daemon 193.2 MB
Sending build context to Docker daemon 193.8 MB
Sending build context to Docker daemon 194.3 MB
Sending build context to Docker daemon 194.9 MB
Sending build context to Docker daemon 195.5 MB
Sending build context to Docker daemon 196 MB
Sending build context to Docker daemon 196.6 MB
Sending build context to Docker daemon 197.1 MB
Sending build context to Docker daemon 197.7 MB
Sending build context to Docker daemon 198.2 MB
Sending build context to Docker daemon 198.8 MB
Sending build context to Docker daemon 199.4 MB
Sending build context to Docker daemon 199.9 MB
Sending build context to Docker daemon 200.5 MB
Sending build context to Docker daemon 201 MB
Sending build context to Docker daemon 201.6 MB
Sending build context to Docker daemon 202.1 MB
Sending build context to Docker daemon 202.7 MB
Sending build context to Docker daemon 203.3 MB
Sending build context to Docker daemon 203.8 MB
Sending build context to Docker daemon 204.4 MB
Sending build context to Docker daemon 204.9 MB
Sending build context to Docker daemon 205.5 MB
Sending build context to Docker daemon 206 MB
Sending build context to Docker daemon 206.6 MB
Sending build context to Docker daemon 207.2 MB
Sending build context to Docker daemon 207.7 MB
Sending build context to Docker daemon 208.3 MB
Sending build context to Docker daemon 208.8 MB
Sending build context to Docker daemon 209.4 MB
Sending build context to Docker daemon 209.9 MB
Sending build context to Docker daemon 210.5 MB
Sending build context to Docker daemon 211.1 MB
Sending build context to Docker daemon 211.6 MB
Sending build context to Docker daemon 212.2 MB
Sending build context to Docker daemon 212.7 MB
Sending build context to Docker daemon 213.3 MB
Sending build context to Docker daemon 213.8 MB
Sending build context to Docker daemon 214.4 MB
Sending build context to Docker daemon 215 MB
Sending build context to Docker daemon 215.5 MB
Sending build context to Docker daemon 216.1 MB
Sending build context to Docker daemon 216.6 MB
Sending build context to Docker daemon 217.2 MB
Sending build context to Docker daemon 217.7 MB
Sending build context to Docker daemon 218.3 MB
Sending build context to Docker daemon 218.9 MB
Sending build context to Docker daemon 219.4 MB
Sending build context to Docker daemon 220 MB
Sending build context to Docker daemon 220.5 MB
Sending build context to Docker daemon 221.1 MB
Sending build context to Docker daemon 221.6 MB
Sending build context to Docker daemon 222.2 MB
Sending build context to Docker daemon 222.8 MB
Sending build context to Docker daemon 223.3 MB
Sending build context to Docker daemon 223.9 MB
Sending build context to Docker daemon 224.4 MB
Sending build context to Docker daemon 225 MB
Sending build context to Docker daemon 225.5 MB
Sending build context to Docker daemon 226.1 MB
Sending build context to Docker daemon 226.7 MB
Sending build context to Docker daemon 227.2 MB
Sending build context to Docker daemon 227.8 MB
Sending build context to Docker daemon 228.3 MB
Sending build context to Docker daemon 228.9 MB
Sending build context to Docker daemon 229.4 MB
Sending build context to Docker daemon 230 MB
Sending build context to Docker daemon 230.6 MB
Sending build context to Docker daemon 231.1 MB
Sending build context to Docker daemon 231.7 MB
Sending build context to Docker daemon 232.2 MB
Sending build context to Docker daemon 232.8 MB
Sending build context to Docker daemon 233.3 MB
Sending build context to Docker daemon 233.9 MB
Sending build context to Docker daemon 234.5 MB
Sending build context to Docker daemon 235 MB
Sending build context to Docker daemon 235.6 MB
Sending build context to Docker daemon 236.1 MB
Sending build context to Docker daemon 236.7 MB
Sending build context to Docker daemon 237.2 MB
Sending build context to Docker daemon 237.8 MB
Sending build context to Docker daemon 238.4 MB
Sending build context to Docker daemon 238.9 MB
Sending build context to Docker daemon 239.5 MB
Sending build context to Docker daemon 240 MB
Sending build context to Docker daemon 240.6 MB
Sending build context to Docker daemon 241.1 MB
Sending build context to Docker daemon 241.7 MB
Sending build context to Docker daemon 242.2 MB
Sending build context to Docker daemon 242.8 MB
Sending build context to Docker daemon 243.3 MB
Sending build context to Docker daemon 243.9 MB
Sending build context to Docker daemon 244.4 MB
Sending build context to Docker daemon 245 MB
Sending build context to Docker daemon 245.6 MB
Sending build context to Docker daemon 246.1 MB
Sending build context to Docker daemon 246.7 MB
Sending build context to Docker daemon 247.2 MB
Sending build context to Docker daemon 247.8 MB
Sending build context to Docker daemon 248.3 MB
Sending build context to Docker daemon 248.9 MB
Sending build context to Docker daemon 249.5 MB
Sending build context to Docker daemon 250 MB
Sending build context to Docker daemon 250.5 MB
Sending build context to Docker daemon 251.1 MB
Sending build context to Docker daemon 251.6 MB
Sending build context to Docker daemon 252.1 MB
Sending build context to Docker daemon 252.7 MB
Sending build context to Docker daemon 253.3 MB
Sending build context to Docker daemon 253.8 MB
Sending build context to Docker daemon 254.4 MB
Sending build context to Docker daemon 254.9 MB
Sending build context to Docker daemon 255.5 MB
Sending build context to Docker daemon 256 MB
Sending build context to Docker daemon 256.6 MB
Sending build context to Docker daemon 257.2 MB
Sending build context to Docker daemon 257.7 MB
Sending build context to Docker daemon 258.3 MB
Sending build context to Docker daemon 258.8 MB
Sending build context to Docker daemon 259.4 MB
Sending build context to Docker daemon 259.9 MB
Sending build context to Docker daemon 260.5 MB
Sending build context to Docker daemon 261.1 MB
Sending build context to Docker daemon 261.6 MB
Sending build context to Docker daemon 262.2 MB
Sending build context to Docker daemon 262.7 MB
Sending build context to Docker daemon 263.3 MB
Sending build context to Docker daemon 263.8 MB
Sending build context to Docker daemon 264.4 MB
Sending build context to Docker daemon 265 MB
Sending build context to Docker daemon 265.5 MB
Sending build context to Docker daemon 266.1 MB
Sending build context to Docker daemon 266.6 MB
Sending build context to Docker daemon 267.2 MB
Sending build context to Docker daemon 267.7 MB
Sending build context to Docker daemon 268.3 MB
Sending build context to Docker daemon 268.9 MB
Sending build context to Docker daemon 269.4 MB
Sending build context to Docker daemon 270 MB
Sending build context to Docker daemon 270.5 MB
Sending build context to Docker daemon 271 MB
Sending build context to Docker daemon 271.6 MB
Sending build context to Docker daemon 272.1 MB
Sending build context to Docker daemon 272.6 MB
Sending build context to Docker daemon 273.2 MB
Sending build context to Docker daemon 273.7 MB
Sending build context to Docker daemon 274.3 MB
Sending build context to Docker daemon 274.8 MB
Sending build context to Docker daemon 275.4 MB
Sending build context to Docker daemon 275.9 MB
Sending build context to Docker daemon 276.4 MB
Sending build context to Docker daemon 277 MB
Sending build context to Docker daemon 277.5 MB
Sending build context to Docker daemon 278 MB
Sending build context to Docker daemon 278.5 MB
Sending build context to Docker daemon 279.1 MB
Sending build context to Docker daemon 279.6 MB
Sending build context to Docker daemon 280.1 MB
Sending build context to Docker daemon 280.7 MB
Sending build context to Docker daemon 281.2 MB
Sending build context to Docker daemon 281.7 MB
Sending build context to Docker daemon 282.3 MB
Sending build context to Docker daemon 282.8 MB
Sending build context to Docker daemon 283.4 MB
Sending build context to Docker daemon 283.6 MB
Sending build context to Docker daemon 283.6 MB
Step 1 : FROM ubuntu:14.04
---> 89d5d8e8bafb
Step 2 : RUN rm -rf /var/lib/apt/lists/get.docker.io_ubuntu_dists_docker_main_binary-amd64_Packages /var/lib/apt/lists/get.docker.io_ubuntu_dists_docker_main_binary-i386_Packages /var/lib/apt/lists/get.docker.io_ubuntu_dists_docker_Release /var/lib/apt/lists/get.docker.io_ubuntu_dists_docker_Release.gpg /var/lib/apt/lists/lock /var/lib/apt/lists/packages.apache.org_asf%5finternal_dists_trusty_main_binary-amd64_Packages /var/lib/apt/lists/packages.apache.org_asf%5finternal_dists_trusty_main_binary-i386_Packages /var/lib/apt/lists/packages.apache.org_asf%5finternal_dists_trusty_Release /var/lib/apt/lists/packages.apache.org_asf%5finternal_dists_trusty_Release.gpg /var/lib/apt/lists/partial /var/lib/apt/lists/ppa.launchpad.net_webupd8team_java_ubuntu_dists_trusty_main_binary-amd64_Packages /var/lib/apt/lists/ppa.launchpad.net_webupd8team_java_ubuntu_dists_trusty_main_binary-i386_Packages /var/lib/apt/lists/ppa.launchpad.net_webupd8team_java_ubuntu_dists_trusty_main_i18n_Translation-en /
var/lib/apt/lists/ppa.launchpad.net_webupd8team_java_ubuntu_dists_trusty_main_source_Sources /var/lib/apt/lists/ppa.launchpad.net_webupd8team_java_ubuntu_dists_trusty_Release /var/lib/apt/lists/security.ubuntu.com_ubuntu_dists_trusty-security_InRelease /var/lib/apt/lists/security.ubuntu.com_ubuntu_dists_trusty-security_main_binary-amd64_Packages /var/lib/apt/lists/security.ubuntu.com_ubuntu_dists_trusty-security_main_binary-i386_Packages /var/lib/apt/lists/security.ubuntu.com_ubuntu_dists_trusty-security_main_i18n_Translation-en /var/lib/apt/lists/security.ubuntu.com_ubuntu_dists_trusty-security_main_source_Sources /var/lib/apt/lists/security.ubuntu.com_ubuntu_dists_trusty-security_multiverse_binary-amd64_Packages /var/lib/apt/lists/security.ubuntu.com_ubuntu_dists_trusty-security_multiverse_binary-i386_Packages /var/lib/apt/lists/security.ubuntu.com_ubuntu_dists_trusty-security_multiverse_i18n_Translation-en /var/lib/apt/lists/security.ubuntu.com_ubuntu_dists_trusty-security_multiv
erse_source_Sources /var/lib/apt/lists/security.ubuntu.com_ubuntu_dists_trusty-security_restricted_binary-amd64_Packages /var/lib/apt/lists/security.ubuntu.com_ubuntu_dists_trusty-security_restricted_binary-i386_Packages /var/lib/apt/lists/security.ubuntu.com_ubuntu_dists_trusty-security_restricted_i18n_Translation-en /var/lib/apt/lists/security.ubuntu.com_ubuntu_dists_trusty-security_restricted_source_Sources /var/lib/apt/lists/security.ubuntu.com_ubuntu_dists_trusty-security_universe_binary-amd64_Packages /var/lib/apt/lists/security.ubuntu.com_ubuntu_dists_trusty-security_universe_binary-i386_Packages /var/lib/apt/lists/security.ubuntu.com_ubuntu_dists_trusty-security_universe_i18n_Translation-en /var/lib/apt/lists/security.ubuntu.com_ubuntu_dists_trusty-security_universe_source_Sources /var/lib/apt/lists/us.archive.ubuntu.com_ubuntu_dists_trusty-backports_InRelease /var/lib/apt/lists/us.archive.ubuntu.com_ubuntu_dists_trusty-backports_main_binary-amd64_Packages /var/lib/apt/lists
/us.archive.ubuntu.com_ubuntu_dists_trusty-backports_main_binary-i386_Packages /var/lib/apt/lists/us.archive.ubuntu.com_ubuntu_dists_trusty-backports_main_i18n_Translation-en /var/lib/apt/lists/us.archive.ubuntu.com_ubuntu_dists_trusty-backports_main_source_Sources /var/lib/apt/lists/us.archive.ubuntu.com_ubuntu_dists_trusty-backports_multiverse_binary-amd64_Packages /var/lib/apt/lists/us.archive.ubuntu.com_ubuntu_dists_trusty-backports_multiverse_binary-i386_Packages /var/lib/apt/lists/us.archive.ubuntu.com_ubuntu_dists_trusty-backports_multiverse_i18n_Translation-en /var/lib/apt/lists/us.archive.ubuntu.com_ubuntu_dists_trusty-backports_multiverse_source_Sources /var/lib/apt/lists/us.archive.ubuntu.com_ubuntu_dists_trusty-backports_restricted_binary-amd64_Packages /var/lib/apt/lists/us.archive.ubuntu.com_ubuntu_dists_trusty-backports_restricted_binary-i386_Packages /var/lib/apt/lists/us.archive.ubuntu.com_ubuntu_dists_trusty-backports_restricted_i18n_Translation-en /var/lib/apt/lis
ts/us.archive.ubuntu.com_ubuntu_dists_trusty-backports_restricted_source_Sources /var/lib/apt/lists/us.archive.ubuntu.com_ubuntu_dists_trusty-backports_universe_binary-amd64_Packages /var/lib/apt/lists/us.archive.ubuntu.com_ubuntu_dists_trusty-backports_universe_binary-i386_Packages /var/lib/apt/lists/us.archive.ubuntu.com_ubuntu_dists_trusty-backports_universe_i18n_Translation-en /var/lib/apt/lists/us.archive.ubuntu.com_ubuntu_dists_trusty-backports_universe_source_Sources /var/lib/apt/lists/us.archive.ubuntu.com_ubuntu_dists_trusty_main_binary-amd64_Packages /var/lib/apt/lists/us.archive.ubuntu.com_ubuntu_dists_trusty_main_binary-i386_Packages /var/lib/apt/lists/us.archive.ubuntu.com_ubuntu_dists_trusty_main_i18n_Translation-en /var/lib/apt/lists/us.archive.ubuntu.com_ubuntu_dists_trusty_main_source_Sources /var/lib/apt/lists/us.archive.ubuntu.com_ubuntu_dists_trusty_multiverse_binary-amd64_Packages /var/lib/apt/lists/us.archive.ubuntu.com_ubuntu_dists_trusty_multiverse_binary-i38
6_Packages /var/lib/apt/lists/us.archive.ubuntu.com_ubuntu_dists_trusty_multiverse_i18n_Translation-en /var/lib/apt/lists/us.archive.ubuntu.com_ubuntu_dists_trusty_multiverse_source_Sources /var/lib/apt/lists/us.archive.ubuntu.com_ubuntu_dists_trusty_Release /var/lib/apt/lists/us.archive.ubuntu.com_ubuntu_dists_trusty_Release.gpg /var/lib/apt/lists/us.archive.ubuntu.com_ubuntu_dists_trusty_restricted_binary-amd64_Packages /var/lib/apt/lists/us.archive.ubuntu.com_ubuntu_dists_trusty_restricted_binary-i386_Packages /var/lib/apt/lists/us.archive.ubuntu.com_ubuntu_dists_trusty_restricted_i18n_Translation-en /var/lib/apt/lists/us.archive.ubuntu.com_ubuntu_dists_trusty_restricted_source_Sources /var/lib/apt/lists/us.archive.ubuntu.com_ubuntu_dists_trusty_universe_binary-amd64_Packages /var/lib/apt/lists/us.archive.ubuntu.com_ubuntu_dists_trusty_universe_binary-i386_Packages /var/lib/apt/lists/us.archive.ubuntu.com_ubuntu_dists_trusty_universe_i18n_Translation-en /var/lib/apt/lists/us.arch
ive.ubuntu.com_ubuntu_dists_trusty_universe_source_Sources /var/lib/apt/lists/us.archive.ubuntu.com_ubuntu_dists_trusty-updates_InRelease /var/lib/apt/lists/us.archive.ubuntu.com_ubuntu_dists_trusty-updates_main_binary-amd64_Packages /var/lib/apt/lists/us.archive.ubuntu.com_ubuntu_dists_trusty-updates_main_binary-i386_Packages /var/lib/apt/lists/us.archive.ubuntu.com_ubuntu_dists_trusty-updates_main_i18n_Translation-en /var/lib/apt/lists/us.archive.ubuntu.com_ubuntu_dists_trusty-updates_main_source_Sources /var/lib/apt/lists/us.archive.ubuntu.com_ubuntu_dists_trusty-updates_multiverse_binary-amd64_Packages /var/lib/apt/lists/us.archive.ubuntu.com_ubuntu_dists_trusty-updates_multiverse_binary-i386_Packages /var/lib/apt/lists/us.archive.ubuntu.com_ubuntu_dists_trusty-updates_multiverse_i18n_Translation-en /var/lib/apt/lists/us.archive.ubuntu.com_ubuntu_dists_trusty-updates_multiverse_source_Sources /var/lib/apt/lists/us.archive.ubuntu.com_ubuntu_dists_trusty-updates_restricted_binary-
amd64_Packages /var/lib/apt/lists/us.archive.ubuntu.com_ubuntu_dists_trusty-updates_restricted_binary-i386_Packages /var/lib/apt/lists/us.archive.ubuntu.com_ubuntu_dists_trusty-updates_restricted_i18n_Translation-en /var/lib/apt/lists/us.archive.ubuntu.com_ubuntu_dists_trusty-updates_restricted_source_Sources /var/lib/apt/lists/us.archive.ubuntu.com_ubuntu_dists_trusty-updates_universe_binary-amd64_Packages /var/lib/apt/lists/us.archive.ubuntu.com_ubuntu_dists_trusty-updates_universe_binary-i386_Packages /var/lib/apt/lists/us.archive.ubuntu.com_ubuntu_dists_trusty-updates_universe_i18n_Translation-en /var/lib/apt/lists/us.archive.ubuntu.com_ubuntu_dists_trusty-updates_universe_source_Sources
---> Running in fca9a1194347
---> dc73068de3cd
Removing intermediate container fca9a1194347
Step 3 : RUN apt-get update
---> Running in d596d0d9580f
Ign http://archive.ubuntu.com trusty InRelease
Get:1 http://archive.ubuntu.com trusty-updates InRelease [64.4 kB]
Get:2 http://archive.ubuntu.com trusty-security InRelease [64.4 kB]
Hit http://archive.ubuntu.com trusty Release.gpg
Hit http://archive.ubuntu.com trusty Release
Get:3 http://archive.ubuntu.com trusty-updates/main Sources [311 kB]
Get:4 http://archive.ubuntu.com trusty-updates/restricted Sources [5219 B]
Get:5 http://archive.ubuntu.com trusty-updates/universe Sources [183 kB]
Get:6 http://archive.ubuntu.com trusty-updates/main amd64 Packages [836 kB]
Get:7 http://archive.ubuntu.com trusty-updates/restricted amd64 Packages [23.4 kB]
Get:8 http://archive.ubuntu.com trusty-updates/universe amd64 Packages [432 kB]
Get:9 http://archive.ubuntu.com trusty-security/main Sources [127 kB]
Get:10 http://archive.ubuntu.com trusty-security/restricted Sources [3920 B]
Get:11 http://archive.ubuntu.com trusty-security/universe Sources [36.5 kB]
Get:12 http://archive.ubuntu.com trusty-security/main amd64 Packages [477 kB]
Get:13 http://archive.ubuntu.com trusty-security/restricted amd64 Packages [20.2 kB]
Get:14 http://archive.ubuntu.com trusty-security/universe amd64 Packages [156 kB]
Get:15 http://archive.ubuntu.com trusty/main Sources [1335 kB]
Get:16 http://archive.ubuntu.com trusty/restricted Sources [5335 B]
Get:17 http://archive.ubuntu.com trusty/universe Sources [7926 kB]
Get:18 http://archive.ubuntu.com trusty/main amd64 Packages [1743 kB]
Get:19 http://archive.ubuntu.com trusty/restricted amd64 Packages [16.0 kB]
Get:20 http://archive.ubuntu.com trusty/universe amd64 Packages [7589 kB]
Fetched 21.4 MB in 5s (3891 kB/s)
[91mW: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/trusty-updates/main/binary-amd64/Packages Hash Sum mismatch
W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/trusty-updates/universe/binary-amd64/Packages Hash Sum mismatch
E: Some index files failed to download. They have been ignored, or old ones used instead.
[0mThe command '/bin/sh -c apt-get update' returned a non-zero code: 100
- Mesos ReviewBot
On Dec. 9, 2015, 11:33 p.m., Guangya Liu wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40632/
> -----------------------------------------------------------
>
> (Updated Dec. 9, 2015, 11:33 p.m.)
>
>
> Review request for mesos and Klaus Ma.
>
>
> Repository: mesos
>
>
> Description
> -------
>
> WIP: Enabled oversubscribed resources for reservations in allocator.
>
>
> Diffs
> -----
>
> src/cli/execute.cpp ac933b6a8636af3356f3d6a99656c0185077c262
> src/master/allocator/mesos/hierarchical.cpp 5da825a1d578a9ee40b4985378fddb3c5fb3b416
>
> Diff: https://reviews.apache.org/r/40632/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Guangya Liu
>
>
Re: Review Request 40632: WIP: Enabled oversubscribed resources for
reservations in allocator.
Posted by Guangya Liu <gy...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40632/
-----------------------------------------------------------
(Updated 十二月 9, 2015, 11:33 p.m.)
Review request for mesos and Klaus Ma.
Repository: mesos
Description
-------
WIP: Enabled oversubscribed resources for reservations in allocator.
Diffs (updated)
-----
src/cli/execute.cpp ac933b6a8636af3356f3d6a99656c0185077c262
src/master/allocator/mesos/hierarchical.cpp 5da825a1d578a9ee40b4985378fddb3c5fb3b416
Diff: https://reviews.apache.org/r/40632/diff/
Testing
-------
Thanks,
Guangya Liu
Re: Review Request 40632: WIP: Enabled oversubscribed resources for
reservations in allocator.
Posted by Guangya Liu <gy...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40632/
-----------------------------------------------------------
(Updated 十二月 9, 2015, 12:38 p.m.)
Review request for mesos and Klaus Ma.
Repository: mesos
Description
-------
WIP: Enabled oversubscribed resources for reservations in allocator.
Diffs (updated)
-----
src/cli/execute.cpp ac933b6a8636af3356f3d6a99656c0185077c262
src/master/allocator/mesos/hierarchical.cpp 5da825a1d578a9ee40b4985378fddb3c5fb3b416
Diff: https://reviews.apache.org/r/40632/diff/
Testing
-------
Thanks,
Guangya Liu
Re: Review Request 40632: WIP: Enabled oversubscribed resources for
reservations in allocator.
Posted by Guangya Liu <gy...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40632/
-----------------------------------------------------------
(Updated 十二月 3, 2015, 4:34 a.m.)
Review request for mesos and Klaus Ma.
Repository: mesos
Description
-------
WIP: Enabled oversubscribed resources for reservations in allocator.
Diffs
-----
src/master/allocator/mesos/hierarchical.cpp aee8ced1fbfec8cf30cb939ff67fadfc6b08f37a
Diff: https://reviews.apache.org/r/40632/diff/
Testing
-------
Thanks,
Guangya Liu
Re: Review Request 40632: WIP: Enabled oversubscribed resources for
reservations in allocator.
Posted by Mesos ReviewBot <re...@mesos.apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40632/#review108022
-----------------------------------------------------------
Patch looks great!
Reviews applied: [40339, 40524, 40529, 40532, 40632]
Passed command: export OS=ubuntu:14.04;export CONFIGURATION="--verbose";export COMPILER=gcc; ./support/docker_build.sh
- Mesos ReviewBot
On Nov. 25, 2015, 7:31 a.m., Guangya Liu wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40632/
> -----------------------------------------------------------
>
> (Updated Nov. 25, 2015, 7:31 a.m.)
>
>
> Review request for mesos and Klaus Ma.
>
>
> Repository: mesos
>
>
> Description
> -------
>
> WIP: Enabled oversubscribed resources for reservations in allocator.
>
>
> Diffs
> -----
>
> src/master/allocator/mesos/hierarchical.cpp aee8ced1fbfec8cf30cb939ff67fadfc6b08f37a
>
> Diff: https://reviews.apache.org/r/40632/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Guangya Liu
>
>
Re: Review Request 40632: WIP: Enabled oversubscribed resources for
reservations in allocator.
Posted by Guangya Liu <gy...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40632/
-----------------------------------------------------------
(Updated 十一月 25, 2015, 7:31 a.m.)
Review request for mesos and Klaus Ma.
Repository: mesos
Description
-------
WIP: Enabled oversubscribed resources for reservations in allocator.
Diffs (updated)
-----
src/master/allocator/mesos/hierarchical.cpp aee8ced1fbfec8cf30cb939ff67fadfc6b08f37a
Diff: https://reviews.apache.org/r/40632/diff/
Testing
-------
Thanks,
Guangya Liu