You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@mesos.apache.org by Michael Park <mc...@gmail.com> on 2015/06/24 17:35:23 UTC

Review Request 35816: Fixed the incorrect CHECK_EQ in updateAllocation.

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/35816/
-----------------------------------------------------------

Review request for mesos, Ben Mahler and Jie Yu.


Repository: mesos


Description
-------

Fixed the incorrect `CHECK_EQ` introduced in [r35699](https://reviews.apache.org/r/35699/).

>From @bmahler:
> Whoops, Jie just noticed that this isn't correct because 'updatedAllocation' is for the framework only, whereas 'total' and 'available' are for all frameworks on the slave.


Diffs
-----

  src/master/allocator/mesos/hierarchical.hpp ef18ff850addfb5ce3500ed28e8ffd801e2d24eb 
  src/tests/hierarchical_allocator_tests.cpp 3258840135290cd008ca09235d18b7f093dafd2e 

Diff: https://reviews.apache.org/r/35816/diff/


Testing
-------

(1) Added a test `HierarchicalAllocatorTest.UpdateAllocationMultipleFrameworks` which breaks the previous `CHECK_EQ` condition.
(2) `make check`


Thanks,

Michael Park

Re: Review Request 35816: Fixed the incorrect CHECK_EQ in updateAllocation.

Posted by Alexander Rukletsov <ru...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/35816/#review89666
-----------------------------------------------------------



src/tests/hierarchical_allocator_tests.cpp (line 791)
<https://reviews.apache.org/r/35816/#comment142321>

    Why do you pause the clock here?


- Alexander Rukletsov


On June 24, 2015, 3:35 p.m., Michael Park wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/35816/
> -----------------------------------------------------------
> 
> (Updated June 24, 2015, 3:35 p.m.)
> 
> 
> Review request for mesos, Ben Mahler and Jie Yu.
> 
> 
> Repository: mesos
> 
> 
> Description
> -------
> 
> Fixed the incorrect `CHECK_EQ` introduced in [r35699](https://reviews.apache.org/r/35699/).
> 
> From @bmahler:
> > Whoops, Jie just noticed that this isn't correct because 'updatedAllocation' is for the framework only, whereas 'total' and 'available' are for all frameworks on the slave.
> 
> 
> Diffs
> -----
> 
>   src/master/allocator/mesos/hierarchical.hpp ef18ff850addfb5ce3500ed28e8ffd801e2d24eb 
>   src/tests/hierarchical_allocator_tests.cpp 3258840135290cd008ca09235d18b7f093dafd2e 
> 
> Diff: https://reviews.apache.org/r/35816/diff/
> 
> 
> Testing
> -------
> 
> (1) Added a test `HierarchicalAllocatorTest.UpdateAllocationMultipleFrameworks` which breaks the previous `CHECK_EQ` condition.
> (2) `make check`
> 
> 
> Thanks,
> 
> Michael Park
> 
>

Re: Review Request 35816: Fixed the incorrect CHECK_EQ in updateAllocation.

Posted by Mesos ReviewBot <re...@mesos.apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/35816/#review89179
-----------------------------------------------------------


Patch looks great!

Reviews applied: [35816]

All tests passed.

- Mesos ReviewBot


On June 24, 2015, 3:35 p.m., Michael Park wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/35816/
> -----------------------------------------------------------
> 
> (Updated June 24, 2015, 3:35 p.m.)
> 
> 
> Review request for mesos, Ben Mahler and Jie Yu.
> 
> 
> Repository: mesos
> 
> 
> Description
> -------
> 
> Fixed the incorrect `CHECK_EQ` introduced in [r35699](https://reviews.apache.org/r/35699/).
> 
> From @bmahler:
> > Whoops, Jie just noticed that this isn't correct because 'updatedAllocation' is for the framework only, whereas 'total' and 'available' are for all frameworks on the slave.
> 
> 
> Diffs
> -----
> 
>   src/master/allocator/mesos/hierarchical.hpp ef18ff850addfb5ce3500ed28e8ffd801e2d24eb 
>   src/tests/hierarchical_allocator_tests.cpp 3258840135290cd008ca09235d18b7f093dafd2e 
> 
> Diff: https://reviews.apache.org/r/35816/diff/
> 
> 
> Testing
> -------
> 
> (1) Added a test `HierarchicalAllocatorTest.UpdateAllocationMultipleFrameworks` which breaks the previous `CHECK_EQ` condition.
> (2) `make check`
> 
> 
> Thanks,
> 
> Michael Park
> 
>

Re: Review Request 35816: Fixed the incorrect CHECK_EQ in updateAllocation.

Posted by Michael Park <mc...@gmail.com>.

> On June 25, 2015, 12:31 a.m., Ben Mahler wrote:
> > Thanks Michael! Did you see jie's comment on [MESOS-1187](https://issues.apache.org/jira/browse/MESOS-1187)? Looks like that will make these kinds of checks problematic.
> > 
> > Also, it's going to be possible for the slave to be over-allocated due to oversubscription (currently the total from addSlave doesn't include the oversubpription resources, and in the future we might want to express over-allocation, see: [MESOS-2930](https://issues.apache.org/jira/browse/MESOS-2930).

Hey Ben! I think the `CHECK` does need to be dropped, more so because of https://reviews.apache.org/r/35836. Question: should I just drop the `CHECK` in this patch and keep the test? or should I just drop this patch all together?
> Did you see jie's comment on MESOS-1187? Looks like that will make these kinds of checks problematic.

Yeah, I spoke to Jie about [MESOS-1187](https://issues.apache.org/jira/browse/MESOS-1187) earlier today. That's really a separate concern though, since that bug breaks our expectations of how `contains` and `equality` works for `Resources` in general.


- Michael


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/35816/#review89295
-----------------------------------------------------------


On June 24, 2015, 3:35 p.m., Michael Park wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/35816/
> -----------------------------------------------------------
> 
> (Updated June 24, 2015, 3:35 p.m.)
> 
> 
> Review request for mesos, Ben Mahler and Jie Yu.
> 
> 
> Repository: mesos
> 
> 
> Description
> -------
> 
> Fixed the incorrect `CHECK_EQ` introduced in [r35699](https://reviews.apache.org/r/35699/).
> 
> From @bmahler:
> > Whoops, Jie just noticed that this isn't correct because 'updatedAllocation' is for the framework only, whereas 'total' and 'available' are for all frameworks on the slave.
> 
> 
> Diffs
> -----
> 
>   src/master/allocator/mesos/hierarchical.hpp ef18ff850addfb5ce3500ed28e8ffd801e2d24eb 
>   src/tests/hierarchical_allocator_tests.cpp 3258840135290cd008ca09235d18b7f093dafd2e 
> 
> Diff: https://reviews.apache.org/r/35816/diff/
> 
> 
> Testing
> -------
> 
> (1) Added a test `HierarchicalAllocatorTest.UpdateAllocationMultipleFrameworks` which breaks the previous `CHECK_EQ` condition.
> (2) `make check`
> 
> 
> Thanks,
> 
> Michael Park
> 
>

Re: Review Request 35816: Fixed the incorrect CHECK_EQ in updateAllocation.

Posted by Ben Mahler <be...@gmail.com>.

> On June 25, 2015, 12:31 a.m., Ben Mahler wrote:
> > Thanks Michael! Did you see jie's comment on [MESOS-1187](https://issues.apache.org/jira/browse/MESOS-1187)? Looks like that will make these kinds of checks problematic.
> > 
> > Also, it's going to be possible for the slave to be over-allocated due to oversubscription (currently the total from addSlave doesn't include the oversubpription resources, and in the future we might want to express over-allocation, see: [MESOS-2930](https://issues.apache.org/jira/browse/MESOS-2930).
> 
> Michael Park wrote:
>     Hey Ben! I think the `CHECK` does need to be dropped, more so because of https://reviews.apache.org/r/35836. Question: should I just drop the `CHECK` in this patch and keep the test? or should I just drop this patch all together?
>     > Did you see jie's comment on MESOS-1187? Looks like that will make these kinds of checks problematic.
>     
>     Yeah, I spoke to Jie about [MESOS-1187](https://issues.apache.org/jira/browse/MESOS-1187) earlier today. That's really a separate concern though, since that bug breaks our expectations of how `contains` and `equality` works for `Resources` in general.

Doesn't MESOS-1187 make these kinds of checks prone to failing?


- Ben


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/35816/#review89295
-----------------------------------------------------------


On June 24, 2015, 3:35 p.m., Michael Park wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/35816/
> -----------------------------------------------------------
> 
> (Updated June 24, 2015, 3:35 p.m.)
> 
> 
> Review request for mesos, Ben Mahler and Jie Yu.
> 
> 
> Repository: mesos
> 
> 
> Description
> -------
> 
> Fixed the incorrect `CHECK_EQ` introduced in [r35699](https://reviews.apache.org/r/35699/).
> 
> From @bmahler:
> > Whoops, Jie just noticed that this isn't correct because 'updatedAllocation' is for the framework only, whereas 'total' and 'available' are for all frameworks on the slave.
> 
> 
> Diffs
> -----
> 
>   src/master/allocator/mesos/hierarchical.hpp ef18ff850addfb5ce3500ed28e8ffd801e2d24eb 
>   src/tests/hierarchical_allocator_tests.cpp 3258840135290cd008ca09235d18b7f093dafd2e 
> 
> Diff: https://reviews.apache.org/r/35816/diff/
> 
> 
> Testing
> -------
> 
> (1) Added a test `HierarchicalAllocatorTest.UpdateAllocationMultipleFrameworks` which breaks the previous `CHECK_EQ` condition.
> (2) `make check`
> 
> 
> Thanks,
> 
> Michael Park
> 
>

Re: Review Request 35816: Fixed the incorrect CHECK_EQ in updateAllocation.

Posted by Michael Park <mc...@gmail.com>.

> On June 25, 2015, 12:31 a.m., Ben Mahler wrote:
> > Thanks Michael! Did you see jie's comment on [MESOS-1187](https://issues.apache.org/jira/browse/MESOS-1187)? Looks like that will make these kinds of checks problematic.
> > 
> > Also, it's going to be possible for the slave to be over-allocated due to oversubscription (currently the total from addSlave doesn't include the oversubpription resources, and in the future we might want to express over-allocation, see: [MESOS-2930](https://issues.apache.org/jira/browse/MESOS-2930).
> 
> Michael Park wrote:
>     Hey Ben! I think the `CHECK` does need to be dropped, more so because of https://reviews.apache.org/r/35836. Question: should I just drop the `CHECK` in this patch and keep the test? or should I just drop this patch all together?
>     > Did you see jie's comment on MESOS-1187? Looks like that will make these kinds of checks problematic.
>     
>     Yeah, I spoke to Jie about [MESOS-1187](https://issues.apache.org/jira/browse/MESOS-1187) earlier today. That's really a separate concern though, since that bug breaks our expectations of how `contains` and `equality` works for `Resources` in general.
> 
> Ben Mahler wrote:
>     Doesn't MESOS-1187 make these kinds of checks prone to failing?

It does, but checks such as: `CHECK(slaves[slaveId].allocated.contains(resources))` is prone to failing, which also means our logic in `Resources::apply` where we do a bunch of `if (!result.contains(stripped))` etc are prone to failing.

My point is that we need to address MESOS-1187 by making these operations be well-behaved. Avoiding a few `CHECK`s don't mean much when the `if` conditions that we rely on are also broken due to precision errors.


- Michael


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/35816/#review89295
-----------------------------------------------------------


On June 24, 2015, 3:35 p.m., Michael Park wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/35816/
> -----------------------------------------------------------
> 
> (Updated June 24, 2015, 3:35 p.m.)
> 
> 
> Review request for mesos, Ben Mahler and Jie Yu.
> 
> 
> Repository: mesos
> 
> 
> Description
> -------
> 
> Fixed the incorrect `CHECK_EQ` introduced in [r35699](https://reviews.apache.org/r/35699/).
> 
> From @bmahler:
> > Whoops, Jie just noticed that this isn't correct because 'updatedAllocation' is for the framework only, whereas 'total' and 'available' are for all frameworks on the slave.
> 
> 
> Diffs
> -----
> 
>   src/master/allocator/mesos/hierarchical.hpp ef18ff850addfb5ce3500ed28e8ffd801e2d24eb 
>   src/tests/hierarchical_allocator_tests.cpp 3258840135290cd008ca09235d18b7f093dafd2e 
> 
> Diff: https://reviews.apache.org/r/35816/diff/
> 
> 
> Testing
> -------
> 
> (1) Added a test `HierarchicalAllocatorTest.UpdateAllocationMultipleFrameworks` which breaks the previous `CHECK_EQ` condition.
> (2) `make check`
> 
> 
> Thanks,
> 
> Michael Park
> 
>

Re: Review Request 35816: Fixed the incorrect CHECK_EQ in updateAllocation.

Posted by Ben Mahler <be...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/35816/#review89295
-----------------------------------------------------------


Thanks Michael! Did you see jie's comment on [MESOS-1187](https://issues.apache.org/jira/browse/MESOS-1187)? Looks like that will make these kinds of checks problematic.

Also, it's going to be possible for the slave to be over-allocated due to oversubscription (currently the total from addSlave doesn't include the oversubpription resources, and in the future we might want to express over-allocation, see: [MESOS-2930](https://issues.apache.org/jira/browse/MESOS-2930).

- Ben Mahler


On June 24, 2015, 3:35 p.m., Michael Park wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/35816/
> -----------------------------------------------------------
> 
> (Updated June 24, 2015, 3:35 p.m.)
> 
> 
> Review request for mesos, Ben Mahler and Jie Yu.
> 
> 
> Repository: mesos
> 
> 
> Description
> -------
> 
> Fixed the incorrect `CHECK_EQ` introduced in [r35699](https://reviews.apache.org/r/35699/).
> 
> From @bmahler:
> > Whoops, Jie just noticed that this isn't correct because 'updatedAllocation' is for the framework only, whereas 'total' and 'available' are for all frameworks on the slave.
> 
> 
> Diffs
> -----
> 
>   src/master/allocator/mesos/hierarchical.hpp ef18ff850addfb5ce3500ed28e8ffd801e2d24eb 
>   src/tests/hierarchical_allocator_tests.cpp 3258840135290cd008ca09235d18b7f093dafd2e 
> 
> Diff: https://reviews.apache.org/r/35816/diff/
> 
> 
> Testing
> -------
> 
> (1) Added a test `HierarchicalAllocatorTest.UpdateAllocationMultipleFrameworks` which breaks the previous `CHECK_EQ` condition.
> (2) `make check`
> 
> 
> Thanks,
> 
> Michael Park
> 
>