You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@mesos.apache.org by Ben Mahler <be...@gmail.com> on 2014/12/16 07:28:13 UTC
Review Request 29083: Updated Sorter to allow transforming allocated
resources.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/29083/
-----------------------------------------------------------
Review request for mesos and Jie Yu.
Bugs: MESOS-2099
https://issues.apache.org/jira/browse/MESOS-2099
Repository: mesos-git
Description
-------
In order for frameworks to perform resource transformations on their allocations, the allocator must be able to update the allocation in the sorter objects.
Note that the sorters cannot simply take a Transformation, as done in r/29084/. This is because sorters may not store the complete allocation for a client (e.g. role sorter doesn't track reserved resources).
This means that taking a Transformation is problematic for the role sorter in the allocator.
Diffs
-----
src/master/drf_sorter.hpp c47b56afdb8f9f3ff0793f0d10a4e4c656e6a212
src/master/drf_sorter.cpp 0ad6c52759e1ceec4b43f99d8fa1e75d7e2e1c31
src/master/sorter.hpp 0150f90b9aab2ecb4edd07dca7f0d58997e422aa
src/tests/sorter_tests.cpp 0516ab573d9f4f2af249978e15ebf52c2afb5359
Diff: https://reviews.apache.org/r/29083/diff/
Testing
-------
Added a test.
Thanks,
Ben Mahler
Re: Review Request 29083: Updated Sorter to allow transforming
allocated resources.
Posted by Ben Mahler <be...@gmail.com>.
> On Dec. 16, 2014, 5:23 p.m., Dominic Hamon wrote:
> > src/master/drf_sorter.cpp, line 125
> > <https://reviews.apache.org/r/29083/diff/1/?file=792872#file792872line125>
> >
> > const_iterator
This was originally copied from the other code in here, but now it's not relevant anymore.
- Ben
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/29083/#review65210
-----------------------------------------------------------
On Dec. 16, 2014, 6:28 a.m., Ben Mahler wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/29083/
> -----------------------------------------------------------
>
> (Updated Dec. 16, 2014, 6:28 a.m.)
>
>
> Review request for mesos and Jie Yu.
>
>
> Bugs: MESOS-2099
> https://issues.apache.org/jira/browse/MESOS-2099
>
>
> Repository: mesos-git
>
>
> Description
> -------
>
> In order for frameworks to perform resource transformations on their allocations, the allocator must be able to update the allocation in the sorter objects.
>
> Note that the sorters cannot simply take a Transformation, as done in r/29084/. This is because sorters may not store the complete allocation for a client (e.g. role sorter doesn't track reserved resources).
>
> This means that taking a Transformation is problematic for the role sorter in the allocator.
>
>
> Diffs
> -----
>
> src/master/drf_sorter.hpp c47b56afdb8f9f3ff0793f0d10a4e4c656e6a212
> src/master/drf_sorter.cpp 0ad6c52759e1ceec4b43f99d8fa1e75d7e2e1c31
> src/master/sorter.hpp 0150f90b9aab2ecb4edd07dca7f0d58997e422aa
> src/tests/sorter_tests.cpp 0516ab573d9f4f2af249978e15ebf52c2afb5359
>
> Diff: https://reviews.apache.org/r/29083/diff/
>
>
> Testing
> -------
>
> Added a test.
>
>
> Thanks,
>
> Ben Mahler
>
>
Re: Review Request 29083: Updated Sorter to allow transforming
allocated resources.
Posted by Dominic Hamon <dh...@twopensource.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/29083/#review65210
-----------------------------------------------------------
src/master/drf_sorter.cpp
<https://reviews.apache.org/r/29083/#comment108240>
const_iterator
- Dominic Hamon
On Dec. 15, 2014, 10:28 p.m., Ben Mahler wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/29083/
> -----------------------------------------------------------
>
> (Updated Dec. 15, 2014, 10:28 p.m.)
>
>
> Review request for mesos and Jie Yu.
>
>
> Bugs: MESOS-2099
> https://issues.apache.org/jira/browse/MESOS-2099
>
>
> Repository: mesos-git
>
>
> Description
> -------
>
> In order for frameworks to perform resource transformations on their allocations, the allocator must be able to update the allocation in the sorter objects.
>
> Note that the sorters cannot simply take a Transformation, as done in r/29084/. This is because sorters may not store the complete allocation for a client (e.g. role sorter doesn't track reserved resources).
>
> This means that taking a Transformation is problematic for the role sorter in the allocator.
>
>
> Diffs
> -----
>
> src/master/drf_sorter.hpp c47b56afdb8f9f3ff0793f0d10a4e4c656e6a212
> src/master/drf_sorter.cpp 0ad6c52759e1ceec4b43f99d8fa1e75d7e2e1c31
> src/master/sorter.hpp 0150f90b9aab2ecb4edd07dca7f0d58997e422aa
> src/tests/sorter_tests.cpp 0516ab573d9f4f2af249978e15ebf52c2afb5359
>
> Diff: https://reviews.apache.org/r/29083/diff/
>
>
> Testing
> -------
>
> Added a test.
>
>
> Thanks,
>
> Ben Mahler
>
>
Re: Review Request 29083: Updated Sorter to allow transforming
allocated resources.
Posted by Jie Yu <yu...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/29083/#review65394
-----------------------------------------------------------
Ship it!
Ship It!
- Jie Yu
On Dec. 17, 2014, 11:20 p.m., Ben Mahler wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/29083/
> -----------------------------------------------------------
>
> (Updated Dec. 17, 2014, 11:20 p.m.)
>
>
> Review request for mesos and Jie Yu.
>
>
> Bugs: MESOS-2099
> https://issues.apache.org/jira/browse/MESOS-2099
>
>
> Repository: mesos-git
>
>
> Description
> -------
>
> In order for frameworks to perform resource transformations on their allocations, the allocator must be able to update the allocation in the sorter objects.
>
> Note that the sorters cannot simply take a Transformation, as done in r/29084/. This is because sorters may not store the complete allocation for a client (e.g. role sorter doesn't track reserved resources).
>
> This means that taking a Transformation is problematic for the role sorter in the allocator.
>
>
> Diffs
> -----
>
> src/master/drf_sorter.hpp c47b56afdb8f9f3ff0793f0d10a4e4c656e6a212
> src/master/drf_sorter.cpp ebc0284170750ee3860e89931a96a2330c795fbe
> src/master/sorter.hpp 0150f90b9aab2ecb4edd07dca7f0d58997e422aa
> src/tests/sorter_tests.cpp c2f4aa18db986d12d8364fe5ca3c0dc1b8f98040
>
> Diff: https://reviews.apache.org/r/29083/diff/
>
>
> Testing
> -------
>
> Added a test.
>
>
> Thanks,
>
> Ben Mahler
>
>
Re: Review Request 29083: Updated Sorter to allow transforming
allocated resources.
Posted by Ben Mahler <be...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/29083/
-----------------------------------------------------------
(Updated Dec. 17, 2014, 11:20 p.m.)
Review request for mesos and Jie Yu.
Changes
-------
Fixed an issue from jie's review.
Bugs: MESOS-2099
https://issues.apache.org/jira/browse/MESOS-2099
Repository: mesos-git
Description
-------
In order for frameworks to perform resource transformations on their allocations, the allocator must be able to update the allocation in the sorter objects.
Note that the sorters cannot simply take a Transformation, as done in r/29084/. This is because sorters may not store the complete allocation for a client (e.g. role sorter doesn't track reserved resources).
This means that taking a Transformation is problematic for the role sorter in the allocator.
Diffs (updated)
-----
src/master/drf_sorter.hpp c47b56afdb8f9f3ff0793f0d10a4e4c656e6a212
src/master/drf_sorter.cpp ebc0284170750ee3860e89931a96a2330c795fbe
src/master/sorter.hpp 0150f90b9aab2ecb4edd07dca7f0d58997e422aa
src/tests/sorter_tests.cpp c2f4aa18db986d12d8364fe5ca3c0dc1b8f98040
Diff: https://reviews.apache.org/r/29083/diff/
Testing
-------
Added a test.
Thanks,
Ben Mahler
Re: Review Request 29083: Updated Sorter to allow transforming
allocated resources.
Posted by Ben Mahler <be...@gmail.com>.
> On Dec. 17, 2014, 1 a.m., Ben Mahler wrote:
> > src/master/drf_sorter.hpp, lines 79-81
> > <https://reviews.apache.org/r/29083/diff/1/?file=792871#file792871line79>
> >
> > Jie and I were chatting and realized that since the sorter aggregates resources across slaves (which breaks for non-scalar things like ranges and sets), the disk transformation would have issues with duplicated IDs here.
> >
> > Jie is updating the transformation to be simpler and non-idempotent which will fix the issue here (can have duplicate persistence IDs in the disk resources), but ideally we stop merging resources across slaves.
> >
> > At the very least a TODO is needed here!
I've added a TODO in sorter.hpp for storing resources grouped by SlaveID.
- Ben
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/29083/#review65267
-----------------------------------------------------------
On Dec. 16, 2014, 6:28 a.m., Ben Mahler wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/29083/
> -----------------------------------------------------------
>
> (Updated Dec. 16, 2014, 6:28 a.m.)
>
>
> Review request for mesos and Jie Yu.
>
>
> Bugs: MESOS-2099
> https://issues.apache.org/jira/browse/MESOS-2099
>
>
> Repository: mesos-git
>
>
> Description
> -------
>
> In order for frameworks to perform resource transformations on their allocations, the allocator must be able to update the allocation in the sorter objects.
>
> Note that the sorters cannot simply take a Transformation, as done in r/29084/. This is because sorters may not store the complete allocation for a client (e.g. role sorter doesn't track reserved resources).
>
> This means that taking a Transformation is problematic for the role sorter in the allocator.
>
>
> Diffs
> -----
>
> src/master/drf_sorter.hpp c47b56afdb8f9f3ff0793f0d10a4e4c656e6a212
> src/master/drf_sorter.cpp 0ad6c52759e1ceec4b43f99d8fa1e75d7e2e1c31
> src/master/sorter.hpp 0150f90b9aab2ecb4edd07dca7f0d58997e422aa
> src/tests/sorter_tests.cpp 0516ab573d9f4f2af249978e15ebf52c2afb5359
>
> Diff: https://reviews.apache.org/r/29083/diff/
>
>
> Testing
> -------
>
> Added a test.
>
>
> Thanks,
>
> Ben Mahler
>
>
Re: Review Request 29083: Updated Sorter to allow transforming
allocated resources.
Posted by Ben Mahler <be...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/29083/#review65267
-----------------------------------------------------------
src/master/drf_sorter.hpp
<https://reviews.apache.org/r/29083/#comment108345>
Jie and I were chatting and realized that since the sorter aggregates resources across slaves (which breaks for non-scalar things like ranges and sets), the disk transformation would have issues with duplicated IDs here.
Jie is updating the transformation to be simpler and non-idempotent which will fix the issue here (can have duplicate persistence IDs in the disk resources), but ideally we stop merging resources across slaves.
At the very least a TODO is needed here!
- Ben Mahler
On Dec. 16, 2014, 6:28 a.m., Ben Mahler wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/29083/
> -----------------------------------------------------------
>
> (Updated Dec. 16, 2014, 6:28 a.m.)
>
>
> Review request for mesos and Jie Yu.
>
>
> Bugs: MESOS-2099
> https://issues.apache.org/jira/browse/MESOS-2099
>
>
> Repository: mesos-git
>
>
> Description
> -------
>
> In order for frameworks to perform resource transformations on their allocations, the allocator must be able to update the allocation in the sorter objects.
>
> Note that the sorters cannot simply take a Transformation, as done in r/29084/. This is because sorters may not store the complete allocation for a client (e.g. role sorter doesn't track reserved resources).
>
> This means that taking a Transformation is problematic for the role sorter in the allocator.
>
>
> Diffs
> -----
>
> src/master/drf_sorter.hpp c47b56afdb8f9f3ff0793f0d10a4e4c656e6a212
> src/master/drf_sorter.cpp 0ad6c52759e1ceec4b43f99d8fa1e75d7e2e1c31
> src/master/sorter.hpp 0150f90b9aab2ecb4edd07dca7f0d58997e422aa
> src/tests/sorter_tests.cpp 0516ab573d9f4f2af249978e15ebf52c2afb5359
>
> Diff: https://reviews.apache.org/r/29083/diff/
>
>
> Testing
> -------
>
> Added a test.
>
>
> Thanks,
>
> Ben Mahler
>
>
Re: Review Request 29083: Updated Sorter to allow transforming
allocated resources.
Posted by Ben Mahler <be...@gmail.com>.
> On Dec. 16, 2014, 10:05 p.m., Jie Yu wrote:
> > src/master/drf_sorter.cpp, line 137
> > <https://reviews.apache.org/r/29083/diff/1/?file=792872#file792872line137>
> >
> > Do you need to update total resources as well?
Thanks! Actually the test I wrote allocated resources before adding them, too bad the code doesn't check that. I fixed the test, but we have some cleanups to do later..!
- Ben
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/29083/#review65248
-----------------------------------------------------------
On Dec. 16, 2014, 6:28 a.m., Ben Mahler wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/29083/
> -----------------------------------------------------------
>
> (Updated Dec. 16, 2014, 6:28 a.m.)
>
>
> Review request for mesos and Jie Yu.
>
>
> Bugs: MESOS-2099
> https://issues.apache.org/jira/browse/MESOS-2099
>
>
> Repository: mesos-git
>
>
> Description
> -------
>
> In order for frameworks to perform resource transformations on their allocations, the allocator must be able to update the allocation in the sorter objects.
>
> Note that the sorters cannot simply take a Transformation, as done in r/29084/. This is because sorters may not store the complete allocation for a client (e.g. role sorter doesn't track reserved resources).
>
> This means that taking a Transformation is problematic for the role sorter in the allocator.
>
>
> Diffs
> -----
>
> src/master/drf_sorter.hpp c47b56afdb8f9f3ff0793f0d10a4e4c656e6a212
> src/master/drf_sorter.cpp 0ad6c52759e1ceec4b43f99d8fa1e75d7e2e1c31
> src/master/sorter.hpp 0150f90b9aab2ecb4edd07dca7f0d58997e422aa
> src/tests/sorter_tests.cpp 0516ab573d9f4f2af249978e15ebf52c2afb5359
>
> Diff: https://reviews.apache.org/r/29083/diff/
>
>
> Testing
> -------
>
> Added a test.
>
>
> Thanks,
>
> Ben Mahler
>
>
Re: Review Request 29083: Updated Sorter to allow transforming
allocated resources.
Posted by Jie Yu <yu...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/29083/#review65248
-----------------------------------------------------------
src/master/drf_sorter.cpp
<https://reviews.apache.org/r/29083/#comment108302>
Do you need to update total resources as well?
- Jie Yu
On Dec. 16, 2014, 6:28 a.m., Ben Mahler wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/29083/
> -----------------------------------------------------------
>
> (Updated Dec. 16, 2014, 6:28 a.m.)
>
>
> Review request for mesos and Jie Yu.
>
>
> Bugs: MESOS-2099
> https://issues.apache.org/jira/browse/MESOS-2099
>
>
> Repository: mesos-git
>
>
> Description
> -------
>
> In order for frameworks to perform resource transformations on their allocations, the allocator must be able to update the allocation in the sorter objects.
>
> Note that the sorters cannot simply take a Transformation, as done in r/29084/. This is because sorters may not store the complete allocation for a client (e.g. role sorter doesn't track reserved resources).
>
> This means that taking a Transformation is problematic for the role sorter in the allocator.
>
>
> Diffs
> -----
>
> src/master/drf_sorter.hpp c47b56afdb8f9f3ff0793f0d10a4e4c656e6a212
> src/master/drf_sorter.cpp 0ad6c52759e1ceec4b43f99d8fa1e75d7e2e1c31
> src/master/sorter.hpp 0150f90b9aab2ecb4edd07dca7f0d58997e422aa
> src/tests/sorter_tests.cpp 0516ab573d9f4f2af249978e15ebf52c2afb5359
>
> Diff: https://reviews.apache.org/r/29083/diff/
>
>
> Testing
> -------
>
> Added a test.
>
>
> Thanks,
>
> Ben Mahler
>
>