You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@mesos.apache.org by Chun-Hung Hsiao <ch...@apache.org> on 2018/07/06 23:27:38 UTC
Re: Review Request 67669: Allow resources to be removed when updating
the sorter.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67669/
-----------------------------------------------------------
(Updated July 6, 2018, 11:27 p.m.)
Review request for mesos, Benjamin Bannier, Benjamin Mahler, Jie Yu, and Meng Zhu.
Changes
-------
Keep the check but only relex it to support MESOS-8825.
Summary (updated)
-----------------
Allow resources to be removed when updating the sorter.
Bugs: MESOS-8995
https://issues.apache.org/jira/browse/MESOS-8995
Repository: mesos
Description (updated)
-------
This patch allows the allocator to remove resources when calling
`Sorter::update`, so Mesos can remove resources that are no longer
available through resource conversions.
An example usage is that we can make `DESTROY_VOLUME` to convert a
volume with a stale profile to an empty resource, so the disk space
will disappear instead of being offered out to other frameworks.
Diffs (updated)
-----
CHANGELOG d0cc0e0a4d2a7aa69cadc21923c7af397ebcbb25
docs/upgrades.md cc8a251945af27a432dbeb5585b998cd84e35b22
src/master/allocator/mesos/hierarchical.cpp 5a6cd3d2fc5bdbaaee2d05b9be9e83d4107c749b
src/master/allocator/sorter/drf/sorter.cpp 755a36ead42a91053c881fc46dc6acb39c819cac
src/master/allocator/sorter/sorter.hpp e5e86513dc4023d9c471753897635923c6c9203f
src/tests/sorter_tests.cpp 266a9e72597b3aadc9756f61736c0adf0b1a5831
Diff: https://reviews.apache.org/r/67669/diff/3/
Changes: https://reviews.apache.org/r/67669/diff/2-3/
Testing
-------
sudo make check
Thanks,
Chun-Hung Hsiao
Re: Review Request 67669: Allowed resources to be removed when
updating the sorter.
Posted by Jie Yu <yu...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67669/#review206068
-----------------------------------------------------------
Ship it!
Ship It!
- Jie Yu
On July 10, 2018, 8:31 p.m., Chun-Hung Hsiao wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/67669/
> -----------------------------------------------------------
>
> (Updated July 10, 2018, 8:31 p.m.)
>
>
> Review request for mesos, Benjamin Bannier, Benjamin Mahler, Jie Yu, and Meng Zhu.
>
>
> Bugs: MESOS-9015
> https://issues.apache.org/jira/browse/MESOS-9015
>
>
> Repository: mesos
>
>
> Description
> -------
>
> This patch allows the allocator to remove resources when calling
> `Sorter::update`, so Mesos can remove resources that are no longer
> available through resource conversions.
>
> An example usage is that we can make `DESTROY_VOLUME` to convert a
> volume with a stale profile to an empty resource, so the disk space
> will disappear instead of being offered out to other frameworks.
>
>
> Diffs
> -----
>
> CHANGELOG 4aef45a28c2127587b003dc0a5e2855ac108e6ff
> docs/upgrades.md ef89a7250fba002b0b41030f19c78268190d2671
> src/master/allocator/sorter/drf/sorter.cpp 755a36ead42a91053c881fc46dc6acb39c819cac
> src/master/allocator/sorter/sorter.hpp e5e86513dc4023d9c471753897635923c6c9203f
> src/tests/sorter_tests.cpp 266a9e72597b3aadc9756f61736c0adf0b1a5831
>
>
> Diff: https://reviews.apache.org/r/67669/diff/5/
>
>
> Testing
> -------
>
> sudo make check
>
>
> Thanks,
>
> Chun-Hung Hsiao
>
>
Re: Review Request 67669: Allowed resources to be removed when
updating the sorter.
Posted by Chun-Hung Hsiao <ch...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67669/
-----------------------------------------------------------
(Updated July 10, 2018, 8:31 p.m.)
Review request for mesos, Benjamin Bannier, Benjamin Mahler, Jie Yu, and Meng Zhu.
Changes
-------
Addressed Benjamin's comments.
Summary (updated)
-----------------
Allowed resources to be removed when updating the sorter.
Bugs: MESOS-9015
https://issues.apache.org/jira/browse/MESOS-9015
Repository: mesos
Description
-------
This patch allows the allocator to remove resources when calling
`Sorter::update`, so Mesos can remove resources that are no longer
available through resource conversions.
An example usage is that we can make `DESTROY_VOLUME` to convert a
volume with a stale profile to an empty resource, so the disk space
will disappear instead of being offered out to other frameworks.
Diffs (updated)
-----
CHANGELOG 4aef45a28c2127587b003dc0a5e2855ac108e6ff
docs/upgrades.md ef89a7250fba002b0b41030f19c78268190d2671
src/master/allocator/sorter/drf/sorter.cpp 755a36ead42a91053c881fc46dc6acb39c819cac
src/master/allocator/sorter/sorter.hpp e5e86513dc4023d9c471753897635923c6c9203f
src/tests/sorter_tests.cpp 266a9e72597b3aadc9756f61736c0adf0b1a5831
Diff: https://reviews.apache.org/r/67669/diff/5/
Changes: https://reviews.apache.org/r/67669/diff/4-5/
Testing
-------
sudo make check
Thanks,
Chun-Hung Hsiao
Re: Review Request 67669: Allow resources to be removed when updating
the sorter.
Posted by Chun-Hung Hsiao <ch...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67669/
-----------------------------------------------------------
(Updated July 10, 2018, 5:58 p.m.)
Review request for mesos, Benjamin Bannier, Benjamin Mahler, Jie Yu, and Meng Zhu.
Bugs: MESOS-9015
https://issues.apache.org/jira/browse/MESOS-9015
Repository: mesos
Description
-------
This patch allows the allocator to remove resources when calling
`Sorter::update`, so Mesos can remove resources that are no longer
available through resource conversions.
An example usage is that we can make `DESTROY_VOLUME` to convert a
volume with a stale profile to an empty resource, so the disk space
will disappear instead of being offered out to other frameworks.
Diffs
-----
CHANGELOG 4aef45a28c2127587b003dc0a5e2855ac108e6ff
docs/upgrades.md ef89a7250fba002b0b41030f19c78268190d2671
src/master/allocator/mesos/hierarchical.cpp 5a6cd3d2fc5bdbaaee2d05b9be9e83d4107c749b
src/master/allocator/sorter/drf/sorter.cpp 755a36ead42a91053c881fc46dc6acb39c819cac
src/master/allocator/sorter/sorter.hpp e5e86513dc4023d9c471753897635923c6c9203f
src/tests/sorter_tests.cpp 266a9e72597b3aadc9756f61736c0adf0b1a5831
Diff: https://reviews.apache.org/r/67669/diff/4/
Testing
-------
sudo make check
Thanks,
Chun-Hung Hsiao
Re: Review Request 67669: Allow resources to be removed when updating
the sorter.
Posted by Chun-Hung Hsiao <ch...@apache.org>.
> On July 10, 2018, 3:33 p.m., Benjamin Bannier wrote:
> > docs/upgrades.md
> > Lines 469 (patched)
> > <https://reviews.apache.org/r/67669/diff/4/?file=2056894#file2056894line469>
> >
> > I don't think we do need to mention any operations. What about e.g.,
> >
> > > The semantics of `Sorter::update` has been changed so that resources can be removed certain agents in the pool without removing the full agent. Users are expected to update the total resources as well by e.g., removing the old and adding back the new resources for an agent.
Updated the text with some adjustments.
> On July 10, 2018, 3:33 p.m., Benjamin Bannier wrote:
> > src/master/allocator/mesos/hierarchical.cpp
> > Lines 887 (patched)
> > <https://reviews.apache.org/r/67669/diff/4/?file=2056895#file2056895line887>
> >
> > Why are changes to the allocator in this file? Should they go somewhere else where we enable a feature or explicitly fix a bug? We should probably also add a test.
>
> Chun-Hung Hsiao wrote:
> We don't change the behavior of the allocator. However, the original CHECK prohibited any resource removal, so we need to change the CHECK. The newly introduced variable is just for rewriting the new CHECK.
>
> Since this is a CHECK, it would not be easy to test the violation. But I'll add an allocator test to make sure that the CHECK passes when we reduce resources.
Moved the allocator changes to r/67876.
> On July 10, 2018, 3:33 p.m., Benjamin Bannier wrote:
> > src/master/allocator/mesos/hierarchical.cpp
> > Lines 924-926 (original), 934-939 (patched)
> > <https://reviews.apache.org/r/67669/diff/4/?file=2056895#file2056895line936>
> >
> > I believe we do not need the potentially expensive calls to `toUnreserved` here as `createStrippedScalarQuantity` creates unreserved resources.
>
> Chun-Hung Hsiao wrote:
> You're right. The `createStrippedScalarQuantity()` implementation is changed recently. Previously we keep the static reservation and thus the additional `toUnreserved()` call.
Moved the allocator changes to r/67876.
- Chun-Hung
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67669/#review205911
-----------------------------------------------------------
On July 10, 2018, 5:58 p.m., Chun-Hung Hsiao wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/67669/
> -----------------------------------------------------------
>
> (Updated July 10, 2018, 5:58 p.m.)
>
>
> Review request for mesos, Benjamin Bannier, Benjamin Mahler, Jie Yu, and Meng Zhu.
>
>
> Bugs: MESOS-9015
> https://issues.apache.org/jira/browse/MESOS-9015
>
>
> Repository: mesos
>
>
> Description
> -------
>
> This patch allows the allocator to remove resources when calling
> `Sorter::update`, so Mesos can remove resources that are no longer
> available through resource conversions.
>
> An example usage is that we can make `DESTROY_VOLUME` to convert a
> volume with a stale profile to an empty resource, so the disk space
> will disappear instead of being offered out to other frameworks.
>
>
> Diffs
> -----
>
> CHANGELOG 4aef45a28c2127587b003dc0a5e2855ac108e6ff
> docs/upgrades.md ef89a7250fba002b0b41030f19c78268190d2671
> src/master/allocator/mesos/hierarchical.cpp 5a6cd3d2fc5bdbaaee2d05b9be9e83d4107c749b
> src/master/allocator/sorter/drf/sorter.cpp 755a36ead42a91053c881fc46dc6acb39c819cac
> src/master/allocator/sorter/sorter.hpp e5e86513dc4023d9c471753897635923c6c9203f
> src/tests/sorter_tests.cpp 266a9e72597b3aadc9756f61736c0adf0b1a5831
>
>
> Diff: https://reviews.apache.org/r/67669/diff/4/
>
>
> Testing
> -------
>
> sudo make check
>
>
> Thanks,
>
> Chun-Hung Hsiao
>
>
Re: Review Request 67669: Allow resources to be removed when updating
the sorter.
Posted by Chun-Hung Hsiao <ch...@apache.org>.
> On July 10, 2018, 3:33 p.m., Benjamin Bannier wrote:
> > src/master/allocator/mesos/hierarchical.cpp
> > Lines 887 (patched)
> > <https://reviews.apache.org/r/67669/diff/4/?file=2056895#file2056895line887>
> >
> > Why are changes to the allocator in this file? Should they go somewhere else where we enable a feature or explicitly fix a bug? We should probably also add a test.
We don't change the behavior of the allocator. However, the original CHECK prohibited any resource removal, so we need to change the CHECK. The newly introduced variable is just for rewriting the new CHECK.
Since this is a CHECK, it would not be easy to test the violation. But I'll add an allocator test to make sure that the CHECK passes when we reduce resources.
- Chun-Hung
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67669/#review205911
-----------------------------------------------------------
On July 10, 2018, 5:58 p.m., Chun-Hung Hsiao wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/67669/
> -----------------------------------------------------------
>
> (Updated July 10, 2018, 5:58 p.m.)
>
>
> Review request for mesos, Benjamin Bannier, Benjamin Mahler, Jie Yu, and Meng Zhu.
>
>
> Bugs: MESOS-9015
> https://issues.apache.org/jira/browse/MESOS-9015
>
>
> Repository: mesos
>
>
> Description
> -------
>
> This patch allows the allocator to remove resources when calling
> `Sorter::update`, so Mesos can remove resources that are no longer
> available through resource conversions.
>
> An example usage is that we can make `DESTROY_VOLUME` to convert a
> volume with a stale profile to an empty resource, so the disk space
> will disappear instead of being offered out to other frameworks.
>
>
> Diffs
> -----
>
> CHANGELOG 4aef45a28c2127587b003dc0a5e2855ac108e6ff
> docs/upgrades.md ef89a7250fba002b0b41030f19c78268190d2671
> src/master/allocator/mesos/hierarchical.cpp 5a6cd3d2fc5bdbaaee2d05b9be9e83d4107c749b
> src/master/allocator/sorter/drf/sorter.cpp 755a36ead42a91053c881fc46dc6acb39c819cac
> src/master/allocator/sorter/sorter.hpp e5e86513dc4023d9c471753897635923c6c9203f
> src/tests/sorter_tests.cpp 266a9e72597b3aadc9756f61736c0adf0b1a5831
>
>
> Diff: https://reviews.apache.org/r/67669/diff/4/
>
>
> Testing
> -------
>
> sudo make check
>
>
> Thanks,
>
> Chun-Hung Hsiao
>
>
Re: Review Request 67669: Allow resources to be removed when updating
the sorter.
Posted by Chun-Hung Hsiao <ch...@apache.org>.
> On July 10, 2018, 3:33 p.m., Benjamin Bannier wrote:
> > src/tests/sorter_tests.cpp
> > Lines 1317-1318 (patched)
> > <https://reviews.apache.org/r/67669/diff/4/?file=2056898#file2056898line1317>
> >
> > Not sure we really need two resource kinds here. For me it just seems to make this test harder without testing anything interesting.
> >
> > I'll leave this up to you.
Good point. I'll update it.
- Chun-Hung
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67669/#review205911
-----------------------------------------------------------
On July 10, 2018, 5:58 p.m., Chun-Hung Hsiao wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/67669/
> -----------------------------------------------------------
>
> (Updated July 10, 2018, 5:58 p.m.)
>
>
> Review request for mesos, Benjamin Bannier, Benjamin Mahler, Jie Yu, and Meng Zhu.
>
>
> Bugs: MESOS-9015
> https://issues.apache.org/jira/browse/MESOS-9015
>
>
> Repository: mesos
>
>
> Description
> -------
>
> This patch allows the allocator to remove resources when calling
> `Sorter::update`, so Mesos can remove resources that are no longer
> available through resource conversions.
>
> An example usage is that we can make `DESTROY_VOLUME` to convert a
> volume with a stale profile to an empty resource, so the disk space
> will disappear instead of being offered out to other frameworks.
>
>
> Diffs
> -----
>
> CHANGELOG 4aef45a28c2127587b003dc0a5e2855ac108e6ff
> docs/upgrades.md ef89a7250fba002b0b41030f19c78268190d2671
> src/master/allocator/mesos/hierarchical.cpp 5a6cd3d2fc5bdbaaee2d05b9be9e83d4107c749b
> src/master/allocator/sorter/drf/sorter.cpp 755a36ead42a91053c881fc46dc6acb39c819cac
> src/master/allocator/sorter/sorter.hpp e5e86513dc4023d9c471753897635923c6c9203f
> src/tests/sorter_tests.cpp 266a9e72597b3aadc9756f61736c0adf0b1a5831
>
>
> Diff: https://reviews.apache.org/r/67669/diff/4/
>
>
> Testing
> -------
>
> sudo make check
>
>
> Thanks,
>
> Chun-Hung Hsiao
>
>
Re: Review Request 67669: Allow resources to be removed when updating
the sorter.
Posted by Chun-Hung Hsiao <ch...@apache.org>.
> On July 10, 2018, 3:33 p.m., Benjamin Bannier wrote:
> > src/master/allocator/mesos/hierarchical.cpp
> > Lines 924-926 (original), 934-939 (patched)
> > <https://reviews.apache.org/r/67669/diff/4/?file=2056895#file2056895line936>
> >
> > I believe we do not need the potentially expensive calls to `toUnreserved` here as `createStrippedScalarQuantity` creates unreserved resources.
You're right. The `createStrippedScalarQuantity()` implementation is changed recently. Previously we keep the static reservation and thus the additional `toUnreserved()` call.
- Chun-Hung
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67669/#review205911
-----------------------------------------------------------
On July 10, 2018, 5:58 p.m., Chun-Hung Hsiao wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/67669/
> -----------------------------------------------------------
>
> (Updated July 10, 2018, 5:58 p.m.)
>
>
> Review request for mesos, Benjamin Bannier, Benjamin Mahler, Jie Yu, and Meng Zhu.
>
>
> Bugs: MESOS-9015
> https://issues.apache.org/jira/browse/MESOS-9015
>
>
> Repository: mesos
>
>
> Description
> -------
>
> This patch allows the allocator to remove resources when calling
> `Sorter::update`, so Mesos can remove resources that are no longer
> available through resource conversions.
>
> An example usage is that we can make `DESTROY_VOLUME` to convert a
> volume with a stale profile to an empty resource, so the disk space
> will disappear instead of being offered out to other frameworks.
>
>
> Diffs
> -----
>
> CHANGELOG 4aef45a28c2127587b003dc0a5e2855ac108e6ff
> docs/upgrades.md ef89a7250fba002b0b41030f19c78268190d2671
> src/master/allocator/mesos/hierarchical.cpp 5a6cd3d2fc5bdbaaee2d05b9be9e83d4107c749b
> src/master/allocator/sorter/drf/sorter.cpp 755a36ead42a91053c881fc46dc6acb39c819cac
> src/master/allocator/sorter/sorter.hpp e5e86513dc4023d9c471753897635923c6c9203f
> src/tests/sorter_tests.cpp 266a9e72597b3aadc9756f61736c0adf0b1a5831
>
>
> Diff: https://reviews.apache.org/r/67669/diff/4/
>
>
> Testing
> -------
>
> sudo make check
>
>
> Thanks,
>
> Chun-Hung Hsiao
>
>
Re: Review Request 67669: Allow resources to be removed when updating
the sorter.
Posted by Benjamin Bannier <be...@mesosphere.io>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67669/#review205911
-----------------------------------------------------------
CHANGELOG
Lines 28 (patched)
<https://reviews.apache.org/r/67669/#comment288832>
Let's use this issue link for this RR.
docs/upgrades.md
Lines 469 (patched)
<https://reviews.apache.org/r/67669/#comment288833>
I don't think we do need to mention any operations. What about e.g.,
> The semantics of `Sorter::update` has been changed so that resources can be removed certain agents in the pool without removing the full agent. Users are expected to update the total resources as well by e.g., removing the old and adding back the new resources for an agent.
src/master/allocator/mesos/hierarchical.cpp
Lines 887 (patched)
<https://reviews.apache.org/r/67669/#comment288837>
Why are changes to the allocator in this file? Should they go somewhere else where we enable a feature or explicitly fix a bug? We should probably also add a test.
src/master/allocator/mesos/hierarchical.cpp
Lines 924-926 (original), 934-939 (patched)
<https://reviews.apache.org/r/67669/#comment288835>
I believe we do not need the potentially expensive calls to `toUnreserved` here as `createStrippedScalarQuantity` creates unreserved resources.
src/tests/sorter_tests.cpp
Lines 1317-1318 (patched)
<https://reviews.apache.org/r/67669/#comment288836>
Not sure we really need two resource kinds here. For me it just seems to make this test harder without testing anything interesting.
I'll leave this up to you.
- Benjamin Bannier
On July 10, 2018, 5:39 a.m., Chun-Hung Hsiao wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/67669/
> -----------------------------------------------------------
>
> (Updated July 10, 2018, 5:39 a.m.)
>
>
> Review request for mesos, Benjamin Bannier, Benjamin Mahler, Jie Yu, and Meng Zhu.
>
>
> Bugs: MESOS-8995
> https://issues.apache.org/jira/browse/MESOS-8995
>
>
> Repository: mesos
>
>
> Description
> -------
>
> This patch allows the allocator to remove resources when calling
> `Sorter::update`, so Mesos can remove resources that are no longer
> available through resource conversions.
>
> An example usage is that we can make `DESTROY_VOLUME` to convert a
> volume with a stale profile to an empty resource, so the disk space
> will disappear instead of being offered out to other frameworks.
>
>
> Diffs
> -----
>
> CHANGELOG 4aef45a28c2127587b003dc0a5e2855ac108e6ff
> docs/upgrades.md ef89a7250fba002b0b41030f19c78268190d2671
> src/master/allocator/mesos/hierarchical.cpp 5a6cd3d2fc5bdbaaee2d05b9be9e83d4107c749b
> src/master/allocator/sorter/drf/sorter.cpp 755a36ead42a91053c881fc46dc6acb39c819cac
> src/master/allocator/sorter/sorter.hpp e5e86513dc4023d9c471753897635923c6c9203f
> src/tests/sorter_tests.cpp 266a9e72597b3aadc9756f61736c0adf0b1a5831
>
>
> Diff: https://reviews.apache.org/r/67669/diff/4/
>
>
> Testing
> -------
>
> sudo make check
>
>
> Thanks,
>
> Chun-Hung Hsiao
>
>
Re: Review Request 67669: Allow resources to be removed when updating
the sorter.
Posted by Chun-Hung Hsiao <ch...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67669/
-----------------------------------------------------------
(Updated July 10, 2018, 3:39 a.m.)
Review request for mesos, Benjamin Bannier, Benjamin Mahler, Jie Yu, and Meng Zhu.
Changes
-------
Fixed a bug in the test.
Bugs: MESOS-8995
https://issues.apache.org/jira/browse/MESOS-8995
Repository: mesos
Description
-------
This patch allows the allocator to remove resources when calling
`Sorter::update`, so Mesos can remove resources that are no longer
available through resource conversions.
An example usage is that we can make `DESTROY_VOLUME` to convert a
volume with a stale profile to an empty resource, so the disk space
will disappear instead of being offered out to other frameworks.
Diffs (updated)
-----
CHANGELOG 4aef45a28c2127587b003dc0a5e2855ac108e6ff
docs/upgrades.md ef89a7250fba002b0b41030f19c78268190d2671
src/master/allocator/mesos/hierarchical.cpp 5a6cd3d2fc5bdbaaee2d05b9be9e83d4107c749b
src/master/allocator/sorter/drf/sorter.cpp 755a36ead42a91053c881fc46dc6acb39c819cac
src/master/allocator/sorter/sorter.hpp e5e86513dc4023d9c471753897635923c6c9203f
src/tests/sorter_tests.cpp 266a9e72597b3aadc9756f61736c0adf0b1a5831
Diff: https://reviews.apache.org/r/67669/diff/4/
Changes: https://reviews.apache.org/r/67669/diff/3-4/
Testing
-------
sudo make check
Thanks,
Chun-Hung Hsiao