You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@mesos.apache.org by Benjamin Mahler <bm...@apache.org> on 2019/08/22 00:31:18 UTC
Review Request 71347: Optimized shrinkResources.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71347/
-----------------------------------------------------------
Review request for mesos, Andrei Sekretenko and Meng Zhu.
Bugs: MESOS-9806
https://issues.apache.org/jira/browse/MESOS-9806
Repository: mesos
Description
-------
There are several optimization used here:
* Avoid unnecessary copying of Resource objects.
* Avoid unnecessary validation.
The shrinkResources function now is a friend of Resources,
in order to perform an in-place shuffle of the vector.
Master branch:
HierarchicalAllocator_WithQuotaParam.LargeAndSmallQuota/2
Made 3500 allocations in 36.1185929secs
Made 0 allocation in 32.62218602secs
After all patches in review chain:
HierarchicalAllocator_WithQuotaParam.LargeAndSmallQuota/2
Made 3500 allocations in 21.389381617secs
Made 0 allocation in 18.593000222secs
Diffs
-----
include/mesos/resource_quantities.hpp c861df8a8a167b19ea8374c22cdd2a8fe567a6a6
include/mesos/resources.hpp a6a052ba51e13e2804eca846f08605e7f0714cfd
include/mesos/v1/resources.hpp e43d1fba69771405f4575cf675d6f0cd13c2c7c9
src/common/resource_quantities.cpp 9577181bc4c05214759332e41f4a7d8b8fb6db1f
src/common/resources.cpp fc7e86ba5eee3deb953d85065a1192374910c5e5
src/common/resources_utils.cpp 720b954b74a5db72438b1846d7df837d6a1fa4a4
src/v1/resources.cpp 88da0a185bd54e7053a221fe0b3255f3c514ac60
Diff: https://reviews.apache.org/r/71347/diff/1/
Testing
-------
make check
Thanks,
Benjamin Mahler
Re: Review Request 71347: Optimized shrinkResources.
Posted by Mesos Reviewbot <re...@mesos.apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71347/#review217401
-----------------------------------------------------------
Patch looks great!
Reviews applied: [71353, 71354, 71347]
Passed command: export OS='ubuntu:14.04' BUILDTOOL='autotools' COMPILER='gcc' CONFIGURATION='--verbose --disable-libtool-wrappers --disable-parallel-test-execution' ENVIRONMENT='GLOG_v=1 MESOS_VERBOSE=1'; ./support/docker-build.sh
- Mesos Reviewbot
On Aug. 22, 2019, 3:34 p.m., Benjamin Mahler wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71347/
> -----------------------------------------------------------
>
> (Updated Aug. 22, 2019, 3:34 p.m.)
>
>
> Review request for mesos, Andrei Sekretenko and Meng Zhu.
>
>
> Bugs: MESOS-9806
> https://issues.apache.org/jira/browse/MESOS-9806
>
>
> Repository: mesos
>
>
> Description
> -------
>
> The shrinkResources function now is a friend of Resources,
> in order to perform an in-place shuffle of the vector.
>
> Master + previous patches:
> *HierarchicalAllocator_WithQuotaParam.LargeAndSmallQuota/2
> Made 3500 allocations in 23.37 secs
> Made 0 allocation in 19.72 secs
>
> Master + previous patches + this patch:
> *HierarchicalAllocator_WithQuotaParam.LargeAndSmallQuota/2
> Made 3500 allocations in 21.71 secs
> Made 0 allocation in 18.49 secs
>
>
> Diffs
> -----
>
> include/mesos/resources.hpp a6a052ba51e13e2804eca846f08605e7f0714cfd
> src/common/resources_utils.cpp 720b954b74a5db72438b1846d7df837d6a1fa4a4
>
>
> Diff: https://reviews.apache.org/r/71347/diff/2/
>
>
> Testing
> -------
>
> make check
>
>
> Thanks,
>
> Benjamin Mahler
>
>
Re: Review Request 71347: Optimized shrinkResources.
Posted by Benjamin Mahler <bm...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71347/
-----------------------------------------------------------
(Updated Aug. 22, 2019, 10:34 p.m.)
Review request for mesos, Andrei Sekretenko and Meng Zhu.
Changes
-------
* Pulled out other optimizations in front of this patch.
Bugs: MESOS-9806
https://issues.apache.org/jira/browse/MESOS-9806
Repository: mesos
Description (updated)
-------
The shrinkResources function now is a friend of Resources,
in order to perform an in-place shuffle of the vector.
Master + previous patches:
*HierarchicalAllocator_WithQuotaParam.LargeAndSmallQuota/2
Made 3500 allocations in 23.37 secs
Made 0 allocation in 19.72 secs
Master + previous patches + this patch:
*HierarchicalAllocator_WithQuotaParam.LargeAndSmallQuota/2
Made 3500 allocations in 21.71 secs
Made 0 allocation in 18.49 secs
Diffs (updated)
-----
include/mesos/resources.hpp a6a052ba51e13e2804eca846f08605e7f0714cfd
src/common/resources_utils.cpp 720b954b74a5db72438b1846d7df837d6a1fa4a4
Diff: https://reviews.apache.org/r/71347/diff/2/
Changes: https://reviews.apache.org/r/71347/diff/1-2/
Testing
-------
make check
Thanks,
Benjamin Mahler
Re: Review Request 71347: Optimized shrinkResources.
Posted by Mesos Reviewbot <re...@mesos.apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71347/#review217374
-----------------------------------------------------------
Patch looks great!
Reviews applied: [71345, 71346, 71347]
Passed command: export OS='ubuntu:14.04' BUILDTOOL='autotools' COMPILER='gcc' CONFIGURATION='--verbose --disable-libtool-wrappers --disable-parallel-test-execution' ENVIRONMENT='GLOG_v=1 MESOS_VERBOSE=1'; ./support/docker-build.sh
- Mesos Reviewbot
On Aug. 22, 2019, 12:31 a.m., Benjamin Mahler wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71347/
> -----------------------------------------------------------
>
> (Updated Aug. 22, 2019, 12:31 a.m.)
>
>
> Review request for mesos, Andrei Sekretenko and Meng Zhu.
>
>
> Bugs: MESOS-9806
> https://issues.apache.org/jira/browse/MESOS-9806
>
>
> Repository: mesos
>
>
> Description
> -------
>
> There are several optimization used here:
>
> * Avoid unnecessary copying of Resource objects.
> * Avoid unnecessary validation.
>
> The shrinkResources function now is a friend of Resources,
> in order to perform an in-place shuffle of the vector.
>
> Master branch:
> HierarchicalAllocator_WithQuotaParam.LargeAndSmallQuota/2
> Made 3500 allocations in 36.1185929secs
> Made 0 allocation in 32.62218602secs
>
> After all patches in review chain:
> HierarchicalAllocator_WithQuotaParam.LargeAndSmallQuota/2
> Made 3500 allocations in 21.389381617secs
> Made 0 allocation in 18.593000222secs
>
>
> Diffs
> -----
>
> include/mesos/resource_quantities.hpp c861df8a8a167b19ea8374c22cdd2a8fe567a6a6
> include/mesos/resources.hpp a6a052ba51e13e2804eca846f08605e7f0714cfd
> include/mesos/v1/resources.hpp e43d1fba69771405f4575cf675d6f0cd13c2c7c9
> src/common/resource_quantities.cpp 9577181bc4c05214759332e41f4a7d8b8fb6db1f
> src/common/resources.cpp fc7e86ba5eee3deb953d85065a1192374910c5e5
> src/common/resources_utils.cpp 720b954b74a5db72438b1846d7df837d6a1fa4a4
> src/v1/resources.cpp 88da0a185bd54e7053a221fe0b3255f3c514ac60
>
>
> Diff: https://reviews.apache.org/r/71347/diff/1/
>
>
> Testing
> -------
>
> make check
>
>
> Thanks,
>
> Benjamin Mahler
>
>
Re: Review Request 71347: Optimized shrinkResources.
Posted by Meng Zhu <mz...@mesosphere.io>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71347/#review217384
-----------------------------------------------------------
src/common/resources.cpp
Line 1324 (original), 1298 (patched)
<https://reviews.apache.org/r/71347/#comment304672>
This worths some comment here? The assumption is currently only some disks resources are deemed indivisible.
src/common/resources_utils.cpp
Line 917 (original), 919-920 (patched)
<https://reviews.apache.org/r/71347/#comment304676>
We can further optimize here to do minimum copying:
Introduce `bool ResourceQuantities::contains(const std::string& name);`
```
Resources result = resources.filter([&target](const Resource& r) {
return target.contains(r.name());
});
```
src/common/resources_utils.cpp
Lines 921-936 (original), 924-965 (patched)
<https://reviews.apache.org/r/71347/#comment304677>
Now, I get the complexity :)
So on the left, we construct the result from scratch by moving resource into it. On the right, we mutate in place.
I am not sure how much saving are there, but it is much less readable comparing the left.
Can you pull this into a separate patch? We can commit the optimization to the singular shrink above and see how this specific change can give.
I am happy to experiment with smaller changes such as the filtering I mentioned above, reserve the vector and other not-so-complex changes.
Ditto below for the limits shrink.
- Meng Zhu
On Aug. 21, 2019, 5:31 p.m., Benjamin Mahler wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71347/
> -----------------------------------------------------------
>
> (Updated Aug. 21, 2019, 5:31 p.m.)
>
>
> Review request for mesos, Andrei Sekretenko and Meng Zhu.
>
>
> Bugs: MESOS-9806
> https://issues.apache.org/jira/browse/MESOS-9806
>
>
> Repository: mesos
>
>
> Description
> -------
>
> There are several optimization used here:
>
> * Avoid unnecessary copying of Resource objects.
> * Avoid unnecessary validation.
>
> The shrinkResources function now is a friend of Resources,
> in order to perform an in-place shuffle of the vector.
>
> 1.8.1:
> HierarchicalAllocator_WithQuotaParam.LargeAndSmallQuota/2
> Made 3500 allocations in 13.684841376secs
> Made 0 allocation in 10.450296377secs
>
> Master branch:
> HierarchicalAllocator_WithQuotaParam.LargeAndSmallQuota/2
> Made 3500 allocations in 36.1185929secs
> Made 0 allocation in 32.62218602secs
>
> After all patches in review chain:
> HierarchicalAllocator_WithQuotaParam.LargeAndSmallQuota/2
> Made 3500 allocations in 21.389381617secs
> Made 0 allocation in 18.593000222secs
>
>
> Diffs
> -----
>
> include/mesos/resource_quantities.hpp c861df8a8a167b19ea8374c22cdd2a8fe567a6a6
> include/mesos/resources.hpp a6a052ba51e13e2804eca846f08605e7f0714cfd
> include/mesos/v1/resources.hpp e43d1fba69771405f4575cf675d6f0cd13c2c7c9
> src/common/resource_quantities.cpp 9577181bc4c05214759332e41f4a7d8b8fb6db1f
> src/common/resources.cpp fc7e86ba5eee3deb953d85065a1192374910c5e5
> src/common/resources_utils.cpp 720b954b74a5db72438b1846d7df837d6a1fa4a4
> src/v1/resources.cpp 88da0a185bd54e7053a221fe0b3255f3c514ac60
>
>
> Diff: https://reviews.apache.org/r/71347/diff/1/
>
>
> Testing
> -------
>
> make check
>
>
> Thanks,
>
> Benjamin Mahler
>
>