You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@mesos.apache.org by Thomas Marshall <tw...@gmail.com> on 2012/06/27 01:41:32 UTC
Review Request: Made DRF per user and log(N)
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/5599/
-----------------------------------------------------------
Review request for mesos and Benjamin Hindman.
Description
-------
Rewrote the DRF algorithm in DominantShareAllocator to calculate shares on a per user, rather than per framework, basis and to store those shares in sorted order so that allocations have log(n) time complexity instead of n^2.
This patch relies on 3 pending code reviews:
https://reviews.apache.org/r/5448/
https://reviews.apache.org/r/5451/
https://reviews.apache.org/r/5532/
This addresses bugs MESOS-225 and MESOS-226.
https://issues.apache.org/jira/browse/MESOS-225
https://issues.apache.org/jira/browse/MESOS-226
Diffs
-----
src/master/dominant_share_allocator.hpp PRE-CREATION
src/master/dominant_share_allocator.cpp PRE-CREATION
Diff: https://reviews.apache.org/r/5599/diff/
Testing
-------
make check on Lion
Thanks,
Thomas Marshall
Re: Review Request: Made DRF per user and log(N)
Posted by Thomas Marshall <tw...@gmail.com>.
> On July 22, 2012, 6:21 a.m., Benjamin Hindman wrote:
> > src/local/local.hpp, line 26
> > <https://reviews.apache.org/r/5599/diff/9/?file=124305#file124305line26>
> >
> > I would just keep the file called allocator.hpp, especially if we want to wrap the AllocatorProcess in an Allocator.
Fixed.
> On July 22, 2012, 6:21 a.m., Benjamin Hindman wrote:
> > src/master/sorter.hpp, line 117
> > <https://reviews.apache.org/r/5599/diff/9/?file=124314#file124314line117>
> >
> > Let's call this 'sort' (the main functionality of the class).
Fixed.
- Thomas
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/5599/#review9261
-----------------------------------------------------------
On July 18, 2012, 6:19 p.m., Thomas Marshall wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/5599/
> -----------------------------------------------------------
>
> (Updated July 18, 2012, 6:19 p.m.)
>
>
> Review request for mesos and Benjamin Hindman.
>
>
> Description
> -------
>
> Rewrote the DRF algorithm in DominantShareAllocator to calculate shares on a per user, rather than per framework, basis and to store those shares in sorted order so that allocations have log(n) time complexity instead of n^2.
>
> Significant restructuring of the allocator code to make writing new allocators easier:
>
> - The Allocator class is renamed AllocatorProcess.
> - A new Allocator class is introduced which knows only about "elements" (either user names or frameworkIds) and how many resources they have been allocated, and then has an interface for retrieving the elements in the order they should be allocated to.
> - SimpleAllocatorProcess which can be used with different Allocators to carry out different allocation policies, including the possibility of having different policies for the per-user and per-framework allocations.
>
> Why is this restructuring a good thing?
> Now that we're doing allocations based on users, rather than frameworks, we have to do two levels of allocation - determine which user to give resources to, then determine which of their frameworks to actually make the offer to. Rather than hard-coding the allocation policies in, the new Allocator class allows us to reuse the same code to do both levels of allocation, and it allows us to easily swap different policies in or mix and match. You can even imagine some day having an allocator that allows each user to pick the per-framework allocation policy they prefer.
>
> Merely giving SimpleAllocatorProcess different Allocators is not sufficient to implement all allocation policies - only those where all resources are considered equal and there is no revocation. But, you can always create new AllocatorProcess subclasses (there will soon be a code review where I implement a static allocator this way) and these new AllocatorProcess classes can still reuse Allocators as appropriate.
>
> It also makes testing the allocator logic a lot easier since we can instantiate Allocators, give them arbitrary elements, and check that they return the right thing, instead of worrying about instantiating slaves/frameworks and using mock objects.
>
>
> This addresses bugs MESOS-225 and MESOS-226.
> https://issues.apache.org/jira/browse/MESOS-225
> https://issues.apache.org/jira/browse/MESOS-226
>
>
> Diffs
> -----
>
> src/Makefile.am 10f1101
> src/local/local.hpp 8ae5b9e
> src/local/local.cpp 46537d7
> src/master/allocator.hpp e55b9ec
> src/master/allocator_process.hpp PRE-CREATION
> src/master/main.cpp 8d38fb1
> src/master/master.hpp ad2ec25
> src/master/master.cpp b5e09d8
> src/master/simple_allocator_process.hpp PRE-CREATION
> src/master/simple_allocator_process.cpp PRE-CREATION
> src/master/sorter.hpp PRE-CREATION
> src/master/sorter.cpp PRE-CREATION
> src/tests/allocator_tests.cpp b3db13d
> src/tests/fault_tolerance_tests.cpp f892282
> src/tests/master_detector_tests.cpp 758c8b9
> src/tests/master_tests.cpp b586984
> src/tests/resource_offers_tests.cpp c004772
> src/tests/slave_tests.cpp e9b25ba
> src/tests/utils.hpp ced8592
>
> Diff: https://reviews.apache.org/r/5599/diff/
>
>
> Testing
> -------
>
> make check on Lion
>
>
> Thanks,
>
> Thomas Marshall
>
>
Re: Review Request: Made DRF per user and log(N)
Posted by Benjamin Hindman <be...@berkeley.edu>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/5599/#review9261
-----------------------------------------------------------
src/local/local.hpp
<https://reviews.apache.org/r/5599/#comment20038>
I would just keep the file called allocator.hpp, especially if we want to wrap the AllocatorProcess in an Allocator.
src/master/master.cpp
<https://reviews.apache.org/r/5599/#comment19861>
Why this change?
src/master/simple_allocator_process.hpp
<https://reviews.apache.org/r/5599/#comment20040>
Please don't use 'using' in header files ... it pollutes other users of the header.
src/master/simple_allocator_process.hpp
<https://reviews.apache.org/r/5599/#comment20041>
Please preface this with line with:
// Forward declarations.
src/master/simple_allocator_process.hpp
<https://reviews.apache.org/r/5599/#comment20042>
If this maps framework IDs to user names, it should be called 'users', so when you access it it reads naturally, e.g., users[frameworkId].
src/master/simple_allocator_process.hpp
<https://reviews.apache.org/r/5599/#comment20043>
This one should be called 'sorters', since that's what it yields.
src/master/simple_allocator_process.hpp
<https://reviews.apache.org/r/5599/#comment20044>
I think it would be better to templatize AllocatorProcess (or what's currently SimpleAllocatorProcess) with the user and framework sorter. One benefit of this is that we won't need a "reference" object for the framework sorter nor a need to add any notion of "clone" to the Sorter class (which is a departure from how we have managed objects in the rest of the code base).
src/master/simple_allocator_process.hpp
<https://reviews.apache.org/r/5599/#comment20045>
Did you know that these objects leaked? Again, creating objects and passing them into other classes means you have to consider ownership ... who would you want to "own" and delete these?
src/master/simple_allocator_process.cpp
<https://reviews.apache.org/r/5599/#comment20046>
Extra newline.
src/master/simple_allocator_process.cpp
<https://reviews.apache.org/r/5599/#comment20047>
Lost a comment.
src/master/simple_allocator_process.cpp
<https://reviews.apache.org/r/5599/#comment20048>
With the framework sorter templatized here you can just do 'new FrameworkSorter();' instead of 'clone'.
src/master/simple_allocator_process.cpp
<https://reviews.apache.org/r/5599/#comment20059>
Reading 'frameworks[frameworkId]' tells me I'm getting a "framework" corresponding to 'frameworkId' where as 'users[frameworkId]' tells me I'm getting a "user" corresponding to 'frameworkId'.
src/master/simple_allocator_process.cpp
<https://reviews.apache.org/r/5599/#comment20058>
Reading 'users[user]' tells me I'm getting a "user" corresponding to 'user' where as 'sorters[user]' tells me I'm getting a "sorter" corresponding to 'user'.
src/master/simple_allocator_process.cpp
<https://reviews.apache.org/r/5599/#comment20060>
It seems a bit weird to conflate the role of the sorter as also a record of what frameworks correspond to a user. What about just having a multimap (multihashmap) from users (string) to FrameworkID? Then you can eliminate Sorter::empty.
src/master/sorter.hpp
<https://reviews.apache.org/r/5599/#comment20049>
Again, these pollute other files, please remove.
src/master/sorter.hpp
<https://reviews.apache.org/r/5599/#comment20050>
// Forward declarations.
src/master/sorter.hpp
<https://reviews.apache.org/r/5599/#comment20051>
Kill.
src/master/sorter.hpp
<https://reviews.apache.org/r/5599/#comment20052>
If you can just call this add (and below remove) that would be swell (the reasoning here is that add/remove, of both clients and resources, as well as allocated/unallocated and sort are the key verbs).
src/master/sorter.hpp
<https://reviews.apache.org/r/5599/#comment20053>
It probably makes sense to factor the drf stuff into drf.hpp|cpp files.
src/master/sorter.hpp
<https://reviews.apache.org/r/5599/#comment20056>
We don't use snake_case to name our types, please change to CamelCase.
src/master/sorter.hpp
<https://reviews.apache.org/r/5599/#comment20039>
Let's call this 'sort' (the main functionality of the class).
src/master/sorter.hpp
<https://reviews.apache.org/r/5599/#comment20054>
s/updateClient/update/
src/master/sorter.hpp
<https://reviews.apache.org/r/5599/#comment20057>
Why is this needed?
src/master/sorter.cpp
<https://reviews.apache.org/r/5599/#comment20055>
Looking at this class I have no idea what is actually being compared. You should factor out the pair<string, double> into an object that has well defined names that describe what .first and .second actually are.
- Benjamin Hindman
On July 18, 2012, 6:19 p.m., Thomas Marshall wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/5599/
> -----------------------------------------------------------
>
> (Updated July 18, 2012, 6:19 p.m.)
>
>
> Review request for mesos and Benjamin Hindman.
>
>
> Description
> -------
>
> Rewrote the DRF algorithm in DominantShareAllocator to calculate shares on a per user, rather than per framework, basis and to store those shares in sorted order so that allocations have log(n) time complexity instead of n^2.
>
> Significant restructuring of the allocator code to make writing new allocators easier:
>
> - The Allocator class is renamed AllocatorProcess.
> - A new Allocator class is introduced which knows only about "elements" (either user names or frameworkIds) and how many resources they have been allocated, and then has an interface for retrieving the elements in the order they should be allocated to.
> - SimpleAllocatorProcess which can be used with different Allocators to carry out different allocation policies, including the possibility of having different policies for the per-user and per-framework allocations.
>
> Why is this restructuring a good thing?
> Now that we're doing allocations based on users, rather than frameworks, we have to do two levels of allocation - determine which user to give resources to, then determine which of their frameworks to actually make the offer to. Rather than hard-coding the allocation policies in, the new Allocator class allows us to reuse the same code to do both levels of allocation, and it allows us to easily swap different policies in or mix and match. You can even imagine some day having an allocator that allows each user to pick the per-framework allocation policy they prefer.
>
> Merely giving SimpleAllocatorProcess different Allocators is not sufficient to implement all allocation policies - only those where all resources are considered equal and there is no revocation. But, you can always create new AllocatorProcess subclasses (there will soon be a code review where I implement a static allocator this way) and these new AllocatorProcess classes can still reuse Allocators as appropriate.
>
> It also makes testing the allocator logic a lot easier since we can instantiate Allocators, give them arbitrary elements, and check that they return the right thing, instead of worrying about instantiating slaves/frameworks and using mock objects.
>
>
> This addresses bugs MESOS-225 and MESOS-226.
> https://issues.apache.org/jira/browse/MESOS-225
> https://issues.apache.org/jira/browse/MESOS-226
>
>
> Diffs
> -----
>
> src/Makefile.am 10f1101
> src/local/local.hpp 8ae5b9e
> src/local/local.cpp 46537d7
> src/master/allocator.hpp e55b9ec
> src/master/allocator_process.hpp PRE-CREATION
> src/master/main.cpp 8d38fb1
> src/master/master.hpp ad2ec25
> src/master/master.cpp b5e09d8
> src/master/simple_allocator_process.hpp PRE-CREATION
> src/master/simple_allocator_process.cpp PRE-CREATION
> src/master/sorter.hpp PRE-CREATION
> src/master/sorter.cpp PRE-CREATION
> src/tests/allocator_tests.cpp b3db13d
> src/tests/fault_tolerance_tests.cpp f892282
> src/tests/master_detector_tests.cpp 758c8b9
> src/tests/master_tests.cpp b586984
> src/tests/resource_offers_tests.cpp c004772
> src/tests/slave_tests.cpp e9b25ba
> src/tests/utils.hpp ced8592
>
> Diff: https://reviews.apache.org/r/5599/diff/
>
>
> Testing
> -------
>
> make check on Lion
>
>
> Thanks,
>
> Thomas Marshall
>
>
Re: Review Request: Made DRF per user and log(N)
Posted by Thomas Marshall <tw...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/5599/
-----------------------------------------------------------
(Updated Aug. 17, 2012, 11:23 p.m.)
Review request for mesos and Benjamin Hindman.
Changes
-------
Removed dominant_share_allocator.hpp/cpp
Description
-------
Rewrote the DRF algorithm in DominantShareAllocator to calculate shares on a per user, rather than per framework, basis and to store those shares in sorted order so that allocations have log(n) time complexity instead of n^2.
Significant restructuring of the allocator code to make writing new allocators easier:
- The Allocator class is renamed AllocatorProcess.
- A new Allocator class is introduced which knows only about "elements" (either user names or frameworkIds) and how many resources they have been allocated, and then has an interface for retrieving the elements in the order they should be allocated to.
- SimpleAllocatorProcess which can be used with different Allocators to carry out different allocation policies, including the possibility of having different policies for the per-user and per-framework allocations.
Why is this restructuring a good thing?
Now that we're doing allocations based on users, rather than frameworks, we have to do two levels of allocation - determine which user to give resources to, then determine which of their frameworks to actually make the offer to. Rather than hard-coding the allocation policies in, the new Allocator class allows us to reuse the same code to do both levels of allocation, and it allows us to easily swap different policies in or mix and match. You can even imagine some day having an allocator that allows each user to pick the per-framework allocation policy they prefer.
Merely giving SimpleAllocatorProcess different Allocators is not sufficient to implement all allocation policies - only those where all resources are considered equal and there is no revocation. But, you can always create new AllocatorProcess subclasses (there will soon be a code review where I implement a static allocator this way) and these new AllocatorProcess classes can still reuse Allocators as appropriate.
It also makes testing the allocator logic a lot easier since we can instantiate Allocators, give them arbitrary elements, and check that they return the right thing, instead of worrying about instantiating slaves/frameworks and using mock objects.
This addresses bugs MESOS-225 and MESOS-226.
https://issues.apache.org/jira/browse/MESOS-225
https://issues.apache.org/jira/browse/MESOS-226
Diffs (updated)
-----
src/Makefile.am b0cb6cc
src/local/local.hpp 8ae5b9e
src/local/local.cpp 42dbf06
src/master/allocator.hpp 8805066
src/master/allocator.cpp PRE-CREATION
src/master/dominant_share_allocator.hpp 022891b
src/master/dominant_share_allocator.cpp 17c38f0
src/master/drf_sorter.hpp PRE-CREATION
src/master/drf_sorter.cpp PRE-CREATION
src/master/flags.hpp f66f0a3
src/master/hierarchical_allocator_process.hpp PRE-CREATION
src/master/main.cpp 172772f
src/master/master.hpp d21ff3c
src/master/master.cpp 0eca2fa
src/master/sorter.hpp PRE-CREATION
src/tests/allocator_tests.cpp 98efb0d
src/tests/allocator_zookeeper_tests.cpp PRE-CREATION
src/tests/fault_tolerance_tests.cpp f892282
src/tests/gc_tests.cpp 2f0bdde
src/tests/master_detector_tests.cpp 758c8b9
src/tests/master_tests.cpp b586984
src/tests/resource_offers_tests.cpp 1366032
src/tests/utils.hpp caf5926
Diff: https://reviews.apache.org/r/5599/diff/
Testing
-------
make check on Lion
Thanks,
Thomas Marshall
Re: Review Request: Made DRF per user and log(N)
Posted by Thomas Marshall <tw...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/5599/
-----------------------------------------------------------
(Updated Aug. 17, 2012, 8:36 p.m.)
Review request for mesos and Benjamin Hindman.
Changes
-------
Minor bug fix.
Description
-------
Rewrote the DRF algorithm in DominantShareAllocator to calculate shares on a per user, rather than per framework, basis and to store those shares in sorted order so that allocations have log(n) time complexity instead of n^2.
Significant restructuring of the allocator code to make writing new allocators easier:
- The Allocator class is renamed AllocatorProcess.
- A new Allocator class is introduced which knows only about "elements" (either user names or frameworkIds) and how many resources they have been allocated, and then has an interface for retrieving the elements in the order they should be allocated to.
- SimpleAllocatorProcess which can be used with different Allocators to carry out different allocation policies, including the possibility of having different policies for the per-user and per-framework allocations.
Why is this restructuring a good thing?
Now that we're doing allocations based on users, rather than frameworks, we have to do two levels of allocation - determine which user to give resources to, then determine which of their frameworks to actually make the offer to. Rather than hard-coding the allocation policies in, the new Allocator class allows us to reuse the same code to do both levels of allocation, and it allows us to easily swap different policies in or mix and match. You can even imagine some day having an allocator that allows each user to pick the per-framework allocation policy they prefer.
Merely giving SimpleAllocatorProcess different Allocators is not sufficient to implement all allocation policies - only those where all resources are considered equal and there is no revocation. But, you can always create new AllocatorProcess subclasses (there will soon be a code review where I implement a static allocator this way) and these new AllocatorProcess classes can still reuse Allocators as appropriate.
It also makes testing the allocator logic a lot easier since we can instantiate Allocators, give them arbitrary elements, and check that they return the right thing, instead of worrying about instantiating slaves/frameworks and using mock objects.
This addresses bugs MESOS-225 and MESOS-226.
https://issues.apache.org/jira/browse/MESOS-225
https://issues.apache.org/jira/browse/MESOS-226
Diffs (updated)
-----
src/Makefile.am b0cb6cc
src/local/local.hpp 8ae5b9e
src/local/local.cpp 42dbf06
src/master/allocator.hpp 8805066
src/master/allocator.cpp PRE-CREATION
src/master/drf_sorter.hpp PRE-CREATION
src/master/drf_sorter.cpp PRE-CREATION
src/master/flags.hpp f66f0a3
src/master/hierarchical_allocator_process.hpp PRE-CREATION
src/master/main.cpp 172772f
src/master/master.hpp d21ff3c
src/master/master.cpp 0eca2fa
src/master/sorter.hpp PRE-CREATION
src/tests/allocator_tests.cpp 98efb0d
src/tests/allocator_zookeeper_tests.cpp PRE-CREATION
src/tests/fault_tolerance_tests.cpp f892282
src/tests/gc_tests.cpp 2f0bdde
src/tests/master_detector_tests.cpp 758c8b9
src/tests/master_tests.cpp b586984
src/tests/resource_offers_tests.cpp 1366032
src/tests/utils.hpp caf5926
Diff: https://reviews.apache.org/r/5599/diff/
Testing
-------
make check on Lion
Thanks,
Thomas Marshall
Re: Review Request: Made DRF per user and log(N)
Posted by Thomas Marshall <tw...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/5599/
-----------------------------------------------------------
(Updated Aug. 16, 2012, 12:53 a.m.)
Review request for mesos and Benjamin Hindman.
Changes
-------
Updated to latest review.
Description
-------
Rewrote the DRF algorithm in DominantShareAllocator to calculate shares on a per user, rather than per framework, basis and to store those shares in sorted order so that allocations have log(n) time complexity instead of n^2.
Significant restructuring of the allocator code to make writing new allocators easier:
- The Allocator class is renamed AllocatorProcess.
- A new Allocator class is introduced which knows only about "elements" (either user names or frameworkIds) and how many resources they have been allocated, and then has an interface for retrieving the elements in the order they should be allocated to.
- SimpleAllocatorProcess which can be used with different Allocators to carry out different allocation policies, including the possibility of having different policies for the per-user and per-framework allocations.
Why is this restructuring a good thing?
Now that we're doing allocations based on users, rather than frameworks, we have to do two levels of allocation - determine which user to give resources to, then determine which of their frameworks to actually make the offer to. Rather than hard-coding the allocation policies in, the new Allocator class allows us to reuse the same code to do both levels of allocation, and it allows us to easily swap different policies in or mix and match. You can even imagine some day having an allocator that allows each user to pick the per-framework allocation policy they prefer.
Merely giving SimpleAllocatorProcess different Allocators is not sufficient to implement all allocation policies - only those where all resources are considered equal and there is no revocation. But, you can always create new AllocatorProcess subclasses (there will soon be a code review where I implement a static allocator this way) and these new AllocatorProcess classes can still reuse Allocators as appropriate.
It also makes testing the allocator logic a lot easier since we can instantiate Allocators, give them arbitrary elements, and check that they return the right thing, instead of worrying about instantiating slaves/frameworks and using mock objects.
This addresses bugs MESOS-225 and MESOS-226.
https://issues.apache.org/jira/browse/MESOS-225
https://issues.apache.org/jira/browse/MESOS-226
Diffs (updated)
-----
src/Makefile.am b0cb6cc
src/local/local.hpp 8ae5b9e
src/local/local.cpp 42dbf06
src/master/allocator.hpp 8805066
src/master/allocator.cpp PRE-CREATION
src/master/drf_sorter.hpp PRE-CREATION
src/master/drf_sorter.cpp PRE-CREATION
src/master/flags.hpp f66f0a3
src/master/hierarchical_allocator_process.hpp PRE-CREATION
src/master/main.cpp 172772f
src/master/master.hpp d21ff3c
src/master/master.cpp 0eca2fa
src/master/sorter.hpp PRE-CREATION
src/tests/allocator_tests.cpp 98efb0d
src/tests/allocator_zookeeper_tests.cpp PRE-CREATION
src/tests/fault_tolerance_tests.cpp f892282
src/tests/gc_tests.cpp 2f0bdde
src/tests/master_detector_tests.cpp 758c8b9
src/tests/master_tests.cpp b586984
src/tests/resource_offers_tests.cpp 1366032
src/tests/utils.hpp caf5926
Diff: https://reviews.apache.org/r/5599/diff/
Testing
-------
make check on Lion
Thanks,
Thomas Marshall
Re: Review Request: Made DRF per user and log(N)
Posted by Benjamin Hindman <be...@berkeley.edu>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/5599/#review10365
-----------------------------------------------------------
Ship it!
Ye haw!
src/master/allocator.cpp
<https://reviews.apache.org/r/5599/#comment22001>
Let's call this UserFrameworkAllocatorProcess or something more descriptive than SimpleAllocatorProcess.
src/master/drf_sorter.hpp
<https://reviews.apache.org/r/5599/#comment22004>
s|sorter|master/sorter|
src/master/drf_sorter.hpp
<https://reviews.apache.org/r/5599/#comment22003>
Let's not use the typedef.
src/master/drf_sorter.hpp
<https://reviews.apache.org/r/5599/#comment22007>
Kill 'virtual'.
src/master/drf_sorter.hpp
<https://reviews.apache.org/r/5599/#comment22008>
Ditto.
src/master/drf_sorter.hpp
<https://reviews.apache.org/r/5599/#comment22009>
Ditto.
src/master/drf_sorter.hpp
<https://reviews.apache.org/r/5599/#comment22012>
s/clients/client names/
src/master/drf_sorter.cpp
<https://reviews.apache.org/r/5599/#comment22006>
master/
src/master/drf_sorter.cpp
<https://reviews.apache.org/r/5599/#comment22011>
Always remove from allocations.
src/master/drf_sorter.cpp
<https://reviews.apache.org/r/5599/#comment22025>
s/client/name/
src/master/flags.hpp
<https://reviews.apache.org/r/5599/#comment22021>
s/allocator/sorter
src/master/flags.hpp
<https://reviews.apache.org/r/5599/#comment22022>
Ditto.
src/master/simple_allocator_process.hpp
<https://reviews.apache.org/r/5599/#comment22026>
const&
src/master/simple_allocator_process.hpp
<https://reviews.apache.org/r/5599/#comment22013>
We should swap the semantics of these please.
src/master/simple_allocator_process.hpp
<https://reviews.apache.org/r/5599/#comment22020>
Use new self.
src/master/simple_allocator_process.hpp
<https://reviews.apache.org/r/5599/#comment22014>
Swap these too.
src/master/simple_allocator_process.hpp
<https://reviews.apache.org/r/5599/#comment22023>
users ;)
src/master/simple_allocator_process.hpp
<https://reviews.apache.org/r/5599/#comment22017>
const&
src/master/simple_allocator_process.hpp
<https://reviews.apache.org/r/5599/#comment22018>
const&
src/master/simple_allocator_process.hpp
<https://reviews.apache.org/r/5599/#comment22015>
Kill this allocate.
src/master/simple_allocator_process.hpp
<https://reviews.apache.org/r/5599/#comment22016>
Format this line (and others that wrap beyond 80 characters).
src/master/sorter.hpp
<https://reviews.apache.org/r/5599/#comment22024>
s|/should|/ should|
- Benjamin Hindman
On Aug. 13, 2012, 11:59 p.m., Thomas Marshall wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/5599/
> -----------------------------------------------------------
>
> (Updated Aug. 13, 2012, 11:59 p.m.)
>
>
> Review request for mesos and Benjamin Hindman.
>
>
> Description
> -------
>
> Rewrote the DRF algorithm in DominantShareAllocator to calculate shares on a per user, rather than per framework, basis and to store those shares in sorted order so that allocations have log(n) time complexity instead of n^2.
>
> Significant restructuring of the allocator code to make writing new allocators easier:
>
> - The Allocator class is renamed AllocatorProcess.
> - A new Allocator class is introduced which knows only about "elements" (either user names or frameworkIds) and how many resources they have been allocated, and then has an interface for retrieving the elements in the order they should be allocated to.
> - SimpleAllocatorProcess which can be used with different Allocators to carry out different allocation policies, including the possibility of having different policies for the per-user and per-framework allocations.
>
> Why is this restructuring a good thing?
> Now that we're doing allocations based on users, rather than frameworks, we have to do two levels of allocation - determine which user to give resources to, then determine which of their frameworks to actually make the offer to. Rather than hard-coding the allocation policies in, the new Allocator class allows us to reuse the same code to do both levels of allocation, and it allows us to easily swap different policies in or mix and match. You can even imagine some day having an allocator that allows each user to pick the per-framework allocation policy they prefer.
>
> Merely giving SimpleAllocatorProcess different Allocators is not sufficient to implement all allocation policies - only those where all resources are considered equal and there is no revocation. But, you can always create new AllocatorProcess subclasses (there will soon be a code review where I implement a static allocator this way) and these new AllocatorProcess classes can still reuse Allocators as appropriate.
>
> It also makes testing the allocator logic a lot easier since we can instantiate Allocators, give them arbitrary elements, and check that they return the right thing, instead of worrying about instantiating slaves/frameworks and using mock objects.
>
>
> This addresses bugs MESOS-225 and MESOS-226.
> https://issues.apache.org/jira/browse/MESOS-225
> https://issues.apache.org/jira/browse/MESOS-226
>
>
> Diffs
> -----
>
> src/Makefile.am b0cb6cc
> src/local/local.hpp 8ae5b9e
> src/local/local.cpp 42dbf06
> src/master/allocator.hpp 8805066
> src/master/allocator.cpp PRE-CREATION
> src/master/drf_sorter.hpp PRE-CREATION
> src/master/drf_sorter.cpp PRE-CREATION
> src/master/flags.hpp f66f0a3
> src/master/main.cpp 172772f
> src/master/master.hpp d21ff3c
> src/master/master.cpp 0eca2fa
> src/master/simple_allocator_process.hpp PRE-CREATION
> src/master/sorter.hpp PRE-CREATION
> src/tests/allocator_tests.cpp 98efb0d
> src/tests/allocator_zookeeper_tests.cpp PRE-CREATION
> src/tests/fault_tolerance_tests.cpp f892282
> src/tests/gc_tests.cpp 2f0bdde
> src/tests/master_detector_tests.cpp 758c8b9
> src/tests/master_tests.cpp b586984
> src/tests/resource_offers_tests.cpp 1366032
> src/tests/utils.hpp caf5926
>
> Diff: https://reviews.apache.org/r/5599/diff/
>
>
> Testing
> -------
>
> make check on Lion
>
>
> Thanks,
>
> Thomas Marshall
>
>
Re: Review Request: Made DRF per user and log(N)
Posted by Thomas Marshall <tw...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/5599/
-----------------------------------------------------------
(Updated Aug. 13, 2012, 11:59 p.m.)
Review request for mesos and Benjamin Hindman.
Changes
-------
Updated to trunk.
Description
-------
Rewrote the DRF algorithm in DominantShareAllocator to calculate shares on a per user, rather than per framework, basis and to store those shares in sorted order so that allocations have log(n) time complexity instead of n^2.
Significant restructuring of the allocator code to make writing new allocators easier:
- The Allocator class is renamed AllocatorProcess.
- A new Allocator class is introduced which knows only about "elements" (either user names or frameworkIds) and how many resources they have been allocated, and then has an interface for retrieving the elements in the order they should be allocated to.
- SimpleAllocatorProcess which can be used with different Allocators to carry out different allocation policies, including the possibility of having different policies for the per-user and per-framework allocations.
Why is this restructuring a good thing?
Now that we're doing allocations based on users, rather than frameworks, we have to do two levels of allocation - determine which user to give resources to, then determine which of their frameworks to actually make the offer to. Rather than hard-coding the allocation policies in, the new Allocator class allows us to reuse the same code to do both levels of allocation, and it allows us to easily swap different policies in or mix and match. You can even imagine some day having an allocator that allows each user to pick the per-framework allocation policy they prefer.
Merely giving SimpleAllocatorProcess different Allocators is not sufficient to implement all allocation policies - only those where all resources are considered equal and there is no revocation. But, you can always create new AllocatorProcess subclasses (there will soon be a code review where I implement a static allocator this way) and these new AllocatorProcess classes can still reuse Allocators as appropriate.
It also makes testing the allocator logic a lot easier since we can instantiate Allocators, give them arbitrary elements, and check that they return the right thing, instead of worrying about instantiating slaves/frameworks and using mock objects.
This addresses bugs MESOS-225 and MESOS-226.
https://issues.apache.org/jira/browse/MESOS-225
https://issues.apache.org/jira/browse/MESOS-226
Diffs (updated)
-----
src/Makefile.am b0cb6cc
src/local/local.hpp 8ae5b9e
src/local/local.cpp 42dbf06
src/master/allocator.hpp 8805066
src/master/allocator.cpp PRE-CREATION
src/master/drf_sorter.hpp PRE-CREATION
src/master/drf_sorter.cpp PRE-CREATION
src/master/flags.hpp f66f0a3
src/master/main.cpp 172772f
src/master/master.hpp d21ff3c
src/master/master.cpp 0eca2fa
src/master/simple_allocator_process.hpp PRE-CREATION
src/master/sorter.hpp PRE-CREATION
src/tests/allocator_tests.cpp 98efb0d
src/tests/allocator_zookeeper_tests.cpp PRE-CREATION
src/tests/fault_tolerance_tests.cpp f892282
src/tests/gc_tests.cpp 2f0bdde
src/tests/master_detector_tests.cpp 758c8b9
src/tests/master_tests.cpp b586984
src/tests/resource_offers_tests.cpp 1366032
src/tests/utils.hpp caf5926
Diff: https://reviews.apache.org/r/5599/diff/
Testing
-------
make check on Lion
Thanks,
Thomas Marshall
Re: Review Request: Made DRF per user and log(N)
Posted by Thomas Marshall <tw...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/5599/
-----------------------------------------------------------
(Updated Aug. 8, 2012, 9 p.m.)
Review request for mesos and Benjamin Hindman.
Changes
-------
So, after starting to actually implement revocation, it occurred to me that organizing AllocatorProcess as a class hierarchy doesn't allow us to easily turn on and off different options independently (ex. if static allocations and coalescing revocation are implemented as different SimpleAllocatorProcess subclasses, how do we do both at the same time?). So, instead I'm going to pass flags for static/revocation stuff into allocator process and take the right actions based on those flags, keeping all allocation logic in SimpleAllocatorProcess.
This impacts this patch because it allows SimpleAllocatorProcess to be templated, which turns out to be a much cleaner solution to the Sorter injection than some sort of factory function (ironically, after I spent so much time arguing against doing it this way).
Description
-------
Rewrote the DRF algorithm in DominantShareAllocator to calculate shares on a per user, rather than per framework, basis and to store those shares in sorted order so that allocations have log(n) time complexity instead of n^2.
Significant restructuring of the allocator code to make writing new allocators easier:
- The Allocator class is renamed AllocatorProcess.
- A new Allocator class is introduced which knows only about "elements" (either user names or frameworkIds) and how many resources they have been allocated, and then has an interface for retrieving the elements in the order they should be allocated to.
- SimpleAllocatorProcess which can be used with different Allocators to carry out different allocation policies, including the possibility of having different policies for the per-user and per-framework allocations.
Why is this restructuring a good thing?
Now that we're doing allocations based on users, rather than frameworks, we have to do two levels of allocation - determine which user to give resources to, then determine which of their frameworks to actually make the offer to. Rather than hard-coding the allocation policies in, the new Allocator class allows us to reuse the same code to do both levels of allocation, and it allows us to easily swap different policies in or mix and match. You can even imagine some day having an allocator that allows each user to pick the per-framework allocation policy they prefer.
Merely giving SimpleAllocatorProcess different Allocators is not sufficient to implement all allocation policies - only those where all resources are considered equal and there is no revocation. But, you can always create new AllocatorProcess subclasses (there will soon be a code review where I implement a static allocator this way) and these new AllocatorProcess classes can still reuse Allocators as appropriate.
It also makes testing the allocator logic a lot easier since we can instantiate Allocators, give them arbitrary elements, and check that they return the right thing, instead of worrying about instantiating slaves/frameworks and using mock objects.
This addresses bugs MESOS-225 and MESOS-226.
https://issues.apache.org/jira/browse/MESOS-225
https://issues.apache.org/jira/browse/MESOS-226
Diffs (updated)
-----
src/Makefile.am cc3480c
src/local/local.hpp 8ae5b9e
src/local/local.cpp 46537d7
src/master/allocator.hpp c6273b5
src/master/allocator.cpp PRE-CREATION
src/master/dominant_share_allocator.hpp ba5dc29
src/master/dominant_share_allocator.cpp 00a47f7
src/master/drf_sorter.hpp PRE-CREATION
src/master/drf_sorter.cpp PRE-CREATION
src/master/flags.hpp 1227ccc
src/master/main.cpp 172772f
src/master/master.hpp 30e3f75
src/master/master.cpp b174f7f
src/master/simple_allocator_process.hpp PRE-CREATION
src/master/sorter.hpp PRE-CREATION
src/tests/allocator_tests.cpp b3db13d
src/tests/allocator_zookeeper_tests.cpp PRE-CREATION
src/tests/fault_tolerance_tests.cpp f892282
src/tests/master_detector_tests.cpp 758c8b9
src/tests/master_tests.cpp b586984
src/tests/resource_offers_tests.cpp c004772
src/tests/slave_tests.cpp e9b25ba
src/tests/utils.hpp a768360
Diff: https://reviews.apache.org/r/5599/diff/
Testing
-------
make check on Lion
Thanks,
Thomas Marshall
Re: Review Request: Made DRF per user and log(N)
Posted by Thomas Marshall <tw...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/5599/
-----------------------------------------------------------
(Updated Aug. 8, 2012, 12:29 a.m.)
Review request for mesos and Benjamin Hindman.
Changes
-------
Updated to the most recent review.
This patch now assumes that you've applied:
https://reviews.apache.org/r/6037/
Description
-------
Rewrote the DRF algorithm in DominantShareAllocator to calculate shares on a per user, rather than per framework, basis and to store those shares in sorted order so that allocations have log(n) time complexity instead of n^2.
Significant restructuring of the allocator code to make writing new allocators easier:
- The Allocator class is renamed AllocatorProcess.
- A new Allocator class is introduced which knows only about "elements" (either user names or frameworkIds) and how many resources they have been allocated, and then has an interface for retrieving the elements in the order they should be allocated to.
- SimpleAllocatorProcess which can be used with different Allocators to carry out different allocation policies, including the possibility of having different policies for the per-user and per-framework allocations.
Why is this restructuring a good thing?
Now that we're doing allocations based on users, rather than frameworks, we have to do two levels of allocation - determine which user to give resources to, then determine which of their frameworks to actually make the offer to. Rather than hard-coding the allocation policies in, the new Allocator class allows us to reuse the same code to do both levels of allocation, and it allows us to easily swap different policies in or mix and match. You can even imagine some day having an allocator that allows each user to pick the per-framework allocation policy they prefer.
Merely giving SimpleAllocatorProcess different Allocators is not sufficient to implement all allocation policies - only those where all resources are considered equal and there is no revocation. But, you can always create new AllocatorProcess subclasses (there will soon be a code review where I implement a static allocator this way) and these new AllocatorProcess classes can still reuse Allocators as appropriate.
It also makes testing the allocator logic a lot easier since we can instantiate Allocators, give them arbitrary elements, and check that they return the right thing, instead of worrying about instantiating slaves/frameworks and using mock objects.
This addresses bugs MESOS-225 and MESOS-226.
https://issues.apache.org/jira/browse/MESOS-225
https://issues.apache.org/jira/browse/MESOS-226
Diffs (updated)
-----
src/Makefile.am cc3480c
src/local/local.hpp 8ae5b9e
src/local/local.cpp 46537d7
src/master/allocator.hpp c6273b5
src/master/allocator.cpp PRE-CREATION
src/master/dominant_share_allocator.hpp ba5dc29
src/master/dominant_share_allocator.cpp 00a47f7
src/master/drf_sorter.hpp PRE-CREATION
src/master/drf_sorter.cpp PRE-CREATION
src/master/flags.hpp 1227ccc
src/master/main.cpp 172772f
src/master/master.hpp 30e3f75
src/master/master.cpp b174f7f
src/master/simple_allocator_process.hpp PRE-CREATION
src/master/simple_allocator_process.cpp PRE-CREATION
src/master/sorter.hpp PRE-CREATION
src/master/sorter.cpp PRE-CREATION
src/tests/allocator_tests.cpp b3db13d
src/tests/allocator_zookeeper_tests.cpp PRE-CREATION
src/tests/fault_tolerance_tests.cpp f892282
src/tests/master_detector_tests.cpp 758c8b9
src/tests/master_tests.cpp b586984
src/tests/resource_offers_tests.cpp c004772
src/tests/slave_tests.cpp e9b25ba
src/tests/utils.hpp a768360
Diff: https://reviews.apache.org/r/5599/diff/
Testing
-------
make check on Lion
Thanks,
Thomas Marshall
Re: Review Request: Made DRF per user and log(N)
Posted by Thomas Marshall <tw...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/5599/
-----------------------------------------------------------
(Updated July 18, 2012, 6:19 p.m.)
Review request for mesos and Benjamin Hindman.
Changes
-------
Minor bug fix.
Description
-------
Rewrote the DRF algorithm in DominantShareAllocator to calculate shares on a per user, rather than per framework, basis and to store those shares in sorted order so that allocations have log(n) time complexity instead of n^2.
Significant restructuring of the allocator code to make writing new allocators easier:
- The Allocator class is renamed AllocatorProcess.
- A new Allocator class is introduced which knows only about "elements" (either user names or frameworkIds) and how many resources they have been allocated, and then has an interface for retrieving the elements in the order they should be allocated to.
- SimpleAllocatorProcess which can be used with different Allocators to carry out different allocation policies, including the possibility of having different policies for the per-user and per-framework allocations.
Why is this restructuring a good thing?
Now that we're doing allocations based on users, rather than frameworks, we have to do two levels of allocation - determine which user to give resources to, then determine which of their frameworks to actually make the offer to. Rather than hard-coding the allocation policies in, the new Allocator class allows us to reuse the same code to do both levels of allocation, and it allows us to easily swap different policies in or mix and match. You can even imagine some day having an allocator that allows each user to pick the per-framework allocation policy they prefer.
Merely giving SimpleAllocatorProcess different Allocators is not sufficient to implement all allocation policies - only those where all resources are considered equal and there is no revocation. But, you can always create new AllocatorProcess subclasses (there will soon be a code review where I implement a static allocator this way) and these new AllocatorProcess classes can still reuse Allocators as appropriate.
It also makes testing the allocator logic a lot easier since we can instantiate Allocators, give them arbitrary elements, and check that they return the right thing, instead of worrying about instantiating slaves/frameworks and using mock objects.
This addresses bugs MESOS-225 and MESOS-226.
https://issues.apache.org/jira/browse/MESOS-225
https://issues.apache.org/jira/browse/MESOS-226
Diffs (updated)
-----
src/Makefile.am 10f1101
src/local/local.hpp 8ae5b9e
src/local/local.cpp 46537d7
src/master/allocator.hpp e55b9ec
src/master/allocator_process.hpp PRE-CREATION
src/master/main.cpp 8d38fb1
src/master/master.hpp ad2ec25
src/master/master.cpp b5e09d8
src/master/simple_allocator_process.hpp PRE-CREATION
src/master/simple_allocator_process.cpp PRE-CREATION
src/master/sorter.hpp PRE-CREATION
src/master/sorter.cpp PRE-CREATION
src/tests/allocator_tests.cpp b3db13d
src/tests/fault_tolerance_tests.cpp f892282
src/tests/master_detector_tests.cpp 758c8b9
src/tests/master_tests.cpp b586984
src/tests/resource_offers_tests.cpp c004772
src/tests/slave_tests.cpp e9b25ba
src/tests/utils.hpp ced8592
Diff: https://reviews.apache.org/r/5599/diff/
Testing
-------
make check on Lion
Thanks,
Thomas Marshall
Re: Review Request: Made DRF per user and log(N)
Posted by Thomas Marshall <tw...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/5599/
-----------------------------------------------------------
(Updated July 18, 2012, 12:47 a.m.)
Review request for mesos and Benjamin Hindman.
Description
-------
Rewrote the DRF algorithm in DominantShareAllocator to calculate shares on a per user, rather than per framework, basis and to store those shares in sorted order so that allocations have log(n) time complexity instead of n^2.
Significant restructuring of the allocator code to make writing new allocators easier:
- The Allocator class is renamed AllocatorProcess.
- A new Allocator class is introduced which knows only about "elements" (either user names or frameworkIds) and how many resources they have been allocated, and then has an interface for retrieving the elements in the order they should be allocated to.
- SimpleAllocatorProcess which can be used with different Allocators to carry out different allocation policies, including the possibility of having different policies for the per-user and per-framework allocations.
Why is this restructuring a good thing?
Now that we're doing allocations based on users, rather than frameworks, we have to do two levels of allocation - determine which user to give resources to, then determine which of their frameworks to actually make the offer to. Rather than hard-coding the allocation policies in, the new Allocator class allows us to reuse the same code to do both levels of allocation, and it allows us to easily swap different policies in or mix and match. You can even imagine some day having an allocator that allows each user to pick the per-framework allocation policy they prefer.
Merely giving SimpleAllocatorProcess different Allocators is not sufficient to implement all allocation policies - only those where all resources are considered equal and there is no revocation. But, you can always create new AllocatorProcess subclasses (there will soon be a code review where I implement a static allocator this way) and these new AllocatorProcess classes can still reuse Allocators as appropriate.
It also makes testing the allocator logic a lot easier since we can instantiate Allocators, give them arbitrary elements, and check that they return the right thing, instead of worrying about instantiating slaves/frameworks and using mock objects.
This addresses bugs MESOS-225 and MESOS-226.
https://issues.apache.org/jira/browse/MESOS-225
https://issues.apache.org/jira/browse/MESOS-226
Diffs
-----
src/Makefile.am 10f1101
src/local/local.hpp 8ae5b9e
src/local/local.cpp 46537d7
src/master/allocator.hpp e55b9ec
src/master/allocator_process.hpp PRE-CREATION
src/master/main.cpp 8d38fb1
src/master/master.hpp ad2ec25
src/master/master.cpp b5e09d8
src/master/simple_allocator_process.hpp PRE-CREATION
src/master/simple_allocator_process.cpp PRE-CREATION
src/master/sorter.hpp PRE-CREATION
src/master/sorter.cpp PRE-CREATION
src/tests/allocator_tests.cpp b3db13d
src/tests/fault_tolerance_tests.cpp f892282
src/tests/master_detector_tests.cpp 758c8b9
src/tests/master_tests.cpp b586984
src/tests/resource_offers_tests.cpp c004772
src/tests/slave_tests.cpp e9b25ba
src/tests/utils.hpp ced8592
Diff: https://reviews.apache.org/r/5599/diff/
Testing
-------
make check on Lion
Thanks,
Thomas Marshall
Re: Review Request: Made DRF per user and log(N)
Posted by Thomas Marshall <tw...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/5599/
-----------------------------------------------------------
(Updated July 18, 2012, 12:47 a.m.)
Review request for mesos and Benjamin Hindman.
Changes
-------
s/Allocator/Sorter
s/element/client
Changed Sorter iteration functionality from start/next/hasNext to list<string> getOrdering()
Description
-------
Rewrote the DRF algorithm in DominantShareAllocator to calculate shares on a per user, rather than per framework, basis and to store those shares in sorted order so that allocations have log(n) time complexity instead of n^2.
Significant restructuring of the allocator code to make writing new allocators easier:
- The Allocator class is renamed AllocatorProcess.
- A new Allocator class is introduced which knows only about "elements" (either user names or frameworkIds) and how many resources they have been allocated, and then has an interface for retrieving the elements in the order they should be allocated to.
- SimpleAllocatorProcess which can be used with different Allocators to carry out different allocation policies, including the possibility of having different policies for the per-user and per-framework allocations.
Why is this restructuring a good thing?
Now that we're doing allocations based on users, rather than frameworks, we have to do two levels of allocation - determine which user to give resources to, then determine which of their frameworks to actually make the offer to. Rather than hard-coding the allocation policies in, the new Allocator class allows us to reuse the same code to do both levels of allocation, and it allows us to easily swap different policies in or mix and match. You can even imagine some day having an allocator that allows each user to pick the per-framework allocation policy they prefer.
Merely giving SimpleAllocatorProcess different Allocators is not sufficient to implement all allocation policies - only those where all resources are considered equal and there is no revocation. But, you can always create new AllocatorProcess subclasses (there will soon be a code review where I implement a static allocator this way) and these new AllocatorProcess classes can still reuse Allocators as appropriate.
It also makes testing the allocator logic a lot easier since we can instantiate Allocators, give them arbitrary elements, and check that they return the right thing, instead of worrying about instantiating slaves/frameworks and using mock objects.
This addresses bugs MESOS-225 and MESOS-226.
https://issues.apache.org/jira/browse/MESOS-225
https://issues.apache.org/jira/browse/MESOS-226
Diffs (updated)
-----
src/Makefile.am 10f1101
src/local/local.hpp 8ae5b9e
src/local/local.cpp 46537d7
src/master/allocator.hpp e55b9ec
src/master/allocator_process.hpp PRE-CREATION
src/master/main.cpp 8d38fb1
src/master/master.hpp ad2ec25
src/master/master.cpp b5e09d8
src/master/simple_allocator_process.hpp PRE-CREATION
src/master/simple_allocator_process.cpp PRE-CREATION
src/master/sorter.hpp PRE-CREATION
src/master/sorter.cpp PRE-CREATION
src/tests/allocator_tests.cpp b3db13d
src/tests/fault_tolerance_tests.cpp f892282
src/tests/master_detector_tests.cpp 758c8b9
src/tests/master_tests.cpp b586984
src/tests/resource_offers_tests.cpp c004772
src/tests/slave_tests.cpp e9b25ba
src/tests/utils.hpp ced8592
Diff: https://reviews.apache.org/r/5599/diff/
Testing
-------
make check on Lion
Thanks,
Thomas Marshall
Re: Review Request: Made DRF per user and log(N)
Posted by Thomas Marshall <tw...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/5599/
-----------------------------------------------------------
(Updated July 12, 2012, 11:11 p.m.)
Review request for mesos and Benjamin Hindman.
Changes
-------
Updated to trunk.
Description
-------
Rewrote the DRF algorithm in DominantShareAllocator to calculate shares on a per user, rather than per framework, basis and to store those shares in sorted order so that allocations have log(n) time complexity instead of n^2.
Significant restructuring of the allocator code to make writing new allocators easier:
- The Allocator class is renamed AllocatorProcess.
- A new Allocator class is introduced which knows only about "elements" (either user names or frameworkIds) and how many resources they have been allocated, and then has an interface for retrieving the elements in the order they should be allocated to.
- SimpleAllocatorProcess which can be used with different Allocators to carry out different allocation policies, including the possibility of having different policies for the per-user and per-framework allocations.
Why is this restructuring a good thing?
Now that we're doing allocations based on users, rather than frameworks, we have to do two levels of allocation - determine which user to give resources to, then determine which of their frameworks to actually make the offer to. Rather than hard-coding the allocation policies in, the new Allocator class allows us to reuse the same code to do both levels of allocation, and it allows us to easily swap different policies in or mix and match. You can even imagine some day having an allocator that allows each user to pick the per-framework allocation policy they prefer.
Merely giving SimpleAllocatorProcess different Allocators is not sufficient to implement all allocation policies - only those where all resources are considered equal and there is no revocation. But, you can always create new AllocatorProcess subclasses (there will soon be a code review where I implement a static allocator this way) and these new AllocatorProcess classes can still reuse Allocators as appropriate.
It also makes testing the allocator logic a lot easier since we can instantiate Allocators, give them arbitrary elements, and check that they return the right thing, instead of worrying about instantiating slaves/frameworks and using mock objects.
This addresses bugs MESOS-225 and MESOS-226.
https://issues.apache.org/jira/browse/MESOS-225
https://issues.apache.org/jira/browse/MESOS-226
Diffs (updated)
-----
src/Makefile.am 46285ac
src/local/local.hpp 8ae5b9e
src/local/local.cpp 46537d7
src/master/allocator.hpp 38736cb
src/master/allocator.cpp PRE-CREATION
src/master/allocator_process.hpp PRE-CREATION
src/master/dominant_share_allocator.hpp 6587f33
src/master/dominant_share_allocator.cpp 1ea29c7
src/master/main.cpp 8d38fb1
src/master/master.hpp 6cb4810
src/master/master.cpp edefd70
src/master/simple_allocator_process.hpp PRE-CREATION
src/master/simple_allocator_process.cpp PRE-CREATION
src/tests/allocator_tests.cpp 610826b
src/tests/fault_tolerance_tests.cpp dd578aa
src/tests/master_detector_tests.cpp 758c8b9
src/tests/master_tests.cpp b586984
src/tests/resource_offers_tests.cpp d06cae2
src/tests/slave_tests.cpp e9b25ba
src/tests/utils.hpp 1062dfc
Diff: https://reviews.apache.org/r/5599/diff/
Testing
-------
make check on Lion
Thanks,
Thomas Marshall
Re: Review Request: Made DRF per user and log(N)
Posted by Thomas Marshall <tw...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/5599/
-----------------------------------------------------------
(Updated July 11, 2012, 10:33 p.m.)
Review request for mesos and Benjamin Hindman.
Changes
-------
Updated to trunk.
Description
-------
Rewrote the DRF algorithm in DominantShareAllocator to calculate shares on a per user, rather than per framework, basis and to store those shares in sorted order so that allocations have log(n) time complexity instead of n^2.
Significant restructuring of the allocator code to make writing new allocators easier:
- The Allocator class is renamed AllocatorProcess.
- A new Allocator class is introduced which knows only about "elements" (either user names or frameworkIds) and how many resources they have been allocated, and then has an interface for retrieving the elements in the order they should be allocated to.
- SimpleAllocatorProcess which can be used with different Allocators to carry out different allocation policies, including the possibility of having different policies for the per-user and per-framework allocations.
Why is this restructuring a good thing?
Now that we're doing allocations based on users, rather than frameworks, we have to do two levels of allocation - determine which user to give resources to, then determine which of their frameworks to actually make the offer to. Rather than hard-coding the allocation policies in, the new Allocator class allows us to reuse the same code to do both levels of allocation, and it allows us to easily swap different policies in or mix and match. You can even imagine some day having an allocator that allows each user to pick the per-framework allocation policy they prefer.
Merely giving SimpleAllocatorProcess different Allocators is not sufficient to implement all allocation policies - only those where all resources are considered equal and there is no revocation. But, you can always create new AllocatorProcess subclasses (there will soon be a code review where I implement a static allocator this way) and these new AllocatorProcess classes can still reuse Allocators as appropriate.
It also makes testing the allocator logic a lot easier since we can instantiate Allocators, give them arbitrary elements, and check that they return the right thing, instead of worrying about instantiating slaves/frameworks and using mock objects.
This addresses bugs MESOS-225 and MESOS-226.
https://issues.apache.org/jira/browse/MESOS-225
https://issues.apache.org/jira/browse/MESOS-226
Diffs (updated)
-----
src/Makefile.am 99bee0a
src/local/local.hpp 8ae5b9e
src/local/local.cpp f8277d0
src/master/allocator.hpp 12f31db
src/master/allocator.cpp PRE-CREATION
src/master/allocator_process.hpp PRE-CREATION
src/master/dominant_share_allocator.hpp 650e441
src/master/dominant_share_allocator.cpp d8c0052
src/master/main.cpp c459092
src/master/master.hpp 26de789
src/master/master.cpp 4cfd679
src/master/simple_allocator_process.hpp PRE-CREATION
src/master/simple_allocator_process.cpp PRE-CREATION
src/tests/allocator_tests.cpp 610826b
src/tests/fault_tolerance_tests.cpp dd578aa
src/tests/master_detector_tests.cpp afd8a74
src/tests/master_tests.cpp 83134bd
src/tests/resource_offers_tests.cpp d06cae2
src/tests/slave_tests.cpp f3d7d67
src/tests/utils.hpp a48c14f
Diff: https://reviews.apache.org/r/5599/diff/
Testing
-------
make check on Lion
Thanks,
Thomas Marshall
Re: Review Request: Made DRF per user and log(N)
Posted by Thomas Marshall <tw...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/5599/
-----------------------------------------------------------
(Updated July 11, 2012, 9:47 p.m.)
Review request for mesos and Benjamin Hindman.
Changes
-------
Bug fix.
Description (updated)
-------
Rewrote the DRF algorithm in DominantShareAllocator to calculate shares on a per user, rather than per framework, basis and to store those shares in sorted order so that allocations have log(n) time complexity instead of n^2.
Significant restructuring of the allocator code to make writing new allocators easier:
- The Allocator class is renamed AllocatorProcess.
- A new Allocator class is introduced which knows only about "elements" (either user names or frameworkIds) and how many resources they have been allocated, and then has an interface for retrieving the elements in the order they should be allocated to.
- SimpleAllocatorProcess which can be used with different Allocators to carry out different allocation policies, including the possibility of having different policies for the per-user and per-framework allocations.
Why is this restructuring a good thing?
Now that we're doing allocations based on users, rather than frameworks, we have to do two levels of allocation - determine which user to give resources to, then determine which of their frameworks to actually make the offer to. Rather than hard-coding the allocation policies in, the new Allocator class allows us to reuse the same code to do both levels of allocation, and it allows us to easily swap different policies in or mix and match. You can even imagine some day having an allocator that allows each user to pick the per-framework allocation policy they prefer.
Merely giving SimpleAllocatorProcess different Allocators is not sufficient to implement all allocation policies - only those where all resources are considered equal and there is no revocation. But, you can always create new AllocatorProcess subclasses (there will soon be a code review where I implement a static allocator this way) and these new AllocatorProcess classes can still reuse Allocators as appropriate.
It also makes testing the allocator logic a lot easier since we can instantiate Allocators, give them arbitrary elements, and check that they return the right thing, instead of worrying about instantiating slaves/frameworks and using mock objects.
This addresses bugs MESOS-225 and MESOS-226.
https://issues.apache.org/jira/browse/MESOS-225
https://issues.apache.org/jira/browse/MESOS-226
Diffs (updated)
-----
src/Makefile.am eb1944f
src/local/local.hpp 55f9eaf
src/local/local.cpp 8186595
src/master/allocator.hpp 12f31db
src/master/allocator.cpp PRE-CREATION
src/master/allocator_process.hpp PRE-CREATION
src/master/dominant_share_allocator.hpp 650e441
src/master/dominant_share_allocator.cpp a1f2fdc
src/master/main.cpp c321618
src/master/master.hpp e523aa1
src/master/master.cpp f6e5459
src/master/simple_allocator_process.hpp PRE-CREATION
src/master/simple_allocator_process.cpp PRE-CREATION
src/tests/allocator_tests.cpp 610826b
src/tests/fault_tolerance_tests.cpp dd578aa
src/tests/master_detector_tests.cpp afd8a74
src/tests/master_tests.cpp 69a07e9
src/tests/resource_offers_tests.cpp d06cae2
src/tests/slave_tests.cpp a8d6b74
src/tests/utils.hpp ca51af5
Diff: https://reviews.apache.org/r/5599/diff/
Testing
-------
make check on Lion
Thanks,
Thomas Marshall
Re: Review Request: Made DRF per user and log(N)
Posted by Thomas Marshall <tw...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/5599/
-----------------------------------------------------------
(Updated July 10, 2012, 11:39 p.m.)
Review request for mesos and Benjamin Hindman.
Changes
-------
Updated patch to trunk.
Description
-------
Rewrote the DRF algorithm in DominantShareAllocator to calculate shares on a per user, rather than per framework, basis and to store those shares in sorted order so that allocations have log(n) time complexity instead of n^2.
This patch relies on 3 pending code reviews:
https://reviews.apache.org/r/5448/
https://reviews.apache.org/r/5451/
https://reviews.apache.org/r/5532/
This addresses bugs MESOS-225 and MESOS-226.
https://issues.apache.org/jira/browse/MESOS-225
https://issues.apache.org/jira/browse/MESOS-226
Diffs (updated)
-----
src/Makefile.am eb1944f
src/local/local.hpp 55f9eaf
src/local/local.cpp 8186595
src/master/allocator.hpp 12f31db
src/master/allocator.cpp PRE-CREATION
src/master/allocator_process.hpp PRE-CREATION
src/master/dominant_share_allocator.hpp 650e441
src/master/dominant_share_allocator.cpp a1f2fdc
src/master/main.cpp c321618
src/master/master.hpp e523aa1
src/master/master.cpp f6e5459
src/master/simple_allocator_process.hpp PRE-CREATION
src/master/simple_allocator_process.cpp PRE-CREATION
src/tests/allocator_tests.cpp 610826b
src/tests/fault_tolerance_tests.cpp dd578aa
src/tests/master_detector_tests.cpp afd8a74
src/tests/master_tests.cpp 69a07e9
src/tests/resource_offers_tests.cpp d06cae2
src/tests/slave_tests.cpp a8d6b74
src/tests/utils.hpp ca51af5
Diff: https://reviews.apache.org/r/5599/diff/
Testing
-------
make check on Lion
Thanks,
Thomas Marshall
Re: Review Request: Made DRF per user and log(N)
Posted by Thomas Marshall <tw...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/5599/
-----------------------------------------------------------
(Updated July 9, 2012, 10:55 p.m.)
Review request for mesos and Benjamin Hindman.
Changes
-------
Significant restructuring of the allocator code to make writing new allocators easier:
- The Allocator class is renamed AllocatorProcess.
- A new Allocator class is introduced which knows only about "elements" (either user names or frameworkIds) and how many resources they have been allocated, and then has an interface for retrieving the elements in the order they should be allocated to.
- SimpleAllocatorProcess which can be used with different Allocators to carry out different allocation policies, including the possibility of having different policies for the per-user and per-framework allocations.
Why is this restructuring a good thing?
Now that we're doing allocations based on users, rather than frameworks, we have to do two levels of allocation - determine which user to give resources to, then determine which of their frameworks to actually make the offer to. Rather than hard-coding the allocation policies in, the new Allocator class allows us to reuse the same code to do both levels of allocation, and it allows us to easily swap different policies in or mix and match. You can even imagine some day having an allocator that allows each user to pick the per-framework allocation policy they prefer.
Merely giving SimpleAllocatorProcess different Allocators is not sufficient to implement all allocation policies - only those where all resources are considered equal and there is no revocation. But, you can always create new AllocatorProcess subclasses (there will soon be a code review where I implement a static allocator this way) and these new AllocatorProcess classes can still reuse Allocators as appropriate.
Description
-------
Rewrote the DRF algorithm in DominantShareAllocator to calculate shares on a per user, rather than per framework, basis and to store those shares in sorted order so that allocations have log(n) time complexity instead of n^2.
This patch relies on 3 pending code reviews:
https://reviews.apache.org/r/5448/
https://reviews.apache.org/r/5451/
https://reviews.apache.org/r/5532/
This addresses bugs MESOS-225 and MESOS-226.
https://issues.apache.org/jira/browse/MESOS-225
https://issues.apache.org/jira/browse/MESOS-226
Diffs (updated)
-----
src/Makefile.am 8760f59
src/local/local.hpp 55f9eaf
src/local/local.cpp d35639f
src/master/allocator.hpp 0fc61a5
src/master/allocator.cpp PRE-CREATION
src/master/allocator_process.hpp PRE-CREATION
src/master/dominant_share_allocator.hpp PRE-CREATION
src/master/dominant_share_allocator.cpp PRE-CREATION
src/master/main.cpp bcbe35a
src/master/master.hpp 886f79c
src/master/master.cpp 89cdaf6
src/master/simple_allocator_process.hpp PRE-CREATION
src/master/simple_allocator_process.cpp PRE-CREATION
src/tests/allocator_tests.cpp PRE-CREATION
src/tests/fault_tolerance_tests.cpp b2529ca
src/tests/master_detector_tests.cpp 4cefbfa
src/tests/master_tests.cpp fcaf7dc
src/tests/resource_offers_tests.cpp c1f1760
src/tests/slave_tests.cpp 84e8e7d
src/tests/utils.hpp e81ec82
Diff: https://reviews.apache.org/r/5599/diff/
Testing
-------
make check on Lion
Thanks,
Thomas Marshall