[Proposal] Fine Granularity Resource Allocation

["Du, Fan" <fa...@intel.com>]

Hi

Happy new year!

Current resources offering when master performs allocation is 
coarse-grained, i.e. allocating the entire unused resources on slave to 
one framework on each iteration. It’s unable for framework to specify 
the unit of its requested resource, and also after the offering, 
unneeded resources is gave back to master for reallocation again. 
Firstly this process is unnecessary as introducing latency in the 
transaction, moreover, with more logical cpu cores increasing inside one 
physical host, such coarse-grained could possibly bring framework on one 
slave only, this is not robust enough once the slave die. Finally, with 
the unit information of each framework, framework has the ability to 
spread workload across slaves at its will, and choosing the right slave 
for allocation to avoid resource fragmentation could become more 
feasible in the near future.

Could you please review following proposal(my first proposal:) for this?
Thanks a lot.

https://docs.google.com/document/d/1OsdThJ758XgcPnZBcYtPIiXL23l_X1C6XswB-2yhu3k/edit?usp=sharing

Re: [Proposal] Fine Granularity Resource Allocation

["Du, Fan" <fa...@intel.com>]

Hi Guangya

On 2016/1/3 21:15, Guangya Liu wrote:
> There is already a JIRA ticket here
> https://issues.apache.org/jira/browse/MESOS-3765 tracing the
> fine-grained resource offers, after some discussion in community, it is
> suggested to reach an agreement in community before we move forward.
>
> You can append some of your ideas there. Actually for your design
> document, we have discussed this in the JIRA ticket, the conclusion is
> that it is not good to enable framework specify the unit of allocation
> request as fairness cannot be guaranteed/influenced by a framework
> expressing its preferences, since a misbehaving/greedy framework's
> requests will be expressly anti-fair. As such, the requestResources API
> is unrelated, and we should focus on Master/operator-driven granularity
> adjustments.

Ok, let's discuss on Jira.

> I have created a work group for this project and if you are interested
> in this, I can add you to that work group to move this forward.
Please add me in the work group.
So I can contribute to it.
Thanks a lot!

> Thanks,
>
> Guangya
>
>
>
> On Thu, Dec 31, 2015 at 5:00 PM, Du, Fan <fan.du@intel.com
> <ma...@intel.com>> wrote:
>
>     Hi
>
>     Happy new year!
>
>     Current resources offering when master performs allocation is
>     coarse-grained, i.e. allocating the entire unused resources on slave
>     to one framework on each iteration. It’s unable for framework to
>     specify the unit of its requested resource, and also after the
>     offering, unneeded resources is gave back to master for reallocation
>     again. Firstly this process is unnecessary as introducing latency in
>     the transaction, moreover, with more logical cpu cores increasing
>     inside one physical host, such coarse-grained could possibly bring
>     framework on one slave only, this is not robust enough once the
>     slave die. Finally, with the unit information of each framework,
>     framework has the ability to spread workload across slaves at its
>     will, and choosing the right slave for allocation to avoid resource
>     fragmentation could become more feasible in the near future.
>
>     Could you please review following proposal(my first proposal:) for this?
>     Thanks a lot.
>
>     https://docs.google.com/document/d/1OsdThJ758XgcPnZBcYtPIiXL23l_X1C6XswB-2yhu3k/edit?usp=sharing
>
>

Re: [Proposal] Fine Granularity Resource Allocation

[Guangya Liu <gy...@gmail.com>]

There is already a JIRA ticket here
https://issues.apache.org/jira/browse/MESOS-3765 tracing the fine-grained
resource offers, after some discussion in community, it is suggested to
reach an agreement in community before we move forward.

You can append some of your ideas there. Actually for your design document,
we have discussed this in the JIRA ticket, the conclusion is that it is not
good to enable framework specify the unit of allocation request as fairness
cannot be guaranteed/influenced by a framework expressing its preferences,
since a misbehaving/greedy framework's requests will be expressly
anti-fair. As such, the requestResources API is unrelated, and we should
focus on Master/operator-driven granularity adjustments.

I have created a work group for this project and if you are interested in
this, I can add you to that work group to move this forward.
Thanks,

Guangya



On Thu, Dec 31, 2015 at 5:00 PM, Du, Fan <fa...@intel.com> wrote:

> Hi
>
> Happy new year!
>
> Current resources offering when master performs allocation is
> coarse-grained, i.e. allocating the entire unused resources on slave to one
> framework on each iteration. It’s unable for framework to specify the unit
> of its requested resource, and also after the offering, unneeded resources
> is gave back to master for reallocation again. Firstly this process is
> unnecessary as introducing latency in the transaction, moreover, with more
> logical cpu cores increasing inside one physical host, such coarse-grained
> could possibly bring framework on one slave only, this is not robust enough
> once the slave die. Finally, with the unit information of each framework,
> framework has the ability to spread workload across slaves at its will, and
> choosing the right slave for allocation to avoid resource fragmentation
> could become more feasible in the near future.
>
> Could you please review following proposal(my first proposal:) for this?
> Thanks a lot.
>
>
> https://docs.google.com/document/d/1OsdThJ758XgcPnZBcYtPIiXL23l_X1C6XswB-2yhu3k/edit?usp=sharing
>