[jira] [Commented] (STORM-132) Default Storm scheduler should evenly use resources when managing multiple topologies

["ASF GitHub Bot (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/STORM-132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13904148#comment-13904148 ] 

ASF GitHub Bot commented on STORM-132:
--------------------------------------

GitHub user xiaokang opened a pull request:

    https://github.com/apache/incubator-storm/pull/36

    STORM-132 sort supervisor by free slot in desending order

    Sort supervisor by free slot in desending order in EvenScheduler to schedule more evenly between supervisor

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/xiaokang/incubator-storm STORM-132

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/incubator-storm/pull/36.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #36
    
----
commit 6b275d95fbdbc8374a215ecb2551f0fca3438d81
Author: Kang Xiao <kx...@gmail.com>
Date:   2014-02-18T15:23:50Z

    STORM-132 sort supervisor by free slot in desending order
    in EvenScheduler to schedule more evenly between supervisor

----


> Default Storm scheduler should evenly use resources when managing multiple topologies
> -------------------------------------------------------------------------------------
>
>                 Key: STORM-132
>                 URL: https://issues.apache.org/jira/browse/STORM-132
>             Project: Apache Storm (Incubating)
>          Issue Type: Improvement
>            Reporter: James Xu
>
> https://github.com/nathanmarz/storm/issues/359
> Currently, a single topology is evenly spread across the cluster, but this is not the case for multiple topologies (it targets one node first, then the rest). The default scheduler should order the hosts for scheduling in terms of which has the least slots used.
> ----------
> lyogavin: To confirm, we want to firstly look at the number of used slots on each host. Firstly try to balance the used slot number across all the hosts. For the assignment inside each host, we also try to evenly balance the usage of each slot. Am I understanding correctly?
> When i coding, i realized it's actually a little complicated. Looks like there are many policies here we want to consider:
> 1. Evenness of resource usage. (Do we want to evaluate evenness according to number of slots used in each host or the number of executors? Maybe number of executors is better, but also make it very complicated)
> 2. Least rescheduling. We probabaly also want to make the assignment change as less as possible. Looks like this is why DefaultScheduler.bad-slots is coded the current way.
> 3. Number of workers.
> Then the question is how do we prioritize those policies. Sometimes they conflict to each other. For example, sometimes the most even distribution may need the most reassignment.
> Any thoughts?
> ----------
> xumingming: @lyogavin I think you might have over-thought this, as @nathanmarz already confirmed, you just need to update EvenScheduler.sort-slots to make sure the slots in the least used node appear first in the available-slots list.
> ----------
> lyogavin: Thanks James. I got what you mean. So i'll not worry about the usage of the executor usage. Only consider the balance of slots usage.
> But I think just simply change the sort-slots to sort the slots based on slots usage may still not work too well. For example, let's say there are 2 hosts, with 10 slots on each. Say 1st one used 1 slots, 2nd one used 2 slots, and we want to assign another topology with 8 workers. If we simply sort the slots based on usage, we'll end up with the list [0 0 0 0 0 0 0 0 1 1], then the new assignment we'll get would be all from the 1st host. Not balanced.
> So in the above pull request, I implemented a solution in the way similar to watershed algorithm. It would firstly pick the slots from least used host, until that host uses the same number of slots as the second least used slots. Then it evenly picks slots from the 2 least used hosts until reaches the 3rd one. Iterating this way, we can get the best balanced assignment.
> What do you think?



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)