[jira] [Closed] (HELIX-672) Improve partition distribution strategy for small clusters

["Jiajun Wang (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HELIX-672?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jiajun Wang closed HELIX-672.
-----------------------------
       Resolution: Fixed
    Fix Version/s: 0.8.x

We have new algorithm CRUSH-ed released in 0.8.0

> Improve partition distribution strategy for small clusters
> ----------------------------------------------------------
>
>                 Key: HELIX-672
>                 URL: https://issues.apache.org/jira/browse/HELIX-672
>             Project: Apache Helix
>          Issue Type: Task
>            Reporter: Jiajun Wang
>            Assignee: Jiajun Wang
>            Priority: Major
>             Fix For: 0.8.x
>
>
> Current, CRUSH (also multi-round CRUSH) strategy provides even partition distribution based on statistically randomization. All CRUSH based strategy works well when the cluster is reasonably large.
> However, in some of our small production clusters, we find the distribution is not good as expected. The difference between heavy load node and idle node could be 30% in our case. Given we assign resource according to the max load, we are wasting 30% of resource on the idle nodes. And it could be worse in other clusters.
> We need to find a new algorithm or improve CRUSH for better evenness for small clusters.
> Note that, meanwhile, we should keep the good features such as minimum movements, deterministic calculating result.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)