You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Jeremy Hanna (JIRA)" <ji...@apache.org> on 2010/04/29 22:24:54 UTC

[jira] Created: (CASSANDRA-1037) Improve load balancing to take into account load in terms of operations

Improve load balancing to take into account load in terms of operations
-----------------------------------------------------------------------

                 Key: CASSANDRA-1037
                 URL: https://issues.apache.org/jira/browse/CASSANDRA-1037
             Project: Cassandra
          Issue Type: Improvement
            Reporter: Jeremy Hanna
            Priority: Minor


Currently in cassandra, the load balancing takes into account disk space.  When using an order-preserving partitioner, there can be hot spots in the various ranges of tokens in terms of operations.  We would like to propose improving the load balancing so that it takes that the number of operations into account.

There are two places where this can be handled:

1. when the cluster decides on which nodes need to be balanced out.
2. how to balance an individual node - where to split

For number 1, the number of operations that a node performed could be factored in to how important it is to balance that node.

For number 2, we are already using a midpoint in the node when trying to load balance with respect to space.  We would propose adding a weight to the midpoint to lean towards splitting so that the operational load could be better handled, not just space.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (CASSANDRA-1037) Improve load balancing to take into account load in terms of operations

Posted by "Jeremy Hanna (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/CASSANDRA-1037?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jeremy Hanna updated CASSANDRA-1037:
------------------------------------

    Comment: was deleted

(was: This could also take into account any number of variables - like memory usage.  The number of operations performed just seemed to be the most logical to start with.  Another metric suggested by Eric was memory.  The decision about which node needs to be balanced (1) as well as how to balance (2) could be an aggregate of a few metrics - space, ops, and memory, for example.)

> Improve load balancing to take into account load in terms of operations
> -----------------------------------------------------------------------
>
>                 Key: CASSANDRA-1037
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-1037
>             Project: Cassandra
>          Issue Type: Improvement
>            Reporter: Jeremy Hanna
>            Priority: Minor
>
> Currently in cassandra, the load balancing takes into account disk space.  When using an order-preserving partitioner, there can be hot spots in the various ranges of tokens in terms of operations.  We would like to propose improving the load balancing so that it takes that the number of operations into account.
> There are two places where this can be handled:
> 1. when the cluster decides on which nodes need to be balanced out.
> 2. how to balance an individual node - where to split
> For number 1, the number of operations that a node performed could be factored in to how important it is to balance that node.
> For number 2, we are already using a midpoint in the node when trying to load balance with respect to space.  We would propose adding a weight to the midpoint to lean towards splitting so that the operational load could be better handled, not just space.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (CASSANDRA-1037) Improve load balancing to take into account load in terms of operations

Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/CASSANDRA-1037?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jonathan Ellis updated CASSANDRA-1037:
--------------------------------------

    Fix Version/s: 0.8
                       (was: 0.7)

> Improve load balancing to take into account load in terms of operations
> -----------------------------------------------------------------------
>
>                 Key: CASSANDRA-1037
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-1037
>             Project: Cassandra
>          Issue Type: Improvement
>            Reporter: Jeremy Hanna
>            Priority: Minor
>             Fix For: 0.8
>
>
> Currently in cassandra, the load balancing takes into account disk space.  When using an order-preserving partitioner, there can be hot spots in the various ranges of tokens in terms of operations.  We would like to propose improving the load balancing so that it takes that the number of operations into account.
> There are two places where this can be handled:
> 1. when the cluster decides on which nodes need to be balanced out.
> 2. how to balance an individual node - where to split
> For number 1, the number of operations that a node performed could be factored in to how important it is to balance that node.
> For number 2, we are already using a midpoint in the node when trying to load balance with respect to space.  We would propose adding a weight to the midpoint to lean towards splitting so that the operational load could be better handled, not just space.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (CASSANDRA-1037) Improve load balancing to take into account load in terms of operations

Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/CASSANDRA-1037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12881228#action_12881228 ] 

Jonathan Ellis commented on CASSANDRA-1037:
-------------------------------------------

i'm skeptical that you'd be able to balance fast enough to deal w/ hot spots significantly better than "balance by disk space", but i suppose it's worth a try

> Improve load balancing to take into account load in terms of operations
> -----------------------------------------------------------------------
>
>                 Key: CASSANDRA-1037
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-1037
>             Project: Cassandra
>          Issue Type: Improvement
>            Reporter: Jeremy Hanna
>            Priority: Minor
>             Fix For: 0.7
>
>
> Currently in cassandra, the load balancing takes into account disk space.  When using an order-preserving partitioner, there can be hot spots in the various ranges of tokens in terms of operations.  We would like to propose improving the load balancing so that it takes that the number of operations into account.
> There are two places where this can be handled:
> 1. when the cluster decides on which nodes need to be balanced out.
> 2. how to balance an individual node - where to split
> For number 1, the number of operations that a node performed could be factored in to how important it is to balance that node.
> For number 2, we are already using a midpoint in the node when trying to load balance with respect to space.  We would propose adding a weight to the midpoint to lean towards splitting so that the operational load could be better handled, not just space.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (CASSANDRA-1037) Improve load balancing to take into account load in terms of operations

Posted by "Jeremy Hanna (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/CASSANDRA-1037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12862392#action_12862392 ] 

Jeremy Hanna commented on CASSANDRA-1037:
-----------------------------------------

This could also take into account any number of variables - like memory usage.  The number of operations performed just seemed to be the most logical to start with.  Another metric suggested by Eric was memory.  The decision about which node needs to be balanced (1) as well as how to balance (2) could be an aggregate of a few metrics - space, ops, and memory, for example.

> Improve load balancing to take into account load in terms of operations
> -----------------------------------------------------------------------
>
>                 Key: CASSANDRA-1037
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-1037
>             Project: Cassandra
>          Issue Type: Improvement
>            Reporter: Jeremy Hanna
>            Priority: Minor
>
> Currently in cassandra, the load balancing takes into account disk space.  When using an order-preserving partitioner, there can be hot spots in the various ranges of tokens in terms of operations.  We would like to propose improving the load balancing so that it takes that the number of operations into account.
> There are two places where this can be handled:
> 1. when the cluster decides on which nodes need to be balanced out.
> 2. how to balance an individual node - where to split
> For number 1, the number of operations that a node performed could be factored in to how important it is to balance that node.
> For number 2, we are already using a midpoint in the node when trying to load balance with respect to space.  We would propose adding a weight to the midpoint to lean towards splitting so that the operational load could be better handled, not just space.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (CASSANDRA-1037) Improve load balancing to take into account load in terms of operations

Posted by "Jeremy Hanna (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/CASSANDRA-1037?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jeremy Hanna updated CASSANDRA-1037:
------------------------------------

    Fix Version/s: 0.7

We'll work to help get this in for 0.7, though it doesn't necessarily need to involve api changes.  It is a significant change to behavior though, so we'll try to get this in.

This is a significant item to help with multi-tenant clusters.

> Improve load balancing to take into account load in terms of operations
> -----------------------------------------------------------------------
>
>                 Key: CASSANDRA-1037
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-1037
>             Project: Cassandra
>          Issue Type: Improvement
>            Reporter: Jeremy Hanna
>            Priority: Minor
>             Fix For: 0.7
>
>
> Currently in cassandra, the load balancing takes into account disk space.  When using an order-preserving partitioner, there can be hot spots in the various ranges of tokens in terms of operations.  We would like to propose improving the load balancing so that it takes that the number of operations into account.
> There are two places where this can be handled:
> 1. when the cluster decides on which nodes need to be balanced out.
> 2. how to balance an individual node - where to split
> For number 1, the number of operations that a node performed could be factored in to how important it is to balance that node.
> For number 2, we are already using a midpoint in the node when trying to load balance with respect to space.  We would propose adding a weight to the midpoint to lean towards splitting so that the operational load could be better handled, not just space.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.