You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Jackson Chung (JIRA)" <ji...@apache.org> on 2011/07/19 01:24:57 UTC

[jira] [Created] (CASSANDRA-2917) expose calculate midrange for token in jmx

expose calculate midrange for token in jmx
------------------------------------------

                 Key: CASSANDRA-2917
                 URL: https://issues.apache.org/jira/browse/CASSANDRA-2917
             Project: Cassandra
          Issue Type: Improvement
            Reporter: Jackson Chung
            Priority: Minor


currently there is no easy way to get midrange, especially for OPP. For simplicity, you could call OrderPreservingPartitioner.midpoint(Token, Token), that gives you a rough estimate (and you'd still need to remove non-utf8 characters.)

A more accurate but difficult way is to sample the keys in that range and pick the midpoint of those. We should expose that via jmx, because without this, supporting OPP w/o this is quite challenging.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Updated] (CASSANDRA-2917) expose calculate midrange for token in jmx

Posted by "Sam Tunnicliffe (Updated) (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/CASSANDRA-2917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Sam Tunnicliffe updated CASSANDRA-2917:
---------------------------------------

    Attachment:     (was: trunk-2917.txt)
    
> expose calculate midrange for token in jmx
> ------------------------------------------
>
>                 Key: CASSANDRA-2917
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-2917
>             Project: Cassandra
>          Issue Type: Improvement
>            Reporter: Jackson Chung
>            Priority: Minor
>              Labels: lhf
>
> currently there is no easy way to get midrange, especially for OPP. For simplicity, you could call OrderPreservingPartitioner.midpoint(Token, Token), that gives you a rough estimate (and you'd still need to remove non-utf8 characters.)
> A more accurate but difficult way is to sample the keys in that range and pick the midpoint of those. We should expose that via jmx, because without this, supporting OPP w/o this is quite challenging.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (CASSANDRA-2917) expose calculate midrange for token in jmx

Posted by "Brandon Williams (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/CASSANDRA-2917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13067491#comment-13067491 ] 

Brandon Williams commented on CASSANDRA-2917:
---------------------------------------------

Note that non-utf8 chars are still a problem this way; snakeyaml will complain when you add them as the initial_token.

> expose calculate midrange for token in jmx
> ------------------------------------------
>
>                 Key: CASSANDRA-2917
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-2917
>             Project: Cassandra
>          Issue Type: Improvement
>            Reporter: Jackson Chung
>            Priority: Minor
>              Labels: lhf
>
> currently there is no easy way to get midrange, especially for OPP. For simplicity, you could call OrderPreservingPartitioner.midpoint(Token, Token), that gives you a rough estimate (and you'd still need to remove non-utf8 characters.)
> A more accurate but difficult way is to sample the keys in that range and pick the midpoint of those. We should expose that via jmx, because without this, supporting OPP w/o this is quite challenging.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (CASSANDRA-2917) expose calculate midrange for token in jmx

Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/CASSANDRA-2917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13067495#comment-13067495 ] 

Jonathan Ellis commented on CASSANDRA-2917:
-------------------------------------------

Dealing with the non-utf8 characters is the responsibility of whoever calls this.  For most cases I imagine you could just strip them out, or you could "round" to the nearest utf8.

All we want this ticket to do is expose the sampling Cassandra has, so we can do the rest externally.

> expose calculate midrange for token in jmx
> ------------------------------------------
>
>                 Key: CASSANDRA-2917
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-2917
>             Project: Cassandra
>          Issue Type: Improvement
>            Reporter: Jackson Chung
>            Priority: Minor
>              Labels: lhf
>
> currently there is no easy way to get midrange, especially for OPP. For simplicity, you could call OrderPreservingPartitioner.midpoint(Token, Token), that gives you a rough estimate (and you'd still need to remove non-utf8 characters.)
> A more accurate but difficult way is to sample the keys in that range and pick the midpoint of those. We should expose that via jmx, because without this, supporting OPP w/o this is quite challenging.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (CASSANDRA-2917) expose calculate midrange for token in jmx

Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/CASSANDRA-2917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13067465#comment-13067465 ] 

Jonathan Ellis commented on CASSANDRA-2917:
-------------------------------------------

Easy: expose SS.getBootstrapToken

More work but more useful: expose this code instead:

{code}
        Range range = getLocalPrimaryRange();
        List<DecoratedKey> keys = new ArrayList<DecoratedKey>();
        for (ColumnFamilyStore cfs : ColumnFamilyStore.all())
        {
            for (DecoratedKey key : cfs.allKeySamples())
            {
                if (range.contains(key.token))
                    keys.add(key);
            }
        }
        FBUtilities.sortSampledKeys(keys, range);
{code}

This would allow more sophisticated rebalancing for [B]OPP clusters.

> expose calculate midrange for token in jmx
> ------------------------------------------
>
>                 Key: CASSANDRA-2917
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-2917
>             Project: Cassandra
>          Issue Type: Improvement
>            Reporter: Jackson Chung
>            Priority: Minor
>              Labels: lhf
>
> currently there is no easy way to get midrange, especially for OPP. For simplicity, you could call OrderPreservingPartitioner.midpoint(Token, Token), that gives you a rough estimate (and you'd still need to remove non-utf8 characters.)
> A more accurate but difficult way is to sample the keys in that range and pick the midpoint of those. We should expose that via jmx, because without this, supporting OPP w/o this is quite challenging.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Updated] (CASSANDRA-2917) expose calculate midrange for token in jmx

Posted by "Sam Tunnicliffe (Updated) (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/CASSANDRA-2917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Sam Tunnicliffe updated CASSANDRA-2917:
---------------------------------------

    Attachment: trunk-2917-1.txt
    
> expose calculate midrange for token in jmx
> ------------------------------------------
>
>                 Key: CASSANDRA-2917
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-2917
>             Project: Cassandra
>          Issue Type: Improvement
>          Components: Tools
>            Reporter: Jackson Chung
>            Assignee: Sam Tunnicliffe
>            Priority: Minor
>              Labels: lhf
>             Fix For: 1.1
>
>         Attachments: trunk-2917-1.txt, trunk-2917.txt
>
>
> currently there is no easy way to get midrange, especially for OPP. For simplicity, you could call OrderPreservingPartitioner.midpoint(Token, Token), that gives you a rough estimate (and you'd still need to remove non-utf8 characters.)
> A more accurate but difficult way is to sample the keys in that range and pick the midpoint of those. We should expose that via jmx, because without this, supporting OPP w/o this is quite challenging.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (CASSANDRA-2917) expose calculate midrange for token in jmx

Posted by "Brandon Williams (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/CASSANDRA-2917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13068083#comment-13068083 ] 

Brandon Williams commented on CASSANDRA-2917:
---------------------------------------------

It looks like getBootstrapToken already does the sampling, but falls back to midpoint when there are less than 3 keys sampled.

> expose calculate midrange for token in jmx
> ------------------------------------------
>
>                 Key: CASSANDRA-2917
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-2917
>             Project: Cassandra
>          Issue Type: Improvement
>            Reporter: Jackson Chung
>            Priority: Minor
>              Labels: lhf
>
> currently there is no easy way to get midrange, especially for OPP. For simplicity, you could call OrderPreservingPartitioner.midpoint(Token, Token), that gives you a rough estimate (and you'd still need to remove non-utf8 characters.)
> A more accurate but difficult way is to sample the keys in that range and pick the midpoint of those. We should expose that via jmx, because without this, supporting OPP w/o this is quite challenging.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Updated] (CASSANDRA-2917) expose calculate midrange for token in jmx

Posted by "Sam Tunnicliffe (Updated) (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/CASSANDRA-2917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Sam Tunnicliffe updated CASSANDRA-2917:
---------------------------------------

    Attachment: trunk-2917.txt

Updated patch to include new command in NodeCmd. 
                
> expose calculate midrange for token in jmx
> ------------------------------------------
>
>                 Key: CASSANDRA-2917
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-2917
>             Project: Cassandra
>          Issue Type: Improvement
>            Reporter: Jackson Chung
>            Priority: Minor
>              Labels: lhf
>         Attachments: trunk-2917.txt
>
>
> currently there is no easy way to get midrange, especially for OPP. For simplicity, you could call OrderPreservingPartitioner.midpoint(Token, Token), that gives you a rough estimate (and you'd still need to remove non-utf8 characters.)
> A more accurate but difficult way is to sample the keys in that range and pick the midpoint of those. We should expose that via jmx, because without this, supporting OPP w/o this is quite challenging.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (CASSANDRA-2917) expose calculate midrange for token in jmx

Posted by "Brandon Williams (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/CASSANDRA-2917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13067518#comment-13067518 ] 

Brandon Williams commented on CASSANDRA-2917:
---------------------------------------------

Agreed, just pointing this out for anyone who expects this to be a simple c&p from the output.

> expose calculate midrange for token in jmx
> ------------------------------------------
>
>                 Key: CASSANDRA-2917
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-2917
>             Project: Cassandra
>          Issue Type: Improvement
>            Reporter: Jackson Chung
>            Priority: Minor
>              Labels: lhf
>
> currently there is no easy way to get midrange, especially for OPP. For simplicity, you could call OrderPreservingPartitioner.midpoint(Token, Token), that gives you a rough estimate (and you'd still need to remove non-utf8 characters.)
> A more accurate but difficult way is to sample the keys in that range and pick the midpoint of those. We should expose that via jmx, because without this, supporting OPP w/o this is quite challenging.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (CASSANDRA-2917) expose calculate midrange for token in jmx

Posted by "Jonathan Ellis (Commented) (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/CASSANDRA-2917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13202970#comment-13202970 ] 

Jonathan Ellis commented on CASSANDRA-2917:
-------------------------------------------

Thanks for the patch, Sam!

I think that like the bootstrap token calculator we want to skip the System keyspace here.  I've added a followup patch that does that over on https://github.com/jbellis/cassandra/tree/2917, with some refactoring to use the more-efficient CFS.keySamples method.  What do you think?

(P.S. As a style note, I also dropped the Transform business -- while I'm a fan of functional programmming, it's not always a great fit in Java and for simple cases like this is substantially more verbose than the imperative version.)

                
> expose calculate midrange for token in jmx
> ------------------------------------------
>
>                 Key: CASSANDRA-2917
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-2917
>             Project: Cassandra
>          Issue Type: Improvement
>            Reporter: Jackson Chung
>            Priority: Minor
>              Labels: lhf
>         Attachments: trunk-2917.txt
>
>
> currently there is no easy way to get midrange, especially for OPP. For simplicity, you could call OrderPreservingPartitioner.midpoint(Token, Token), that gives you a rough estimate (and you'd still need to remove non-utf8 characters.)
> A more accurate but difficult way is to sample the keys in that range and pick the midpoint of those. We should expose that via jmx, because without this, supporting OPP w/o this is quite challenging.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Updated] (CASSANDRA-2917) expose calculate midrange for token in jmx

Posted by "Sam Tunnicliffe (Updated) (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/CASSANDRA-2917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Sam Tunnicliffe updated CASSANDRA-2917:
---------------------------------------

    Attachment: trunk-2917.txt
    
> expose calculate midrange for token in jmx
> ------------------------------------------
>
>                 Key: CASSANDRA-2917
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-2917
>             Project: Cassandra
>          Issue Type: Improvement
>            Reporter: Jackson Chung
>            Priority: Minor
>              Labels: lhf
>         Attachments: trunk-2917.txt
>
>
> currently there is no easy way to get midrange, especially for OPP. For simplicity, you could call OrderPreservingPartitioner.midpoint(Token, Token), that gives you a rough estimate (and you'd still need to remove non-utf8 characters.)
> A more accurate but difficult way is to sample the keys in that range and pick the midpoint of those. We should expose that via jmx, because without this, supporting OPP w/o this is quite challenging.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Updated] (CASSANDRA-2917) expose calculate midrange for token in jmx

Posted by "Jonathan Ellis (Updated) (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/CASSANDRA-2917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jonathan Ellis updated CASSANDRA-2917:
--------------------------------------

         Reviewer: jbellis
      Component/s: Tools
    Fix Version/s: 1.1
         Assignee: Sam Tunnicliffe
    
> expose calculate midrange for token in jmx
> ------------------------------------------
>
>                 Key: CASSANDRA-2917
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-2917
>             Project: Cassandra
>          Issue Type: Improvement
>          Components: Tools
>            Reporter: Jackson Chung
>            Assignee: Sam Tunnicliffe
>            Priority: Minor
>              Labels: lhf
>             Fix For: 1.1
>
>         Attachments: trunk-2917.txt
>
>
> currently there is no easy way to get midrange, especially for OPP. For simplicity, you could call OrderPreservingPartitioner.midpoint(Token, Token), that gives you a rough estimate (and you'd still need to remove non-utf8 characters.)
> A more accurate but difficult way is to sample the keys in that range and pick the midpoint of those. We should expose that via jmx, because without this, supporting OPP w/o this is quite challenging.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (CASSANDRA-2917) expose calculate midrange for token in jmx

Posted by "Jonathan Ellis (Commented) (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/CASSANDRA-2917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13204129#comment-13204129 ] 

Jonathan Ellis commented on CASSANDRA-2917:
-------------------------------------------

bq. I agree its very verbose and only went down that route as I saw them used in a few other places in the codebase

Sorry about that. :)

bq. what's the best branch to work off when creating small patches like this

When in doubt, stick with trunk. In this case I'm comfortable committing to 1.1 as well since it's pretty self-contained.
                
> expose calculate midrange for token in jmx
> ------------------------------------------
>
>                 Key: CASSANDRA-2917
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-2917
>             Project: Cassandra
>          Issue Type: Improvement
>          Components: Tools
>            Reporter: Jackson Chung
>            Assignee: Sam Tunnicliffe
>            Priority: Minor
>              Labels: lhf
>             Fix For: 1.1
>
>         Attachments: trunk-2917-1.txt, trunk-2917.txt
>
>
> currently there is no easy way to get midrange, especially for OPP. For simplicity, you could call OrderPreservingPartitioner.midpoint(Token, Token), that gives you a rough estimate (and you'd still need to remove non-utf8 characters.)
> A more accurate but difficult way is to sample the keys in that range and pick the midpoint of those. We should expose that via jmx, because without this, supporting OPP w/o this is quite challenging.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Updated] (CASSANDRA-2917) expose calculate midrange for token in jmx

Posted by "Brandon Williams (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/CASSANDRA-2917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Brandon Williams updated CASSANDRA-2917:
----------------------------------------

    Labels: lhf  (was: )

> expose calculate midrange for token in jmx
> ------------------------------------------
>
>                 Key: CASSANDRA-2917
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-2917
>             Project: Cassandra
>          Issue Type: Improvement
>            Reporter: Jackson Chung
>            Priority: Minor
>              Labels: lhf
>
> currently there is no easy way to get midrange, especially for OPP. For simplicity, you could call OrderPreservingPartitioner.midpoint(Token, Token), that gives you a rough estimate (and you'd still need to remove non-utf8 characters.)
> A more accurate but difficult way is to sample the keys in that range and pick the midpoint of those. We should expose that via jmx, because without this, supporting OPP w/o this is quite challenging.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (CASSANDRA-2917) expose calculate midrange for token in jmx

Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/CASSANDRA-2917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13068087#comment-13068087 ] 

Jonathan Ellis commented on CASSANDRA-2917:
-------------------------------------------

Right, that's why I said "Easy: expose gBT."

But again, more useful to expose the sampling data so you can do things like "divide it into thirds so I can add two nodes."

> expose calculate midrange for token in jmx
> ------------------------------------------
>
>                 Key: CASSANDRA-2917
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-2917
>             Project: Cassandra
>          Issue Type: Improvement
>            Reporter: Jackson Chung
>            Priority: Minor
>              Labels: lhf
>
> currently there is no easy way to get midrange, especially for OPP. For simplicity, you could call OrderPreservingPartitioner.midpoint(Token, Token), that gives you a rough estimate (and you'd still need to remove non-utf8 characters.)
> A more accurate but difficult way is to sample the keys in that range and pick the midpoint of those. We should expose that via jmx, because without this, supporting OPP w/o this is quite challenging.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (CASSANDRA-2917) expose calculate midrange for token in jmx

Posted by "Sam Tunnicliffe (Commented) (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/CASSANDRA-2917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13203558#comment-13203558 ] 

Sam Tunnicliffe commented on CASSANDRA-2917:
--------------------------------------------

Thanks Jonathan, your updated patch is a big improvement. I had wondered about excluding the System keyspace, or even giving the getRangeKeySample a Keyspace argument. 

+1 on removing the Transform, I agree its very verbose and only went down that route as I saw them used in a few other places in the codebase. I've cherry picked your change & squashed the two to create a new patch, is that the way things usually get handled when patches are reworked? (I did see the mail on the dev list a few weeks back regarding rebase vs merge, but I have to admit I stopped following it after a while).

Out of interest, what's the best branch to work off when creating small patches like this? I branched off trunk initially, but it looks like your 2917 branch didn't? 
                
> expose calculate midrange for token in jmx
> ------------------------------------------
>
>                 Key: CASSANDRA-2917
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-2917
>             Project: Cassandra
>          Issue Type: Improvement
>          Components: Tools
>            Reporter: Jackson Chung
>            Assignee: Sam Tunnicliffe
>            Priority: Minor
>              Labels: lhf
>             Fix For: 1.1
>
>         Attachments: trunk-2917.txt
>
>
> currently there is no easy way to get midrange, especially for OPP. For simplicity, you could call OrderPreservingPartitioner.midpoint(Token, Token), that gives you a rough estimate (and you'd still need to remove non-utf8 characters.)
> A more accurate but difficult way is to sample the keys in that range and pick the midpoint of those. We should expose that via jmx, because without this, supporting OPP w/o this is quite challenging.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira