You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Jonathan Ellis (JIRA)" <ji...@apache.org> on 2010/06/03 07:53:55 UTC

[jira] Created: (CASSANDRA-1157) add support for multiple indexexpressions

add support for multiple indexexpressions
-----------------------------------------

                 Key: CASSANDRA-1157
                 URL: https://issues.apache.org/jira/browse/CASSANDRA-1157
             Project: Cassandra
          Issue Type: Sub-task
            Reporter: Jonathan Ellis
             Fix For: 0.7


we should:

 - use the statistics from CASSANDRA-1155 to figure out which index has the highest selectivity, and start with that
 - if other indexes have high selectivity (average number of columns in an index row is less than 1% of total in that CF), we should do a merge join
 - otherwise, just loop the results from the first and reject un-satisfied expressions


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (CASSANDRA-1157) add support for multiple indexexpressions

Posted by "Nate McCall (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/CASSANDRA-1157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12892014#action_12892014 ] 

Nate McCall commented on CASSANDRA-1157:
----------------------------------------

"The problem was a bug in the system test - it wasn't actually rebuilding the RowPredicate, "  see #1 above :-)

v2 applies cleanly after rebase and passes nosetests

> add support for multiple indexexpressions
> -----------------------------------------
>
>                 Key: CASSANDRA-1157
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-1157
>             Project: Cassandra
>          Issue Type: Sub-task
>            Reporter: Jonathan Ellis
>            Assignee: Jonathan Ellis
>             Fix For: 0.7 beta 1
>
>         Attachments: 1157-v2.txt, 1157.txt, 1157_thrift_validation_fix_w_tests.txt
>
>
> we should:
>  - use the statistics from CASSANDRA-1155 to figure out which index has the highest selectivity, and start with that
>  - if other indexes have high selectivity (average number of columns in an index row is less than 1% of total in that CF), we should do a merge join
>  - otherwise, just loop the results from the first and reject un-satisfied expressions

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (CASSANDRA-1157) add support for multiple indexexpressions

Posted by "Stu Hood (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/CASSANDRA-1157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12906229#action_12906229 ] 

Stu Hood commented on CASSANDRA-1157:
-------------------------------------

Sorry to chime in so late (and probably on the wrong ticket), but it doesn't look like get_indexed_slices needs to be a separate method from get_range_slices.

IndexClause only has a start key for get_indexed_slices, but it would seem that the reasoning behind using 'KeyRange' for get_range_slices applies there as well, since if you know the range you care about in the primary index, you don't want to continue scanning until you exhaust 'count' (or the cluster).

Since it would appear that get_indexed_slices would benefit from a KeyRange, why not smash get_(range|indexed)_slices together, and make IndexClause an optional field on KeyRange?

> add support for multiple indexexpressions
> -----------------------------------------
>
>                 Key: CASSANDRA-1157
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-1157
>             Project: Cassandra
>          Issue Type: Sub-task
>            Reporter: Jonathan Ellis
>            Assignee: Jonathan Ellis
>             Fix For: 0.7 beta 1
>
>         Attachments: 1157-v2.txt, 1157.txt, 1157_thrift_validation_fix_w_tests.txt
>
>
> we should:
>  - use the statistics from CASSANDRA-1155 to figure out which index has the highest selectivity, and start with that
>  - if other indexes have high selectivity (average number of columns in an index row is less than 1% of total in that CF), we should do a merge join
>  - otherwise, just loop the results from the first and reject un-satisfied expressions

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (CASSANDRA-1157) add support for multiple indexexpressions

Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/CASSANDRA-1157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12891807#action_12891807 ] 

Jonathan Ellis commented on CASSANDRA-1157:
-------------------------------------------

punting on merge join for now.  otherwise, patch works as described above.

> add support for multiple indexexpressions
> -----------------------------------------
>
>                 Key: CASSANDRA-1157
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-1157
>             Project: Cassandra
>          Issue Type: Sub-task
>            Reporter: Jonathan Ellis
>            Assignee: Jonathan Ellis
>             Fix For: 0.7 beta 1
>
>         Attachments: 1157.txt
>
>
> we should:
>  - use the statistics from CASSANDRA-1155 to figure out which index has the highest selectivity, and start with that
>  - if other indexes have high selectivity (average number of columns in an index row is less than 1% of total in that CF), we should do a merge join
>  - otherwise, just loop the results from the first and reject un-satisfied expressions

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (CASSANDRA-1157) add support for multiple indexexpressions

Posted by "Nate McCall (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/CASSANDRA-1157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12891893#action_12891893 ] 

Nate McCall commented on CASSANDRA-1157:
----------------------------------------

test_thrift_server fails with the message below.

There are two issues here though:

1) it looks like the RowPredicate is not getting reset with the IndexClause (ln: 1251), when I add that, I get a similar error from the next scan about "unable to scan unindexed column" again because RowPredicate is not reset. After reseting, I still get the same error. 

2) Though the system test implies indexed and non-indexed will play nice on the same IndexExpression, ThriftValidation does not agree, throwing an exception on the first missing column encountered.


Initial error after rebase and patches applied:
~~~~

nate:cassandra-TRUNK$ python=test nosetests --tests=system.test_thrift_server:TestMutations.test_index_scan
/usr/local/lib/python2.6/dist-packages/thrift/Thrift.py:58: DeprecationWarning: BaseException.message has been deprecated as of Python 2.6
  self.message = message
E
======================================================================
ERROR: system.test_thrift_server.TestMutations.test_index_scan
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/var/lib/python-support/python2.6/nose/case.py", line 182, in runTest
    self.test(*self.arg)
  File "/home/nate/workspace/cassandra-TRUNK/test/system/test_thrift_server.py", line 1252, in test_index_scan
    _expect_exception(lambda: client.scan(cp, rp, sp, ConsistencyLevel.ONE), InvalidRequestException)
  File "/home/nate/workspace/cassandra-TRUNK/test/system/test_thrift_server.py", line 173, in _expect_exception
    raise Exception('expected %s; got %s' % (type_.__name__, r))
Exception: expected InvalidRequestException; got [KeySlice(columns=[ColumnOrSuperColumn(column=Column(ttl=None, name='birthdate', value='\x00\x00\x00\x00\x00\x00\x00\x01', clock=Clock(timestamp=0)), super_column=None)], key='key1')]

----------------------------------------------------------------------
Ran 1 test in 2.430s

> add support for multiple indexexpressions
> -----------------------------------------
>
>                 Key: CASSANDRA-1157
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-1157
>             Project: Cassandra
>          Issue Type: Sub-task
>            Reporter: Jonathan Ellis
>            Assignee: Jonathan Ellis
>             Fix For: 0.7 beta 1
>
>         Attachments: 1157.txt
>
>
> we should:
>  - use the statistics from CASSANDRA-1155 to figure out which index has the highest selectivity, and start with that
>  - if other indexes have high selectivity (average number of columns in an index row is less than 1% of total in that CF), we should do a merge join
>  - otherwise, just loop the results from the first and reject un-satisfied expressions

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (CASSANDRA-1157) add support for multiple indexexpressions

Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/CASSANDRA-1157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jonathan Ellis updated CASSANDRA-1157:
--------------------------------------

    Attachment: 1157-v2.txt

I did correct TV to allow unindexed expressions as long as there was a single indexed one.  The problem was a bug in the system test -- it wasn't actually rebuilding the RowPredicate, so each call was using the original single-expressing clause.  v2 attached.

(Your TV change, btw, would allow queries with no indexed expressions, as long as there was more than one undexed one.)

> add support for multiple indexexpressions
> -----------------------------------------
>
>                 Key: CASSANDRA-1157
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-1157
>             Project: Cassandra
>          Issue Type: Sub-task
>            Reporter: Jonathan Ellis
>            Assignee: Jonathan Ellis
>             Fix For: 0.7 beta 1
>
>         Attachments: 1157-v2.txt, 1157.txt, 1157_thrift_validation_fix_w_tests.txt
>
>
> we should:
>  - use the statistics from CASSANDRA-1155 to figure out which index has the highest selectivity, and start with that
>  - if other indexes have high selectivity (average number of columns in an index row is less than 1% of total in that CF), we should do a merge join
>  - otherwise, just loop the results from the first and reject un-satisfied expressions

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (CASSANDRA-1157) add support for multiple indexexpressions

Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/CASSANDRA-1157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jonathan Ellis updated CASSANDRA-1157:
--------------------------------------

    Attachment: 1157.txt

> add support for multiple indexexpressions
> -----------------------------------------
>
>                 Key: CASSANDRA-1157
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-1157
>             Project: Cassandra
>          Issue Type: Sub-task
>            Reporter: Jonathan Ellis
>             Fix For: 0.7 beta 1
>
>         Attachments: 1157.txt
>
>
> we should:
>  - use the statistics from CASSANDRA-1155 to figure out which index has the highest selectivity, and start with that
>  - if other indexes have high selectivity (average number of columns in an index row is less than 1% of total in that CF), we should do a merge join
>  - otherwise, just loop the results from the first and reject un-satisfied expressions

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (CASSANDRA-1157) add support for multiple indexexpressions

Posted by "Hudson (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/CASSANDRA-1157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12892769#action_12892769 ] 

Hudson commented on CASSANDRA-1157:
-----------------------------------

Integrated in Cassandra #501 (See [http://hudson.zones.apache.org/hudson/job/Cassandra/501/])
    

> add support for multiple indexexpressions
> -----------------------------------------
>
>                 Key: CASSANDRA-1157
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-1157
>             Project: Cassandra
>          Issue Type: Sub-task
>            Reporter: Jonathan Ellis
>            Assignee: Jonathan Ellis
>             Fix For: 0.7 beta 1
>
>         Attachments: 1157-v2.txt, 1157.txt, 1157_thrift_validation_fix_w_tests.txt
>
>
> we should:
>  - use the statistics from CASSANDRA-1155 to figure out which index has the highest selectivity, and start with that
>  - if other indexes have high selectivity (average number of columns in an index row is less than 1% of total in that CF), we should do a merge join
>  - otherwise, just loop the results from the first and reject un-satisfied expressions

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (CASSANDRA-1157) add support for multiple indexexpressions

Posted by "Nate McCall (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/CASSANDRA-1157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Nate McCall updated CASSANDRA-1157:
-----------------------------------

    Attachment: 1157_thrift_validation_fix_w_tests.txt

modified thrift validation to allow for indexed and unindexed columns in the same expression as long as there are more than one valid one

> add support for multiple indexexpressions
> -----------------------------------------
>
>                 Key: CASSANDRA-1157
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-1157
>             Project: Cassandra
>          Issue Type: Sub-task
>            Reporter: Jonathan Ellis
>            Assignee: Jonathan Ellis
>             Fix For: 0.7 beta 1
>
>         Attachments: 1157.txt, 1157_thrift_validation_fix_w_tests.txt
>
>
> we should:
>  - use the statistics from CASSANDRA-1155 to figure out which index has the highest selectivity, and start with that
>  - if other indexes have high selectivity (average number of columns in an index row is less than 1% of total in that CF), we should do a merge join
>  - otherwise, just loop the results from the first and reject un-satisfied expressions

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.