You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Jonathan Ellis (JIRA)" <ji...@apache.org> on 2011/07/13 01:11:00 UTC
[jira] [Created] (CASSANDRA-2894) add paging to get_count
add paging to get_count
-----------------------
Key: CASSANDRA-2894
URL: https://issues.apache.org/jira/browse/CASSANDRA-2894
Project: Cassandra
Issue Type: Improvement
Components: API
Reporter: Jonathan Ellis
Priority: Minor
Fix For: 1.0
It is non-intuitive that get_count materializes the entire slice-to-count on the coordinator node (to perform read repair and > CL.ONE consistency). Even experienced users have been known to cause memory problems by requesting large counts.
The user cannot page the count himself, because you need a start and stop column to do that, and get_count only returns an integer.
So the best fix is for us to do the paging under the hood, in CassandraServer. Add a limit to the slicepredicate they specify, and page through it.
We could add a global setting for count_slice_size, and document that counts of more columns than that will have higher latency (because they make multiple calls through StorageProxy for the pages).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-2894) add paging to get_count
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-2894:
--------------------------------------
Attachment: 2894-formatted.txt
Looks good to me. Attached version w/ minor formatting changes.
Two comments:
- I think this stomps on the count setting in the predicate, if one is present. Granted, that's an unusual thing to do, but it's valid.
- Can you add a system test (test/system/test_thrift_server.py) exercising paging?
> add paging to get_count
> -----------------------
>
> Key: CASSANDRA-2894
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2894
> Project: Cassandra
> Issue Type: Improvement
> Components: API
> Reporter: Jonathan Ellis
> Priority: Minor
> Labels: lhf
> Fix For: 1.0
>
> Attachments: 2894-formatted.txt, CASSANDRA-2894.patch
>
>
> It is non-intuitive that get_count materializes the entire slice-to-count on the coordinator node (to perform read repair and > CL.ONE consistency). Even experienced users have been known to cause memory problems by requesting large counts.
> The user cannot page the count himself, because you need a start and stop column to do that, and get_count only returns an integer.
> So the best fix is for us to do the paging under the hood, in CassandraServer. Add a limit to the slicepredicate they specify, and page through it.
> We could add a global setting for count_slice_size, and document that counts of more columns than that will have higher latency (because they make multiple calls through StorageProxy for the pages).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (CASSANDRA-2894) add paging to get_count
Posted by "Vijay (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vijay reassigned CASSANDRA-2894:
--------------------------------
Assignee: (was: Vijay)
Moving myself out of the Assignee list...
> add paging to get_count
> -----------------------
>
> Key: CASSANDRA-2894
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2894
> Project: Cassandra
> Issue Type: Improvement
> Components: API
> Reporter: Jonathan Ellis
> Priority: Minor
> Labels: lhf
> Fix For: 1.0
>
> Attachments: CASSANDRA-2894.patch
>
>
> It is non-intuitive that get_count materializes the entire slice-to-count on the coordinator node (to perform read repair and > CL.ONE consistency). Even experienced users have been known to cause memory problems by requesting large counts.
> The user cannot page the count himself, because you need a start and stop column to do that, and get_count only returns an integer.
> So the best fix is for us to do the paging under the hood, in CassandraServer. Add a limit to the slicepredicate they specify, and page through it.
> We could add a global setting for count_slice_size, and document that counts of more columns than that will have higher latency (because they make multiple calls through StorageProxy for the pages).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2894) add paging to get_count
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13073306#comment-13073306 ]
Jonathan Ellis commented on CASSANDRA-2894:
-------------------------------------------
Over a minute? Ouch. :)
Is the time from the setup or the get_count calls? Suspect doing the inserts in a batch will help a lot.
> add paging to get_count
> -----------------------
>
> Key: CASSANDRA-2894
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2894
> Project: Cassandra
> Issue Type: Improvement
> Components: API
> Reporter: Jonathan Ellis
> Priority: Minor
> Labels: lhf
> Fix For: 1.0
>
> Attachments: 2894-formatted.txt, 2894-with-test.txt, CASSANDRA-2894.patch
>
>
> It is non-intuitive that get_count materializes the entire slice-to-count on the coordinator node (to perform read repair and > CL.ONE consistency). Even experienced users have been known to cause memory problems by requesting large counts.
> The user cannot page the count himself, because you need a start and stop column to do that, and get_count only returns an integer.
> So the best fix is for us to do the paging under the hood, in CassandraServer. Add a limit to the slicepredicate they specify, and page through it.
> We could add a global setting for count_slice_size, and document that counts of more columns than that will have higher latency (because they make multiple calls through StorageProxy for the pages).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-2894) add paging to get_count
Posted by "Byron Clark (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Byron Clark updated CASSANDRA-2894:
-----------------------------------
Attachment: 2894-with-test.txt
[^2894-with-test.txt] incorporates Jonathan's changes and adds a system test to exercise paging. The bad news is that the test takes over a minute to run on my machine.
This patch also handles (and tests for) the case where count is set to something other than Integer.MAX_VALUE on the SliceRange.
> add paging to get_count
> -----------------------
>
> Key: CASSANDRA-2894
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2894
> Project: Cassandra
> Issue Type: Improvement
> Components: API
> Reporter: Jonathan Ellis
> Priority: Minor
> Labels: lhf
> Fix For: 1.0
>
> Attachments: 2894-formatted.txt, 2894-with-test.txt, CASSANDRA-2894.patch
>
>
> It is non-intuitive that get_count materializes the entire slice-to-count on the coordinator node (to perform read repair and > CL.ONE consistency). Even experienced users have been known to cause memory problems by requesting large counts.
> The user cannot page the count himself, because you need a start and stop column to do that, and get_count only returns an integer.
> So the best fix is for us to do the paging under the hood, in CassandraServer. Add a limit to the slicepredicate they specify, and page through it.
> We could add a global setting for count_slice_size, and document that counts of more columns than that will have higher latency (because they make multiple calls through StorageProxy for the pages).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2894) add paging to get_count
Posted by "Byron Clark (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13071302#comment-13071302 ]
Byron Clark commented on CASSANDRA-2894:
----------------------------------------
[^CASSANDRA-2894.patch] implements this behavior without the global setting for count_slice_size.
Behavior is based on the paging done in HintedHandoffManager from the 0.8 branch. The maximum page size is 16384 columns, largely because I have no idea what a good value for that setting would be.
Assumptions made:
- Using the last column returned from get_slice as the starting column for the next call to get_slice won't skip over any columns.
> add paging to get_count
> -----------------------
>
> Key: CASSANDRA-2894
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2894
> Project: Cassandra
> Issue Type: Improvement
> Components: API
> Reporter: Jonathan Ellis
> Assignee: Vijay
> Priority: Minor
> Labels: lhf
> Fix For: 1.0
>
> Attachments: CASSANDRA-2894.patch
>
>
> It is non-intuitive that get_count materializes the entire slice-to-count on the coordinator node (to perform read repair and > CL.ONE consistency). Even experienced users have been known to cause memory problems by requesting large counts.
> The user cannot page the count himself, because you need a start and stop column to do that, and get_count only returns an integer.
> So the best fix is for us to do the paging under the hood, in CassandraServer. Add a limit to the slicepredicate they specify, and page through it.
> We could add a global setting for count_slice_size, and document that counts of more columns than that will have higher latency (because they make multiple calls through StorageProxy for the pages).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-2894) add paging to get_count
Posted by "Byron Clark (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Byron Clark updated CASSANDRA-2894:
-----------------------------------
Attachment: 2894-with-test.txt
> add paging to get_count
> -----------------------
>
> Key: CASSANDRA-2894
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2894
> Project: Cassandra
> Issue Type: Improvement
> Components: API
> Reporter: Jonathan Ellis
> Priority: Minor
> Labels: lhf
> Fix For: 1.0
>
> Attachments: 2894-formatted.txt, 2894-with-test.txt, CASSANDRA-2894.patch
>
>
> It is non-intuitive that get_count materializes the entire slice-to-count on the coordinator node (to perform read repair and > CL.ONE consistency). Even experienced users have been known to cause memory problems by requesting large counts.
> The user cannot page the count himself, because you need a start and stop column to do that, and get_count only returns an integer.
> So the best fix is for us to do the paging under the hood, in CassandraServer. Add a limit to the slicepredicate they specify, and page through it.
> We could add a global setting for count_slice_size, and document that counts of more columns than that will have higher latency (because they make multiple calls through StorageProxy for the pages).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-2894) add paging to get_count
Posted by "Byron Clark (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Byron Clark updated CASSANDRA-2894:
-----------------------------------
Attachment: (was: 2894-with-test.txt)
> add paging to get_count
> -----------------------
>
> Key: CASSANDRA-2894
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2894
> Project: Cassandra
> Issue Type: Improvement
> Components: API
> Reporter: Jonathan Ellis
> Priority: Minor
> Labels: lhf
> Fix For: 1.0
>
> Attachments: 2894-formatted.txt, 2894-with-test.txt, CASSANDRA-2894.patch
>
>
> It is non-intuitive that get_count materializes the entire slice-to-count on the coordinator node (to perform read repair and > CL.ONE consistency). Even experienced users have been known to cause memory problems by requesting large counts.
> The user cannot page the count himself, because you need a start and stop column to do that, and get_count only returns an integer.
> So the best fix is for us to do the paging under the hood, in CassandraServer. Add a limit to the slicepredicate they specify, and page through it.
> We could add a global setting for count_slice_size, and document that counts of more columns than that will have higher latency (because they make multiple calls through StorageProxy for the pages).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-2894) add paging to get_count
Posted by "Byron Clark (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Byron Clark updated CASSANDRA-2894:
-----------------------------------
Attachment: CASSANDRA-2894.patch
> add paging to get_count
> -----------------------
>
> Key: CASSANDRA-2894
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2894
> Project: Cassandra
> Issue Type: Improvement
> Components: API
> Reporter: Jonathan Ellis
> Assignee: Vijay
> Priority: Minor
> Labels: lhf
> Fix For: 1.0
>
> Attachments: CASSANDRA-2894.patch
>
>
> It is non-intuitive that get_count materializes the entire slice-to-count on the coordinator node (to perform read repair and > CL.ONE consistency). Even experienced users have been known to cause memory problems by requesting large counts.
> The user cannot page the count himself, because you need a start and stop column to do that, and get_count only returns an integer.
> So the best fix is for us to do the paging under the hood, in CassandraServer. Add a limit to the slicepredicate they specify, and page through it.
> We could add a global setting for count_slice_size, and document that counts of more columns than that will have higher latency (because they make multiple calls through StorageProxy for the pages).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2894) add paging to get_count
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13071480#comment-13071480 ]
Jonathan Ellis commented on CASSANDRA-2894:
-------------------------------------------
cql count is broken anyway; it should count rows, not columns. fixing that is out of scope here.
> add paging to get_count
> -----------------------
>
> Key: CASSANDRA-2894
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2894
> Project: Cassandra
> Issue Type: Improvement
> Components: API
> Reporter: Jonathan Ellis
> Priority: Minor
> Labels: lhf
> Fix For: 1.0
>
> Attachments: CASSANDRA-2894.patch
>
>
> It is non-intuitive that get_count materializes the entire slice-to-count on the coordinator node (to perform read repair and > CL.ONE consistency). Even experienced users have been known to cause memory problems by requesting large counts.
> The user cannot page the count himself, because you need a start and stop column to do that, and get_count only returns an integer.
> So the best fix is for us to do the paging under the hood, in CassandraServer. Add a limit to the slicepredicate they specify, and page through it.
> We could add a global setting for count_slice_size, and document that counts of more columns than that will have higher latency (because they make multiple calls through StorageProxy for the pages).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2894) add paging to get_count
Posted by "Vijay (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13071311#comment-13071311 ]
Vijay commented on CASSANDRA-2894:
----------------------------------
Hi Byron, How abt CQL?
> add paging to get_count
> -----------------------
>
> Key: CASSANDRA-2894
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2894
> Project: Cassandra
> Issue Type: Improvement
> Components: API
> Reporter: Jonathan Ellis
> Priority: Minor
> Labels: lhf
> Fix For: 1.0
>
> Attachments: CASSANDRA-2894.patch
>
>
> It is non-intuitive that get_count materializes the entire slice-to-count on the coordinator node (to perform read repair and > CL.ONE consistency). Even experienced users have been known to cause memory problems by requesting large counts.
> The user cannot page the count himself, because you need a start and stop column to do that, and get_count only returns an integer.
> So the best fix is for us to do the paging under the hood, in CassandraServer. Add a limit to the slicepredicate they specify, and page through it.
> We could add a global setting for count_slice_size, and document that counts of more columns than that will have higher latency (because they make multiple calls through StorageProxy for the pages).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2894) add paging to get_count
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13073324#comment-13073324 ]
Hudson commented on CASSANDRA-2894:
-----------------------------------
Integrated in Cassandra #989 (See [https://builds.apache.org/job/Cassandra/989/])
add paging to get_count
patch by Byron Clark; reviewed by jbellis for CASSANDRA-2894
jbellis : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1152545
Files :
* /cassandra/trunk/CHANGES.txt
* /cassandra/trunk/test/system/test_thrift_server.py
* /cassandra/trunk/src/java/org/apache/cassandra/thrift/CassandraServer.java
> add paging to get_count
> -----------------------
>
> Key: CASSANDRA-2894
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2894
> Project: Cassandra
> Issue Type: Improvement
> Components: API
> Reporter: Jonathan Ellis
> Assignee: Byron Clark
> Priority: Minor
> Labels: lhf
> Fix For: 1.0
>
> Attachments: 2894-formatted.txt, 2894-with-batch-test.txt, 2894-with-test.txt, CASSANDRA-2894.patch
>
>
> It is non-intuitive that get_count materializes the entire slice-to-count on the coordinator node (to perform read repair and > CL.ONE consistency). Even experienced users have been known to cause memory problems by requesting large counts.
> The user cannot page the count himself, because you need a start and stop column to do that, and get_count only returns an integer.
> So the best fix is for us to do the paging under the hood, in CassandraServer. Add a limit to the slicepredicate they specify, and page through it.
> We could add a global setting for count_slice_size, and document that counts of more columns than that will have higher latency (because they make multiple calls through StorageProxy for the pages).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (CASSANDRA-2894) add paging to get_count
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis resolved CASSANDRA-2894.
---------------------------------------
Resolution: Fixed
Reviewer: jbellis
Assignee: Byron Clark
committed, thanks!
created CASSANDRA-2977 for the multiget case, for completeness; however, I incline towards option 1 ("do nothing").
> add paging to get_count
> -----------------------
>
> Key: CASSANDRA-2894
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2894
> Project: Cassandra
> Issue Type: Improvement
> Components: API
> Reporter: Jonathan Ellis
> Assignee: Byron Clark
> Priority: Minor
> Labels: lhf
> Fix For: 1.0
>
> Attachments: 2894-formatted.txt, 2894-with-batch-test.txt, 2894-with-test.txt, CASSANDRA-2894.patch
>
>
> It is non-intuitive that get_count materializes the entire slice-to-count on the coordinator node (to perform read repair and > CL.ONE consistency). Even experienced users have been known to cause memory problems by requesting large counts.
> The user cannot page the count himself, because you need a start and stop column to do that, and get_count only returns an integer.
> So the best fix is for us to do the paging under the hood, in CassandraServer. Add a limit to the slicepredicate they specify, and page through it.
> We could add a global setting for count_slice_size, and document that counts of more columns than that will have higher latency (because they make multiple calls through StorageProxy for the pages).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (CASSANDRA-2894) add paging to get_count
Posted by "Vijay (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vijay reassigned CASSANDRA-2894:
--------------------------------
Assignee: Vijay
> add paging to get_count
> -----------------------
>
> Key: CASSANDRA-2894
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2894
> Project: Cassandra
> Issue Type: Improvement
> Components: API
> Reporter: Jonathan Ellis
> Assignee: Vijay
> Priority: Minor
> Labels: lhf
> Fix For: 1.0
>
>
> It is non-intuitive that get_count materializes the entire slice-to-count on the coordinator node (to perform read repair and > CL.ONE consistency). Even experienced users have been known to cause memory problems by requesting large counts.
> The user cannot page the count himself, because you need a start and stop column to do that, and get_count only returns an integer.
> So the best fix is for us to do the paging under the hood, in CassandraServer. Add a limit to the slicepredicate they specify, and page through it.
> We could add a global setting for count_slice_size, and document that counts of more columns than that will have higher latency (because they make multiple calls through StorageProxy for the pages).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-2894) add paging to get_count
Posted by "Byron Clark (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Byron Clark updated CASSANDRA-2894:
-----------------------------------
Attachment: 2894-with-batch-test.txt
[^2894-with-batch-test.txt] uses a batch mutate to setup the test and it now runs as quickly as test_count.
> add paging to get_count
> -----------------------
>
> Key: CASSANDRA-2894
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2894
> Project: Cassandra
> Issue Type: Improvement
> Components: API
> Reporter: Jonathan Ellis
> Priority: Minor
> Labels: lhf
> Fix For: 1.0
>
> Attachments: 2894-formatted.txt, 2894-with-batch-test.txt, 2894-with-test.txt, CASSANDRA-2894.patch
>
>
> It is non-intuitive that get_count materializes the entire slice-to-count on the coordinator node (to perform read repair and > CL.ONE consistency). Even experienced users have been known to cause memory problems by requesting large counts.
> The user cannot page the count himself, because you need a start and stop column to do that, and get_count only returns an integer.
> So the best fix is for us to do the paging under the hood, in CassandraServer. Add a limit to the slicepredicate they specify, and page through it.
> We could add a global setting for count_slice_size, and document that counts of more columns than that will have higher latency (because they make multiple calls through StorageProxy for the pages).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira