You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Brandon Williams (JIRA)" <ji...@apache.org> on 2010/02/22 20:22:28 UTC
[jira] Created: (CASSANDRA-821) get_range_slice performance
get_range_slice performance
---------------------------
Key: CASSANDRA-821
URL: https://issues.apache.org/jira/browse/CASSANDRA-821
Project: Cassandra
Issue Type: Improvement
Affects Versions: 0.7
Reporter: Brandon Williams
Priority: Minor
get_range_slice performance isn't that great. CASSANDRA-799 helped in the case when the memtable isn't flushed, but overall the operations per second and latency is still pretty bad. On a quad core node with a trivial amount of data I see around 130-150 ops per second with stress.py defaults, and latency is 300-500ms.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-821) get_range_slice performance
Posted by "Brandon Williams (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12844077#action_12844077 ]
Brandon Williams commented on CASSANDRA-821:
--------------------------------------------
In my testing, this patch increases throughput by about 60% and cuts the latency in half, so there is serious potential here.
> get_range_slice performance
> ---------------------------
>
> Key: CASSANDRA-821
> URL: https://issues.apache.org/jira/browse/CASSANDRA-821
> Project: Cassandra
> Issue Type: Improvement
> Affects Versions: 0.6
> Reporter: Brandon Williams
> Assignee: Johan Oskarsson
> Priority: Minor
> Fix For: 0.7
>
> Attachments: CASSANDRA-821.patch
>
>
> get_range_slice performance isn't that great. CASSANDRA-799 helped in the case when the memtable isn't flushed, but overall the operations per second and latency is still pretty bad. On a quad core node with a trivial amount of data I see around 130-150 ops per second with stress.py set to slice 100 keys at a time, and latency is 300-500ms.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-821) get_range_slice performance
Posted by "Johan Oskarsson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-821?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Johan Oskarsson updated CASSANDRA-821:
--------------------------------------
Attachment: CASSANDRA-821.patch
Updated version of the patch incorporating some of ideas from Jonathan's comment.
* The scanner now returns the correct column iterator based on the filter provided (or lack thereof).
* The reducing iterator in ColumnFamilyStore returns the row straight away without the need for a final step
There are two cosmetic details though that I'd like a comment on from someone
* I had to expose a method in the ColumnIterators to seek to the start of the row since the collating iterator will go one row too far to compare row keys. Any advice on how we can make this nicer is welcome.
* I don't think I did the reducing iterator implementation the way you had in mind. I don't quite get how the ColumnIterator in the reduce method can be made into a ColumnFamily without the other iterators for the same row present? So that I can use ColumnFamily.addAll
> get_range_slice performance
> ---------------------------
>
> Key: CASSANDRA-821
> URL: https://issues.apache.org/jira/browse/CASSANDRA-821
> Project: Cassandra
> Issue Type: Improvement
> Affects Versions: 0.6
> Reporter: Brandon Williams
> Assignee: Johan Oskarsson
> Priority: Minor
> Fix For: 0.7
>
> Attachments: CASSANDRA-821.patch, CASSANDRA-821.patch
>
>
> get_range_slice performance isn't that great. CASSANDRA-799 helped in the case when the memtable isn't flushed, but overall the operations per second and latency is still pretty bad. On a quad core node with a trivial amount of data I see around 130-150 ops per second with stress.py set to slice 100 keys at a time, and latency is 300-500ms.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-821) get_range_slice performance
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-821?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-821:
-------------------------------------
Affects Version/s: (was: 0.7)
0.6
Fix Version/s: 0.7
> get_range_slice performance
> ---------------------------
>
> Key: CASSANDRA-821
> URL: https://issues.apache.org/jira/browse/CASSANDRA-821
> Project: Cassandra
> Issue Type: Improvement
> Affects Versions: 0.6
> Reporter: Brandon Williams
> Priority: Minor
> Fix For: 0.7
>
>
> get_range_slice performance isn't that great. CASSANDRA-799 helped in the case when the memtable isn't flushed, but overall the operations per second and latency is still pretty bad. On a quad core node with a trivial amount of data I see around 130-150 ops per second with stress.py set to slice 100 keys at a time, and latency is 300-500ms.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-821) get_range_slice performance
Posted by "Johan Oskarsson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-821?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Johan Oskarsson updated CASSANDRA-821:
--------------------------------------
Attachment: CASSANDRA-821.patch
This patch is very much a work in progress, comments are welcome.
The main objective is to remove the two loops in the operation into one. The current code reads the keys first and then fetches the data key by key, opening and closing files for each in the process. This patch moves that into one loop and shares the open files.
I have compared the results from stress.py against the patched version and a clean trunk with impressive results. However that was just on my desktop and I had a hard time getting results at all without timeouts, so won't post any results until I (or someone else) can verify on a better setup.
Implementation details:
* Renames IteratingRow into SSTableIteratingRow and changes the former into an interface, so that we can handle the memtables in a similar fashion.
* Adds a new constructor to SSTableSliceIterator and SSTableNamesIterator to accept an already open file.
* Bulk of the changes are in the getKeyRange method to adapt to the new way of reading data.
* Have not looked at hooking up the caching layer yet
* Passes all tests, but I suspect there will be bugs and things I have missed.
> get_range_slice performance
> ---------------------------
>
> Key: CASSANDRA-821
> URL: https://issues.apache.org/jira/browse/CASSANDRA-821
> Project: Cassandra
> Issue Type: Improvement
> Affects Versions: 0.6
> Reporter: Brandon Williams
> Priority: Minor
> Fix For: 0.7
>
> Attachments: CASSANDRA-821.patch
>
>
> get_range_slice performance isn't that great. CASSANDRA-799 helped in the case when the memtable isn't flushed, but overall the operations per second and latency is still pretty bad. On a quad core node with a trivial amount of data I see around 130-150 ops per second with stress.py set to slice 100 keys at a time, and latency is 300-500ms.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-821) get_range_slice performance
Posted by "Brandon Williams (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-821?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Brandon Williams updated CASSANDRA-821:
---------------------------------------
Description: get_range_slice performance isn't that great. CASSANDRA-799 helped in the case when the memtable isn't flushed, but overall the operations per second and latency is still pretty bad. On a quad core node with a trivial amount of data I see around 130-150 ops per second with stress.py set to slice 100 keys at a time, and latency is 300-500ms. (was: get_range_slice performance isn't that great. CASSANDRA-799 helped in the case when the memtable isn't flushed, but overall the operations per second and latency is still pretty bad. On a quad core node with a trivial amount of data I see around 130-150 ops per second with stress.py defaults, and latency is 300-500ms.)
> get_range_slice performance
> ---------------------------
>
> Key: CASSANDRA-821
> URL: https://issues.apache.org/jira/browse/CASSANDRA-821
> Project: Cassandra
> Issue Type: Improvement
> Affects Versions: 0.7
> Reporter: Brandon Williams
> Priority: Minor
>
> get_range_slice performance isn't that great. CASSANDRA-799 helped in the case when the memtable isn't flushed, but overall the operations per second and latency is still pretty bad. On a quad core node with a trivial amount of data I see around 130-150 ops per second with stress.py set to slice 100 keys at a time, and latency is 300-500ms.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-821) get_range_slice performance
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12845480#action_12845480 ]
Jonathan Ellis commented on CASSANDRA-821:
------------------------------------------
This is a huge step in the right direction.
The main cleanup that needs to happen is, the sstable iterator should take the same QueryFilter `filter` as the memtable ones, so that the reducing iterator can indeed reduce to a single Row (via cf.addall) and the "pull rows out of the iterator" step stays simple, w/o a second round of filtering. readColumnsForRow should go away entirely this way.
(Table.flusherLock only needs to be acquired for accesses to CFS.memtable variable, btw. I've updated the comment in its definition to make this more clear.)
Minor point: by convention, IteratingRow should become IIteratingRow, and needs apache license header.
> get_range_slice performance
> ---------------------------
>
> Key: CASSANDRA-821
> URL: https://issues.apache.org/jira/browse/CASSANDRA-821
> Project: Cassandra
> Issue Type: Improvement
> Affects Versions: 0.6
> Reporter: Brandon Williams
> Assignee: Johan Oskarsson
> Priority: Minor
> Fix For: 0.7
>
> Attachments: CASSANDRA-821.patch
>
>
> get_range_slice performance isn't that great. CASSANDRA-799 helped in the case when the memtable isn't flushed, but overall the operations per second and latency is still pretty bad. On a quad core node with a trivial amount of data I see around 130-150 ops per second with stress.py set to slice 100 keys at a time, and latency is 300-500ms.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.