You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Peter Schuller (JIRA)" <ji...@apache.org> on 2010/10/24 00:15:24 UTC
[jira] Created: (CASSANDRA-1657) support in-memory column families
support in-memory column families
---------------------------------
Key: CASSANDRA-1657
URL: https://issues.apache.org/jira/browse/CASSANDRA-1657
Project: Cassandra
Issue Type: Improvement
Reporter: Peter Schuller
Priority: Minor
Some workloads are such that you absolutely depend on column families being in-memory for performance, yet you most definitely want all the things that Cassandra offers in terms of replication, consistency, durability etc.
In order to semi-deterministically ensure acceptable performance for such data, Cassandra could support in-memory column families. Such an in-memory column family would imply that mlock() be used on sstables for this column family. On start-up and on compaction completion, they could be mmap():ed with MAP_POPULATE (Linux specific) or else just mmap():ed + mlock():ed in such a way as to otherwise guarantee it is in-memory (such as userland traversal of the entire file).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-1657) support in-memory column
families
Posted by "Ryan King (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12989473#comment-12989473 ]
Ryan King commented on CASSANDRA-1657:
--------------------------------------
For narrow SSTables, shouldn't the row cache be enough for this?
> support in-memory column families
> ---------------------------------
>
> Key: CASSANDRA-1657
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1657
> Project: Cassandra
> Issue Type: Improvement
> Reporter: Peter Schuller
> Priority: Minor
>
> Some workloads are such that you absolutely depend on column families being in-memory for performance, yet you most definitely want all the things that Cassandra offers in terms of replication, consistency, durability etc.
> In order to semi-deterministically ensure acceptable performance for such data, Cassandra could support in-memory column families. Such an in-memory column family would imply that mlock() be used on sstables for this column family. On start-up and on compaction completion, they could be mmap():ed with MAP_POPULATE (Linux specific) or else just mmap():ed + mlock():ed in such a way as to otherwise guarantee it is in-memory (such as userland traversal of the entire file).
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (CASSANDRA-1657) support in-memory column
families
Posted by "Peter Schuller (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12989549#comment-12989549 ]
Peter Schuller commented on CASSANDRA-1657:
-------------------------------------------
Depending on use-case maybe, but I don't think always. Primarily for three reasons:
(1) The row cache is not at all as efficient in terms of memory use. Assuming the premise that you really want it to be guaranteed in memory is true (rather than a 10% active set), the on-disk format is considerably more compact (more so for very small rows with very small columns, than for medium-sized cf:s with less keys). A nice bonus is avoiding GC pressure, and edge cases like the odd large row and difficulty sizing the row cache.
(2) The row-cache has a sever trade-off in terms of start-up time due to it being seek-bound (although for CF:s that are supposed to be in-memory anyway it may be mitigated by pre-heating it by streaming through it; but that feels more like a hack). A kernel-driven sequential pre-population is vastly preferred ;)
(3) Hopefully with future improvements in compaction (e.g. limiting maximum sstable size so that compaction can happen to reasonably sized chunks, thus not doubling memory use) an in-memory CF would stay in-memory completely during compaction (only incurring deferred writes). In other words, it would contribute significantly less to (disk i/o) compaction overhead than if the active set was in row cache.
I do assume that the use-case is such that optimizing for avoiding serialization/deserialization is not the issue here, and that the importance of keeping the data off disk is more important than squeezing the last bit of CPU efficiency out of it.
Since Cassandra is going the route of native code and embracing things like posix_fadvise/mincore/mlock etc anyway, it seemed like a reasonable thing to support. However of course if other features end up catering sufficiently to the same goal, the suggestion may become obsolete.
Also, I still like the stuff in CASSANDRA-1625 (specifically https://issues.apache.org/jira/browse/CASSANDRA-1625?focusedCommentId=12971877&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-12971877) which overlaps with this.
> support in-memory column families
> ---------------------------------
>
> Key: CASSANDRA-1657
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1657
> Project: Cassandra
> Issue Type: Improvement
> Reporter: Peter Schuller
> Priority: Minor
>
> Some workloads are such that you absolutely depend on column families being in-memory for performance, yet you most definitely want all the things that Cassandra offers in terms of replication, consistency, durability etc.
> In order to semi-deterministically ensure acceptable performance for such data, Cassandra could support in-memory column families. Such an in-memory column family would imply that mlock() be used on sstables for this column family. On start-up and on compaction completion, they could be mmap():ed with MAP_POPULATE (Linux specific) or else just mmap():ed + mlock():ed in such a way as to otherwise guarantee it is in-memory (such as userland traversal of the entire file).
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira