You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Stu Hood (JIRA)" <ji...@apache.org> on 2010/04/06 05:22:27 UTC

[jira] Commented: (CASSANDRA-847) Make the reading half of compactions memory-efficient

    [ https://issues.apache.org/jira/browse/CASSANDRA-847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12853711#action_12853711 ] 

Stu Hood commented on CASSANDRA-847:
------------------------------------

Alright, after the hiatus to implement byte[] keys, I'm back on this horse.

> 2. Replace ColumnFamily and SuperColumn with ColumnGroup, implementing IColumn,
> and deleting IColumnContainer.
I don't think that nested structures, each with their own iterators is a good idea... especially when they may be hiding the fact that they are fetching columns from disk. And if they are not fetching transparently from disk, how do we make this any more memory efficient than the current approach?

The beauty in the Slice approach is that a List<Slice> can represent any arbitrarily nested structure you can think of, and yet the Slices are still autonomous. In the very long term I could imagine a Memtable being implemented as a SortedMap<ColumnKey,Slice>, where mutations are resolved into existing Slices, and then each are atomically swapped in order.

> 3. Implement new disk format, read + write, but no compaction yet.
I'm not sure how this is supposed to work: is the idea that we would break backwards compatibility in trunk, and then restore it later on in your steps 4,5,6?

> Make the reading half of compactions memory-efficient
> -----------------------------------------------------
>
>                 Key: CASSANDRA-847
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-847
>             Project: Cassandra
>          Issue Type: Improvement
>            Reporter: Stu Hood
>            Priority: Critical
>             Fix For: 0.7
>
>         Attachments: 0001-Add-structures-that-were-important-to-the-SSTableSca.patch, 0002-Implement-most-of-the-new-SSTableScanner-interface.patch, 0003-Rename-RowIndexedReader-specific-test.patch, 0004-Improve-Scanner-tests-and-separate-SuperCF-handling-.patch, 0005-Add-Scanner-interface-and-a-Filtered-implementation-.patch, 0006-Add-support-for-compaction-of-super-CFs-and-some-tes.patch, 0007-Remove-ColumnKey-bloom-filter-maintenance.patch, 0008-Make-Scanner-extend-Iterator-again.patch, 0009-Make-CompactionIterator-a-ReducingIterator-subclass-.patch, 0010-Alternative-to-ReducingIterator-that-can-return-mult.patch, compaction-bench-847.txt, compaction-bench-trunk.txt, compaction-bench.patch
>
>
> This issue is the next on the road to finally fixing CASSANDRA-16. To make compactions memory efficient, we have to be able to perform the compaction process on the smallest possible chunks that might intersect and contend one-another, meaning that we need a better abstraction for reading from SSTables.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.