You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Stu Hood (JIRA)" <ji...@apache.org> on 2010/03/15 09:09:27 UTC
[jira] Created: (CASSANDRA-897) SSTable Partitioning
SSTable Partitioning
--------------------
Key: CASSANDRA-897
URL: https://issues.apache.org/jira/browse/CASSANDRA-897
Project: Cassandra
Issue Type: Improvement
Reporter: Stu Hood
Priority: Minor
Two problems we've seen bite people:
1. Major compactions require up to 100% additional disk space,
2. Data directories are not fully utilized after a major compaction.
One way to address both of these problems is to partition SSTables such that a-g might be contained in one sstable, and g-z in another. By default, the number of partitions created per compaction could be equal to the number of data directories.
Without changing the file format, it is possible to implement partitioning during compaction by writing multiple sstables: write from a to g into one sstable, close that sstable, select the least loaded drive, and open a new sstable for g-z. The only complication is that in order to not require 100% additional space, you have to eagerly mark sstables as compacted after they are successfully consumed and the most recent output sstable is closed.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (CASSANDRA-897) SSTable Partitioning
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-897?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis resolved CASSANDRA-897.
--------------------------------------
Resolution: Won't Fix
-1 on this in the Apache Veto sense.
Dealing with load balancing by messing with range changes inside a node as well as between nodes is not something we want to inflict on ourselves, and that's what we'd be signing up for.
Remember you do not know ahead of time how many keys a compacted sstable will contain. You know how many you start with in each source sstable, but you do not know how many overlap.
> SSTable Partitioning
> --------------------
>
> Key: CASSANDRA-897
> URL: https://issues.apache.org/jira/browse/CASSANDRA-897
> Project: Cassandra
> Issue Type: Improvement
> Reporter: Stu Hood
> Priority: Minor
>
> Two problems we've seen bite people:
> 1. Major compactions require up to 100% additional disk space,
> 2. Data directories are not fully utilized after a major compaction.
> One way to address both of these problems is to partition SSTables such that a-g might be contained in one sstable, and g-z in another. By default, the number of partitions created per compaction could be equal to the number of data directories.
> Without changing the file format, it is possible to implement partitioning during compaction by writing multiple sstables: write from a to g into one sstable, close that sstable, select the least loaded drive, and open a new sstable for g-z. The only complication is that in order to not require 100% additional space, you have to eagerly mark sstables as compacted after they are successfully consumed and the most recent output sstable is closed.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-897) SSTable Partitioning
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12845353#action_12845353 ]
Jonathan Ellis commented on CASSANDRA-897:
------------------------------------------
Which (to be more explicit) means you don't know when you are done writing (say) the first 1/3 of compacted data, so you can switch to the next destination.
> SSTable Partitioning
> --------------------
>
> Key: CASSANDRA-897
> URL: https://issues.apache.org/jira/browse/CASSANDRA-897
> Project: Cassandra
> Issue Type: Improvement
> Reporter: Stu Hood
> Priority: Minor
>
> Two problems we've seen bite people:
> 1. Major compactions require up to 100% additional disk space,
> 2. Data directories are not fully utilized after a major compaction.
> One way to address both of these problems is to partition SSTables such that a-g might be contained in one sstable, and g-z in another. By default, the number of partitions created per compaction could be equal to the number of data directories.
> Without changing the file format, it is possible to implement partitioning during compaction by writing multiple sstables: write from a to g into one sstable, close that sstable, select the least loaded drive, and open a new sstable for g-z. The only complication is that in order to not require 100% additional space, you have to eagerly mark sstables as compacted after they are successfully consumed and the most recent output sstable is closed.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.