You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Chris Goffinet (JIRA)" <ji...@apache.org> on 2010/10/12 06:05:33 UTC
[jira] Created: (CASSANDRA-1608) Redesigned Compaction
Redesigned Compaction
---------------------
Key: CASSANDRA-1608
URL: https://issues.apache.org/jira/browse/CASSANDRA-1608
Project: Cassandra
Issue Type: Improvement
Components: Core
Affects Versions: 0.7 beta 2
Reporter: Chris Goffinet
After seeing the I/O issues in CASSANDRA-1470, I've been doing some more thinking on this subject that I wanted to lay out.
I propose we redo the concept of how compaction works in Cassandra. At the moment, compaction is kicked off based on a write access pattern, not read access pattern. In most cases, you want the opposite. You want to be able to track how well each SSTable is performing in the system. If we were to keep statistics in-memory of each SSTable, prioritize them based on most accessed, and bloom filter hit/miss ratios, we could intelligently group sstables that are being read most often and schedule them for compaction. We could also schedule lower priority maintenance on SSTable's not often accessed.
I also propose we limit the size of each SSTable to a fix sized, that gives us the ability to better utilize our bloom filters in a predictable manner. At the moment after a certain size, the bloom filters become less reliable. This would also allow us to group data most accessed. Currently the size of an SSTable can grow to a point where large portions of the data might not actually be accessed as often.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-1608) Redesigned Compaction
Posted by "Stu Hood (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12920377#action_12920377 ]
Stu Hood commented on CASSANDRA-1608:
-------------------------------------
> I guess Cassandra would only need a fixed count of exactly 2, making it a non-issue.
Yea, I think we realized this at the same time... rather than deleting from the original filter, you could mark it as superseded in a separate filter for the sstable, which could be sized to accommodate the number of supersedes/deletes you would need to do before you'd want to run a compaction for the sstable: less than 2 bits totally actually.
> Redesigned Compaction
> ---------------------
>
> Key: CASSANDRA-1608
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1608
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Reporter: Chris Goffinet
> Fix For: 0.7.1
>
>
> After seeing the I/O issues in CASSANDRA-1470, I've been doing some more thinking on this subject that I wanted to lay out.
> I propose we redo the concept of how compaction works in Cassandra. At the moment, compaction is kicked off based on a write access pattern, not read access pattern. In most cases, you want the opposite. You want to be able to track how well each SSTable is performing in the system. If we were to keep statistics in-memory of each SSTable, prioritize them based on most accessed, and bloom filter hit/miss ratios, we could intelligently group sstables that are being read most often and schedule them for compaction. We could also schedule lower priority maintenance on SSTable's not often accessed.
> I also propose we limit the size of each SSTable to a fix sized, that gives us the ability to better utilize our bloom filters in a predictable manner. At the moment after a certain size, the bloom filters become less reliable. This would also allow us to group data most accessed. Currently the size of an SSTable can grow to a point where large portions of the data might not actually be accessed as often.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-1608) Redesigned Compaction
Posted by "Stu Hood (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12920380#action_12920380 ]
Stu Hood commented on CASSANDRA-1608:
-------------------------------------
> I guess Cassandra would only need a fixed count of exactly 2, making it a non-issue.
As you said: we would need a counting filter with 2 bits per bucket: if both bits are set, the bucket has collided.
If any of the buckets for a key have collided, you can't perform the delete, but that isn't the end of the world here.
> Redesigned Compaction
> ---------------------
>
> Key: CASSANDRA-1608
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1608
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Reporter: Chris Goffinet
> Fix For: 0.7.1
>
>
> After seeing the I/O issues in CASSANDRA-1470, I've been doing some more thinking on this subject that I wanted to lay out.
> I propose we redo the concept of how compaction works in Cassandra. At the moment, compaction is kicked off based on a write access pattern, not read access pattern. In most cases, you want the opposite. You want to be able to track how well each SSTable is performing in the system. If we were to keep statistics in-memory of each SSTable, prioritize them based on most accessed, and bloom filter hit/miss ratios, we could intelligently group sstables that are being read most often and schedule them for compaction. We could also schedule lower priority maintenance on SSTable's not often accessed.
> I also propose we limit the size of each SSTable to a fix sized, that gives us the ability to better utilize our bloom filters in a predictable manner. At the moment after a certain size, the bloom filters become less reliable. This would also allow us to group data most accessed. Currently the size of an SSTable can grow to a point where large portions of the data might not actually be accessed as often.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-1608) Redesigned Compaction
Posted by "Peter Schuller (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12921914#action_12921914 ]
Peter Schuller commented on CASSANDRA-1608:
-------------------------------------------
With respect to tracking information, I meant tracking the information necessary to make the determination as to which rows to supersede or otherwise act upon, rather than keeping track of what *has* been superseded (which as you point out is dealt with by bloom filters).
In the case of simple criteria (such as seeing a single read spread across more than N sstables) no such tracking is necessary. But I am concerned with strategies that imply having to keep significant amounts of data over time, such as anything based on row-level frequency/recency of access.
In particular, if one hopes to supersede so effectively that hot rows (regardless of sstable spread) end up in separate sstables with high locality of access, will not this need such row-level information tracking?
> Redesigned Compaction
> ---------------------
>
> Key: CASSANDRA-1608
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1608
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Reporter: Chris Goffinet
> Fix For: 0.7.1
>
>
> After seeing the I/O issues in CASSANDRA-1470, I've been doing some more thinking on this subject that I wanted to lay out.
> I propose we redo the concept of how compaction works in Cassandra. At the moment, compaction is kicked off based on a write access pattern, not read access pattern. In most cases, you want the opposite. You want to be able to track how well each SSTable is performing in the system. If we were to keep statistics in-memory of each SSTable, prioritize them based on most accessed, and bloom filter hit/miss ratios, we could intelligently group sstables that are being read most often and schedule them for compaction. We could also schedule lower priority maintenance on SSTable's not often accessed.
> I also propose we limit the size of each SSTable to a fix sized, that gives us the ability to better utilize our bloom filters in a predictable manner. At the moment after a certain size, the bloom filters become less reliable. This would also allow us to group data most accessed. Currently the size of an SSTable can grow to a point where large portions of the data might not actually be accessed as often.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-1608) Redesigned Compaction
Posted by "Kelvin Kakugawa (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12922659#action_12922659 ]
Kelvin Kakugawa commented on CASSANDRA-1608:
--------------------------------------------
An alternative strategy that dovetails w/ the above proposal (fixed-size SSTs and keeping track of SST co-access stats):
Lazily coordinate timestamps across the cluster by maintaining increasing timestamps across SSTs on each replica.
If we can enforce increasing timestamps across SSTs, then lookups will be cheaper. Instead of doing a slice (under the hood) for each lookup, we only need to read from the last SST where a key+column was written based on BFs (FPs not withstanding). Slices will not be improved by this scheme.
A way to implement this would be to:
1) on a CL.ONE write, always write to a lead replica, and
2) on a CL.QUORUM write, a replica will reject a write w/ a timestamp less than the highest timestamp in that CF's last SST (may need to be MT); upon which, the client will need to re-submit a write w/ the appropriate timestamp offset.
A consideration that needs to be taken into account is AES and other SST streaming operations across replicas. e.g. streamed SSTs will need to be lined up by min/max timestamp of the SSTs; if an SST overlaps, then we may need to, either:
1) re-partition/compact the overlapping SSTs, or
2) lazily compact the overlapping SSTs and absorb the cost to lookups, in the meantime.
The benefits of loosely coordinated timestamps are:
1) lookups will be measurably improved,
2) dovetails nicely w/ fixed-size SSTs, and
3) SST co-access stats can be more coarse-grained (based on SST), instead of fine-grained row-level stats.
The above proposal is would align our data model to be closer to BigTable and HBase. i.e. lookups won't be penalized, anymore.
> Redesigned Compaction
> ---------------------
>
> Key: CASSANDRA-1608
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1608
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Reporter: Chris Goffinet
> Fix For: 0.7.1
>
>
> After seeing the I/O issues in CASSANDRA-1470, I've been doing some more thinking on this subject that I wanted to lay out.
> I propose we redo the concept of how compaction works in Cassandra. At the moment, compaction is kicked off based on a write access pattern, not read access pattern. In most cases, you want the opposite. You want to be able to track how well each SSTable is performing in the system. If we were to keep statistics in-memory of each SSTable, prioritize them based on most accessed, and bloom filter hit/miss ratios, we could intelligently group sstables that are being read most often and schedule them for compaction. We could also schedule lower priority maintenance on SSTable's not often accessed.
> I also propose we limit the size of each SSTable to a fix sized, that gives us the ability to better utilize our bloom filters in a predictable manner. At the moment after a certain size, the bloom filters become less reliable. This would also allow us to group data most accessed. Currently the size of an SSTable can grow to a point where large portions of the data might not actually be accessed as often.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-1608) Redesigned Compaction
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1608?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-1608:
--------------------------------------
Affects Version/s: (was: 0.7 beta 2)
Fix Version/s: 0.7.1
> Redesigned Compaction
> ---------------------
>
> Key: CASSANDRA-1608
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1608
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Reporter: Chris Goffinet
> Fix For: 0.7.1
>
>
> After seeing the I/O issues in CASSANDRA-1470, I've been doing some more thinking on this subject that I wanted to lay out.
> I propose we redo the concept of how compaction works in Cassandra. At the moment, compaction is kicked off based on a write access pattern, not read access pattern. In most cases, you want the opposite. You want to be able to track how well each SSTable is performing in the system. If we were to keep statistics in-memory of each SSTable, prioritize them based on most accessed, and bloom filter hit/miss ratios, we could intelligently group sstables that are being read most often and schedule them for compaction. We could also schedule lower priority maintenance on SSTable's not often accessed.
> I also propose we limit the size of each SSTable to a fix sized, that gives us the ability to better utilize our bloom filters in a predictable manner. At the moment after a certain size, the bloom filters become less reliable. This would also allow us to group data most accessed. Currently the size of an SSTable can grow to a point where large portions of the data might not actually be accessed as often.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-1608) Redesigned Compaction
Posted by "Ryan King (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12923191#action_12923191 ]
Ryan King commented on CASSANDRA-1608:
--------------------------------------
I think if we just start with tracking which sstables are accessed together, then prioritizing those for being compacted together we can improve our locality with minimal overhead.
Fixed size (or variable within a range) SSTables could then also help. If we achieve the fixed size by splitting by token range when doing a compaction, we would have much more predictable read time for both skinny rows and wide rows. I'm not sure how well this would react to a rapidly changing workload, though.
> Redesigned Compaction
> ---------------------
>
> Key: CASSANDRA-1608
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1608
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Reporter: Chris Goffinet
> Fix For: 0.7.1
>
>
> After seeing the I/O issues in CASSANDRA-1470, I've been doing some more thinking on this subject that I wanted to lay out.
> I propose we redo the concept of how compaction works in Cassandra. At the moment, compaction is kicked off based on a write access pattern, not read access pattern. In most cases, you want the opposite. You want to be able to track how well each SSTable is performing in the system. If we were to keep statistics in-memory of each SSTable, prioritize them based on most accessed, and bloom filter hit/miss ratios, we could intelligently group sstables that are being read most often and schedule them for compaction. We could also schedule lower priority maintenance on SSTable's not often accessed.
> I also propose we limit the size of each SSTable to a fix sized, that gives us the ability to better utilize our bloom filters in a predictable manner. At the moment after a certain size, the bloom filters become less reliable. This would also allow us to group data most accessed. Currently the size of an SSTable can grow to a point where large portions of the data might not actually be accessed as often.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-1608) Redesigned Compaction
Posted by "Stu Hood (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12925179#action_12925179 ]
Stu Hood commented on CASSANDRA-1608:
-------------------------------------
> But I am concerned with strategies that imply having to keep significant amounts of data over time, such as anything based
> on row-level frequency/recency of access.
Although it probably sounds like I have bloom filter fever, we could use a bloom filter for this as well: filters based on multisets, (spectral bloom filters, count min sketches), allow storing approximate counts for values. The access counts for rows for the lifetime of a memtable could be stored in a filter attached to the memtable: at read time, the approximate access count of the row key could be checked against a threshold of accesses, and be stored in the memtable as superseding.
> Redesigned Compaction
> ---------------------
>
> Key: CASSANDRA-1608
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1608
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Reporter: Chris Goffinet
> Fix For: 0.7.1
>
>
> After seeing the I/O issues in CASSANDRA-1470, I've been doing some more thinking on this subject that I wanted to lay out.
> I propose we redo the concept of how compaction works in Cassandra. At the moment, compaction is kicked off based on a write access pattern, not read access pattern. In most cases, you want the opposite. You want to be able to track how well each SSTable is performing in the system. If we were to keep statistics in-memory of each SSTable, prioritize them based on most accessed, and bloom filter hit/miss ratios, we could intelligently group sstables that are being read most often and schedule them for compaction. We could also schedule lower priority maintenance on SSTable's not often accessed.
> I also propose we limit the size of each SSTable to a fix sized, that gives us the ability to better utilize our bloom filters in a predictable manner. At the moment after a certain size, the bloom filters become less reliable. This would also allow us to group data most accessed. Currently the size of an SSTable can grow to a point where large portions of the data might not actually be accessed as often.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-1608) Redesigned Compaction
Posted by "Stu Hood (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1608?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Stu Hood updated CASSANDRA-1608:
--------------------------------
Comment: was deleted
(was: > I guess Cassandra would only need a fixed count of exactly 2, making it a non-issue.
Yea, I think we realized this at the same time... rather than deleting from the original filter, you could mark it as superseded in a separate filter for the sstable, which could be sized to accommodate the number of supersedes/deletes you would need to do before you'd want to run a compaction for the sstable: less than 2 bits totally actually.
EDIT: Nevermind... false positives in the second set would cause the wrong data to be dropped.)
> Redesigned Compaction
> ---------------------
>
> Key: CASSANDRA-1608
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1608
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Reporter: Chris Goffinet
> Fix For: 0.7.1
>
>
> After seeing the I/O issues in CASSANDRA-1470, I've been doing some more thinking on this subject that I wanted to lay out.
> I propose we redo the concept of how compaction works in Cassandra. At the moment, compaction is kicked off based on a write access pattern, not read access pattern. In most cases, you want the opposite. You want to be able to track how well each SSTable is performing in the system. If we were to keep statistics in-memory of each SSTable, prioritize them based on most accessed, and bloom filter hit/miss ratios, we could intelligently group sstables that are being read most often and schedule them for compaction. We could also schedule lower priority maintenance on SSTable's not often accessed.
> I also propose we limit the size of each SSTable to a fix sized, that gives us the ability to better utilize our bloom filters in a predictable manner. At the moment after a certain size, the bloom filters become less reliable. This would also allow us to group data most accessed. Currently the size of an SSTable can grow to a point where large portions of the data might not actually be accessed as often.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Issue Comment Edited: (CASSANDRA-1608) Redesigned Compaction
Posted by "Stu Hood (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12920377#action_12920377 ]
Stu Hood edited comment on CASSANDRA-1608 at 10/12/10 6:16 PM:
---------------------------------------------------------------
> I guess Cassandra would only need a fixed count of exactly 2, making it a non-issue.
Yea, I think we realized this at the same time... rather than deleting from the original filter, you could mark it as superseded in a separate filter for the sstable, which could be sized to accommodate the number of supersedes/deletes you would need to do before you'd want to run a compaction for the sstable: less than 2 bits totally actually.
EDIT: Nevermind... false positives in the second set would cause the wrong data to be dropped.
was (Author: stuhood):
> I guess Cassandra would only need a fixed count of exactly 2, making it a non-issue.
Yea, I think we realized this at the same time... rather than deleting from the original filter, you could mark it as superseded in a separate filter for the sstable, which could be sized to accommodate the number of supersedes/deletes you would need to do before you'd want to run a compaction for the sstable: less than 2 bits totally actually.
> Redesigned Compaction
> ---------------------
>
> Key: CASSANDRA-1608
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1608
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Reporter: Chris Goffinet
> Fix For: 0.7.1
>
>
> After seeing the I/O issues in CASSANDRA-1470, I've been doing some more thinking on this subject that I wanted to lay out.
> I propose we redo the concept of how compaction works in Cassandra. At the moment, compaction is kicked off based on a write access pattern, not read access pattern. In most cases, you want the opposite. You want to be able to track how well each SSTable is performing in the system. If we were to keep statistics in-memory of each SSTable, prioritize them based on most accessed, and bloom filter hit/miss ratios, we could intelligently group sstables that are being read most often and schedule them for compaction. We could also schedule lower priority maintenance on SSTable's not often accessed.
> I also propose we limit the size of each SSTable to a fix sized, that gives us the ability to better utilize our bloom filters in a predictable manner. At the moment after a certain size, the bloom filters become less reliable. This would also allow us to group data most accessed. Currently the size of an SSTable can grow to a point where large portions of the data might not actually be accessed as often.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-1608) Redesigned Compaction
Posted by "Peter Schuller (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12920374#action_12920374 ]
Peter Schuller commented on CASSANDRA-1608:
-------------------------------------------
Although now that I think about it some more, I guess Cassandra would only need a fixed count of exactly 2, making it a non-issue.
> Redesigned Compaction
> ---------------------
>
> Key: CASSANDRA-1608
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1608
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Reporter: Chris Goffinet
> Fix For: 0.7.1
>
>
> After seeing the I/O issues in CASSANDRA-1470, I've been doing some more thinking on this subject that I wanted to lay out.
> I propose we redo the concept of how compaction works in Cassandra. At the moment, compaction is kicked off based on a write access pattern, not read access pattern. In most cases, you want the opposite. You want to be able to track how well each SSTable is performing in the system. If we were to keep statistics in-memory of each SSTable, prioritize them based on most accessed, and bloom filter hit/miss ratios, we could intelligently group sstables that are being read most often and schedule them for compaction. We could also schedule lower priority maintenance on SSTable's not often accessed.
> I also propose we limit the size of each SSTable to a fix sized, that gives us the ability to better utilize our bloom filters in a predictable manner. At the moment after a certain size, the bloom filters become less reliable. This would also allow us to group data most accessed. Currently the size of an SSTable can grow to a point where large portions of the data might not actually be accessed as often.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-1608) Redesigned Compaction
Posted by "Stu Hood (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12921913#action_12921913 ]
Stu Hood commented on CASSANDRA-1608:
-------------------------------------
> Regarding superseding rows in sstables; what would be the criteria for picking which rows to supersede for?
A configurable threshold on access count is one metric: you could attach an access counter to rows in the memtable or rowcache. If a row reaches a threshold of accesses, then you supersede it by storing the data for the read in the memtable, and marking it superseded in any sstables.
(This works for skinny rows, but just like with the row cache, we need a strategy for wider rows.)
Another metric would be the minimum number of sstables to supersede: if a row is stored in 1 sstable, (obviously) don't supersede, but if it is stored in 4, supersede, etc.
> would that require tracking some information over time at row level granularity? (I'm concerned about the overhead of such tracking.)
I don't think so? If something was recently superseded, it will be stored in 1 sstable, and there is no need to supersede it reaches your threshold.
> If so, an observation is that false positives are allowed for the stats
I think this is what we get from the bloom filter edits: sometimes a supersede will fail for some sstables, but it's not really a big deal, since the whole operation probably still pushes you below your threshold for sstables.
> Redesigned Compaction
> ---------------------
>
> Key: CASSANDRA-1608
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1608
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Reporter: Chris Goffinet
> Fix For: 0.7.1
>
>
> After seeing the I/O issues in CASSANDRA-1470, I've been doing some more thinking on this subject that I wanted to lay out.
> I propose we redo the concept of how compaction works in Cassandra. At the moment, compaction is kicked off based on a write access pattern, not read access pattern. In most cases, you want the opposite. You want to be able to track how well each SSTable is performing in the system. If we were to keep statistics in-memory of each SSTable, prioritize them based on most accessed, and bloom filter hit/miss ratios, we could intelligently group sstables that are being read most often and schedule them for compaction. We could also schedule lower priority maintenance on SSTable's not often accessed.
> I also propose we limit the size of each SSTable to a fix sized, that gives us the ability to better utilize our bloom filters in a predictable manner. At the moment after a certain size, the bloom filters become less reliable. This would also allow us to group data most accessed. Currently the size of an SSTable can grow to a point where large portions of the data might not actually be accessed as often.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-1608) Redesigned Compaction
Posted by "Peter Schuller (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12920372#action_12920372 ]
Peter Schuller commented on CASSANDRA-1608:
-------------------------------------------
One thing to consider when deferring or omitting compaction of sstables is that it, similarly to a cache, will tend to optimize the performance in the common case but is susceptible to a sudden degredation of performance in response to a suddenly changing workload. That might be something to keep within reasonable limits, at least in out-of-the-box configurations.
On Stu's (3); what bloom filter would you have in mind that supports deletions? Is there a good paper to read? Is there anything better than what is described at http://en.wikipedia.org/wiki/Bloom_filter#Counting_filters (indicating some algorithm that consumes roughly half as much as a counting bf; but still scaling linearly with max count)?
> Redesigned Compaction
> ---------------------
>
> Key: CASSANDRA-1608
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1608
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Reporter: Chris Goffinet
> Fix For: 0.7.1
>
>
> After seeing the I/O issues in CASSANDRA-1470, I've been doing some more thinking on this subject that I wanted to lay out.
> I propose we redo the concept of how compaction works in Cassandra. At the moment, compaction is kicked off based on a write access pattern, not read access pattern. In most cases, you want the opposite. You want to be able to track how well each SSTable is performing in the system. If we were to keep statistics in-memory of each SSTable, prioritize them based on most accessed, and bloom filter hit/miss ratios, we could intelligently group sstables that are being read most often and schedule them for compaction. We could also schedule lower priority maintenance on SSTable's not often accessed.
> I also propose we limit the size of each SSTable to a fix sized, that gives us the ability to better utilize our bloom filters in a predictable manner. At the moment after a certain size, the bloom filters become less reliable. This would also allow us to group data most accessed. Currently the size of an SSTable can grow to a point where large portions of the data might not actually be accessed as often.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-1608) Redesigned Compaction
Posted by "Peter Schuller (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12921900#action_12921900 ]
Peter Schuller commented on CASSANDRA-1608:
-------------------------------------------
Regarding superseding rows in sstables; what would be the criteria for picking which rows to supersede for? A simple threshold would be easy and certainly addresses extreme cases of rows being spread. But if one also expects to take into account of often the rows are read, that would imply recenticity or frequency tracking?
A simple threshold on sstable count would certainly help avoiding the extreme cases of reads across many sstables.
But in terms of trying to keep frequently accessed data together with high locality (as briefly alluded to in CASSANDRA-1625), would that require tracking some information over time at row level granularity? (I'm concerned about the overhead of such tracking.)
If so, an observation is that false positives are allowed for the stats. I.e., recenticity/frequency could be associated with something like 64-bit-hash-of-key rather than keys, meaning that some optimizations become possible (e.g. smacking hash-of-key:counter pairs into large byte arrays).
> Redesigned Compaction
> ---------------------
>
> Key: CASSANDRA-1608
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1608
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Reporter: Chris Goffinet
> Fix For: 0.7.1
>
>
> After seeing the I/O issues in CASSANDRA-1470, I've been doing some more thinking on this subject that I wanted to lay out.
> I propose we redo the concept of how compaction works in Cassandra. At the moment, compaction is kicked off based on a write access pattern, not read access pattern. In most cases, you want the opposite. You want to be able to track how well each SSTable is performing in the system. If we were to keep statistics in-memory of each SSTable, prioritize them based on most accessed, and bloom filter hit/miss ratios, we could intelligently group sstables that are being read most often and schedule them for compaction. We could also schedule lower priority maintenance on SSTable's not often accessed.
> I also propose we limit the size of each SSTable to a fix sized, that gives us the ability to better utilize our bloom filters in a predictable manner. At the moment after a certain size, the bloom filters become less reliable. This would also allow us to group data most accessed. Currently the size of an SSTable can grow to a point where large portions of the data might not actually be accessed as often.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Issue Comment Edited: (CASSANDRA-1608) Redesigned Compaction
Posted by "Stu Hood (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12920380#action_12920380 ]
Stu Hood edited comment on CASSANDRA-1608 at 10/12/10 6:33 PM:
---------------------------------------------------------------
> I guess Cassandra would only need a fixed count of exactly 2, making it a non-issue.
As you said: we would need a counting filter with 2 bits per bucket: if both bits are set, the bucket has collided.
If any of the buckets for a key have collided, you can't perform the delete, but that isn't the end of the world here.
EDIT: Actually, if you couldn't perform the delete, you might end up superseding a given sstable multiple times, so this is something we'd want to avoid.
was (Author: stuhood):
> I guess Cassandra would only need a fixed count of exactly 2, making it a non-issue.
As you said: we would need a counting filter with 2 bits per bucket: if both bits are set, the bucket has collided.
If any of the buckets for a key have collided, you can't perform the delete, but that isn't the end of the world here.
> Redesigned Compaction
> ---------------------
>
> Key: CASSANDRA-1608
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1608
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Reporter: Chris Goffinet
> Fix For: 0.7.1
>
>
> After seeing the I/O issues in CASSANDRA-1470, I've been doing some more thinking on this subject that I wanted to lay out.
> I propose we redo the concept of how compaction works in Cassandra. At the moment, compaction is kicked off based on a write access pattern, not read access pattern. In most cases, you want the opposite. You want to be able to track how well each SSTable is performing in the system. If we were to keep statistics in-memory of each SSTable, prioritize them based on most accessed, and bloom filter hit/miss ratios, we could intelligently group sstables that are being read most often and schedule them for compaction. We could also schedule lower priority maintenance on SSTable's not often accessed.
> I also propose we limit the size of each SSTable to a fix sized, that gives us the ability to better utilize our bloom filters in a predictable manner. At the moment after a certain size, the bloom filters become less reliable. This would also allow us to group data most accessed. Currently the size of an SSTable can grow to a point where large portions of the data might not actually be accessed as often.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-1608) Redesigned Compaction
Posted by "Stu Hood (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12920353#action_12920353 ]
Stu Hood commented on CASSANDRA-1608:
-------------------------------------
One way to provide locality of reference for sstables would be to persist summaries of individual rows which 'supersede' the content in sstables written before them. For example, if you have five sstables containing key 'A', you would create a new sstable #6 containing all content for 'A', and marked as superseding for 'A'. Then, since you have a full copy of data for 'A', you no longer need to read from the other sstables.
But how would a reader know which sstables to check for a particular key? We have sstable generation numbers, but they are currently only used as unique ids. Three approaches:
# The 'superseding' mark for a particular key could indicate which generations it superseded, so we could prevent reads to superseded sstables
# If we refactored the system to read memtables/sstables in generation order, we could stop looking for content for A when we reached an sstable that superseded older sstables
# When superseding a value, we could delete it from the bloom filters for the superseded sstables. While hella cool, this solution would be a big change:
** If you reached a threshold of emptiness in the bloom filter, you could perform single-sstable-compactions filtered by values that still match
** Requires a bloom filter which supports deletes (ours don't yet)
> Redesigned Compaction
> ---------------------
>
> Key: CASSANDRA-1608
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1608
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Reporter: Chris Goffinet
> Fix For: 0.7.1
>
>
> After seeing the I/O issues in CASSANDRA-1470, I've been doing some more thinking on this subject that I wanted to lay out.
> I propose we redo the concept of how compaction works in Cassandra. At the moment, compaction is kicked off based on a write access pattern, not read access pattern. In most cases, you want the opposite. You want to be able to track how well each SSTable is performing in the system. If we were to keep statistics in-memory of each SSTable, prioritize them based on most accessed, and bloom filter hit/miss ratios, we could intelligently group sstables that are being read most often and schedule them for compaction. We could also schedule lower priority maintenance on SSTable's not often accessed.
> I also propose we limit the size of each SSTable to a fix sized, that gives us the ability to better utilize our bloom filters in a predictable manner. At the moment after a certain size, the bloom filters become less reliable. This would also allow us to group data most accessed. Currently the size of an SSTable can grow to a point where large portions of the data might not actually be accessed as often.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.