You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Alan Liang (JIRA)" <ji...@apache.org> on 2011/06/02 05:11:47 UTC
[jira] [Created] (CASSANDRA-2735) Timestamp Based Compaction
Strategy
Timestamp Based Compaction Strategy
-----------------------------------
Key: CASSANDRA-2735
URL: https://issues.apache.org/jira/browse/CASSANDRA-2735
Project: Cassandra
Issue Type: New Feature
Components: Core
Reporter: Alan Liang
Assignee: Alan Liang
Priority: Minor
Compaction strategy implementation based on max timestamp ordering of the sstables while satisfying max sstable size, min and max compaction thresholds. It also handles expiration of sstables based on a timestamp.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2735) Timestamp Based Compaction
Strategy
Posted by "Viktor Jevdokimov (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13117068#comment-13117068 ]
Viktor Jevdokimov commented on CASSANDRA-2735:
----------------------------------------------
@Alan: compaction_throughput_mb_per_sec did not helped? I'm very interested in any one.
> Timestamp Based Compaction Strategy
> -----------------------------------
>
> Key: CASSANDRA-2735
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2735
> Project: Cassandra
> Issue Type: New Feature
> Components: Core
> Reporter: Alan Liang
> Assignee: Alan Liang
> Priority: Minor
> Labels: compaction
> Attachments: 0001-timestamp-bucketed-compaction-strategy-V2.patch, 0001-timestamp-bucketed-compaction-strategy.patch
>
>
> Compaction strategy implementation based on max timestamp ordering of the sstables while satisfying max sstable size, min and max compaction thresholds. It also handles expiration of sstables based on a timestamp.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2735) Timestamp Based Compaction
Strategy
Posted by "Sylvain Lebresne (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13051836#comment-13051836 ]
Sylvain Lebresne commented on CASSANDRA-2735:
---------------------------------------------
The goal here is not to have TTL for counters (or anything else). The goal is to have a compaction strategy that as part of what it does can throw up entire sstable when the content is considered "old enough" (and that's actually only part of the strategy, not necessarily its primary goal). As it turns out, this will roughly (and the rough part is important) amount to expire data, including counters. But this will be a very heavy hammer, in particular it will only work if all the counter/data in the column family have the exact same expiration time. And this won't work at all for say counters that you would want to start re-incrementing after expiration. But again, this is not the goal of the ticket.
> Timestamp Based Compaction Strategy
> -----------------------------------
>
> Key: CASSANDRA-2735
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2735
> Project: Cassandra
> Issue Type: New Feature
> Components: Core
> Reporter: Alan Liang
> Assignee: Alan Liang
> Priority: Minor
> Labels: compaction
> Attachments: 0004-timestamp-bucketed-compaction-strategy.patch
>
>
> Compaction strategy implementation based on max timestamp ordering of the sstables while satisfying max sstable size, min and max compaction thresholds. It also handles expiration of sstables based on a timestamp.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (CASSANDRA-2735) Timestamp Based
Compaction Strategy
Posted by "Yang Yang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13051345#comment-13051345 ]
Yang Yang edited comment on CASSANDRA-2735 at 6/17/11 9:52 PM:
---------------------------------------------------------------
there could be a problem with trying to rely on forcing compaction order to make counter expiration work:
if you base the intended order on max timestamp of each sstable, the timestamp is not trustworthy, because a single malicious client request can bump up its timestamp to the future, and arbitrarily change the order of compaction, thus rendering the approach in 2735 useless.
you can't base the order on the physical sstable flush time either, since different nodes have different flush times.
overall I think trying to fix the compaction order is not the correct direction to attack this problem: the issue here is due to the changing order between *individual* counter adds/deletes (auto-expire is same as delete), this order can be different between different counters, so you have to fix the order between the updates within each counter, not the order between *ensembles of counters*. such ensembles of counters do not guarantee any orders at all, due to randomness in flushing time, or message delivery (they have similar effects)
the problem with current counter+delete implementation is that counters use timestamp() to represent their order, but when they are merged, they lose their *individual order* and retain a max timestamp(), which supposedly represents the order of the ensemble, but this is meaningless because the it is the order of the ensemble is different from the true order.
was (Author: yangyangyyy):
there could be a problem with trying to rely on forcing compaction order to make counter expiration work:
if you base the intended order on max timestamp of each sstable, the timestamp is not trustworthy, because a single malicious client request can bump up its timestamp to the future, and arbitrarily change the order of compaction, thus rendering the approach in 2735 useless.
you can't base the order on the physical sstable flush time either, since different nodes have different flush times.
overall I think trying to fix the compaction order is not the correct direction to attack this problem: the issue here is due to the changing order between *individual* counter adds/deletes (auto-expire is same as delete), this order can be different between different counters, so you have to fix the order between the updates within each counter, not the order between *ensembles of counters*. such ensembles of counters do not guarantee any orders at all, due to randomness in flushing time, or message delivery (they have similar effects)
> Timestamp Based Compaction Strategy
> -----------------------------------
>
> Key: CASSANDRA-2735
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2735
> Project: Cassandra
> Issue Type: New Feature
> Components: Core
> Reporter: Alan Liang
> Assignee: Alan Liang
> Priority: Minor
> Labels: compaction
> Attachments: 0004-timestamp-bucketed-compaction-strategy.patch
>
>
> Compaction strategy implementation based on max timestamp ordering of the sstables while satisfying max sstable size, min and max compaction thresholds. It also handles expiration of sstables based on a timestamp.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-2735) Timestamp Based Compaction
Strategy
Posted by "Alan Liang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Alan Liang updated CASSANDRA-2735:
----------------------------------
Attachment: 0002-timestamp-bucketed-compaction-strategy.patch
Rebased once again due to change from AbstractCompactionStrategy to ICompactionStrategy #1610
> Timestamp Based Compaction Strategy
> -----------------------------------
>
> Key: CASSANDRA-2735
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2735
> Project: Cassandra
> Issue Type: New Feature
> Components: Core
> Reporter: Alan Liang
> Assignee: Alan Liang
> Priority: Minor
> Labels: compaction
> Attachments: 0002-timestamp-bucketed-compaction-strategy.patch, 0003-implemented-timestamp-bucketed-compaction-strategy-a.patch
>
>
> Compaction strategy implementation based on max timestamp ordering of the sstables while satisfying max sstable size, min and max compaction thresholds. It also handles expiration of sstables based on a timestamp.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-2735) Timestamp Based Compaction
Strategy
Posted by "Alan Liang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Alan Liang updated CASSANDRA-2735:
----------------------------------
Attachment: (was: 0004-timestamp-bucketed-compaction-strategy.patch)
> Timestamp Based Compaction Strategy
> -----------------------------------
>
> Key: CASSANDRA-2735
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2735
> Project: Cassandra
> Issue Type: New Feature
> Components: Core
> Reporter: Alan Liang
> Assignee: Alan Liang
> Priority: Minor
> Labels: compaction
> Attachments: 0001-timestamp-bucketed-compaction-strategy.patch
>
>
> Compaction strategy implementation based on max timestamp ordering of the sstables while satisfying max sstable size, min and max compaction thresholds. It also handles expiration of sstables based on a timestamp.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-2735) Timestamp Based Compaction
Strategy
Posted by "Alan Liang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Alan Liang updated CASSANDRA-2735:
----------------------------------
Attachment: 0001-timestamp-bucketed-compaction-strategy.patch
Highlights of this patch:
- Introduce a timestamp compaction strategy
- Introduce Expiration Task
- option to delete or move to expired folder
- Tests for timestamp bucketing strategy
This patch depends on https://issues.apache.org/jira/browse/CASSANDRA-2753 to be committed.
> Timestamp Based Compaction Strategy
> -----------------------------------
>
> Key: CASSANDRA-2735
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2735
> Project: Cassandra
> Issue Type: New Feature
> Components: Core
> Reporter: Alan Liang
> Assignee: Alan Liang
> Priority: Minor
> Labels: compaction
> Attachments: 0001-timestamp-bucketed-compaction-strategy.patch
>
>
> Compaction strategy implementation based on max timestamp ordering of the sstables while satisfying max sstable size, min and max compaction thresholds. It also handles expiration of sstables based on a timestamp.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2735) Timestamp Based Compaction
Strategy
Posted by "Alan Liang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13052834#comment-13052834 ]
Alan Liang commented on CASSANDRA-2735:
---------------------------------------
This compaction strategy is useful for time series data. Eg. you capture counts for each minute, hour, day. Ordering and compacting the sstables by column timestamp allows you to expire sstables more effectively compared to the size tiered approach in trunk. This is because the size tiered approach could combine an old sstable with a new sstable, which renders the sstable to look like it is quite new. You would not be able to expire the old data in this case.
> Timestamp Based Compaction Strategy
> -----------------------------------
>
> Key: CASSANDRA-2735
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2735
> Project: Cassandra
> Issue Type: New Feature
> Components: Core
> Reporter: Alan Liang
> Assignee: Alan Liang
> Priority: Minor
> Labels: compaction
> Attachments: 0004-timestamp-bucketed-compaction-strategy.patch
>
>
> Compaction strategy implementation based on max timestamp ordering of the sstables while satisfying max sstable size, min and max compaction thresholds. It also handles expiration of sstables based on a timestamp.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (CASSANDRA-2735) Timestamp Based
Compaction Strategy
Posted by "Alan Liang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13046750#comment-13046750 ]
Alan Liang edited comment on CASSANDRA-2735 at 6/9/11 8:01 PM:
---------------------------------------------------------------
Splitting out the capturing of max client supplied timestamp into a separate ticket (#2753) so that other tickets can benefit.
was (Author: alanliang):
Splitting out the capturing of max client supplied timestamp into a separate ticket so that other tickets can benefit.
> Timestamp Based Compaction Strategy
> -----------------------------------
>
> Key: CASSANDRA-2735
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2735
> Project: Cassandra
> Issue Type: New Feature
> Components: Core
> Reporter: Alan Liang
> Assignee: Alan Liang
> Priority: Minor
> Labels: compaction
> Attachments: 0002-timestamp-bucketed-compaction-strategy.patch, 0003-implemented-timestamp-bucketed-compaction-strategy-a.patch
>
>
> Compaction strategy implementation based on max timestamp ordering of the sstables while satisfying max sstable size, min and max compaction thresholds. It also handles expiration of sstables based on a timestamp.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2735) Timestamp Based Compaction
Strategy
Posted by "Alan Liang (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13117032#comment-13117032 ]
Alan Liang commented on CASSANDRA-2735:
---------------------------------------
We've tested this patch internally and we noticed that this actually resulted in a lot more compactions than the SizeTieredCompactionStrategy. The increase in IO was not acceptable for our use and therefore stopped working on this patch.
Internally, we ended up implementing expiration of sstables within SizeTieredCompactionStrategy. We've called it SizeTieredExpirableCompactionStrategy. Given a set of all sstables, the compaction procedure becomes:
1. Expire sstables based on max timestamp of the sstable. Remove expired sstables from the set.
2. Remove sstables from the set that are >= to a max size
3. Run the SizeTieredCompactionStrategy on the remaining sstables.
The downside with this strategy is that during compaction, newer sstables could be mixed with older sstables and the resultant compacted sstable gets marked with a max timestamp of the newer sstable. This means you won't be able to expire the older rows within the sstable until the entire sstable is to be expired. This problem of compacting really old sstables with newer sstables is mitigated with a restriction that an sstable is taken out of consideration for compaction if it reaches a certain max sstable size. This works because older sstables tend to be larger files.
We found this is currently working for our specific use case of storing timeseries data. I can post the patch for this SizeTieredExpirableCompactionStrategy if there is interest. I'll have to rebase it.
> Timestamp Based Compaction Strategy
> -----------------------------------
>
> Key: CASSANDRA-2735
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2735
> Project: Cassandra
> Issue Type: New Feature
> Components: Core
> Reporter: Alan Liang
> Assignee: Alan Liang
> Priority: Minor
> Labels: compaction
> Attachments: 0001-timestamp-bucketed-compaction-strategy-V2.patch, 0001-timestamp-bucketed-compaction-strategy.patch
>
>
> Compaction strategy implementation based on max timestamp ordering of the sstables while satisfying max sstable size, min and max compaction thresholds. It also handles expiration of sstables based on a timestamp.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-2735) Timestamp Based Compaction
Strategy
Posted by "Alan Liang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Alan Liang updated CASSANDRA-2735:
----------------------------------
Attachment: 0003-implemented-timestamp-bucketed-compaction-strategy-a.patch
Upload correct patch.
> Timestamp Based Compaction Strategy
> -----------------------------------
>
> Key: CASSANDRA-2735
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2735
> Project: Cassandra
> Issue Type: New Feature
> Components: Core
> Reporter: Alan Liang
> Assignee: Alan Liang
> Priority: Minor
> Labels: compaction
> Attachments: 0003-implemented-timestamp-bucketed-compaction-strategy-a.patch
>
>
> Compaction strategy implementation based on max timestamp ordering of the sstables while satisfying max sstable size, min and max compaction thresholds. It also handles expiration of sstables based on a timestamp.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-2735) Timestamp Based Compaction
Strategy
Posted by "Alan Liang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Alan Liang updated CASSANDRA-2735:
----------------------------------
Attachment: (was: 0003-implemented-timestamp-bucketed-compaction-strategy-a.patch)
> Timestamp Based Compaction Strategy
> -----------------------------------
>
> Key: CASSANDRA-2735
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2735
> Project: Cassandra
> Issue Type: New Feature
> Components: Core
> Reporter: Alan Liang
> Assignee: Alan Liang
> Priority: Minor
> Labels: compaction
> Attachments: 0003-implemented-timestamp-bucketed-compaction-strategy-a.patch
>
>
> Compaction strategy implementation based on max timestamp ordering of the sstables while satisfying max sstable size, min and max compaction thresholds. It also handles expiration of sstables based on a timestamp.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-2735) Timestamp Based Compaction
Strategy
Posted by "Alan Liang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Alan Liang updated CASSANDRA-2735:
----------------------------------
Attachment: 0001-timestamp-bucketed-compaction-strategy-V2.patch
rebased onto trunk
> Timestamp Based Compaction Strategy
> -----------------------------------
>
> Key: CASSANDRA-2735
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2735
> Project: Cassandra
> Issue Type: New Feature
> Components: Core
> Reporter: Alan Liang
> Assignee: Alan Liang
> Priority: Minor
> Labels: compaction
> Attachments: 0001-timestamp-bucketed-compaction-strategy-V2.patch, 0001-timestamp-bucketed-compaction-strategy.patch
>
>
> Compaction strategy implementation based on max timestamp ordering of the sstables while satisfying max sstable size, min and max compaction thresholds. It also handles expiration of sstables based on a timestamp.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-2735) Timestamp Based Compaction
Strategy
Posted by "Alan Liang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Alan Liang updated CASSANDRA-2735:
----------------------------------
Attachment: 0003-implemented-timestamp-bucketed-compaction-strategy-a.patch
Rebased.
> Timestamp Based Compaction Strategy
> -----------------------------------
>
> Key: CASSANDRA-2735
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2735
> Project: Cassandra
> Issue Type: New Feature
> Components: Core
> Reporter: Alan Liang
> Assignee: Alan Liang
> Priority: Minor
> Labels: compaction
> Attachments: 0003-implemented-timestamp-bucketed-compaction-strategy-a.patch
>
>
> Compaction strategy implementation based on max timestamp ordering of the sstables while satisfying max sstable size, min and max compaction thresholds. It also handles expiration of sstables based on a timestamp.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (CASSANDRA-2735) Timestamp Based
Compaction Strategy
Posted by "Yang Yang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13051345#comment-13051345 ]
Yang Yang edited comment on CASSANDRA-2735 at 6/17/11 9:47 PM:
---------------------------------------------------------------
there could be a problem with trying to rely on forcing compaction order to make counter expiration work:
if you base the intended order on max timestamp of each sstable, the timestamp is not trustworthy, because a single malicious client request can bump up its timestamp to the future, and arbitrarily change the order of compaction, thus rendering the approach in 2735 useless.
you can't base the order on the physical sstable flush time either, since different nodes have different flush times.
overall I think trying to fix the compaction order is not the correct direction to attack this problem: the issue here is due to the changing order between *individual* counter adds/deletes (auto-expire is same as delete), this order can be different between different counters, so you have to fix the order between the updates within each counter, not the order between *ensembles of counters*. such ensembles of counters do not guarantee any orders at all, due to randomness in flushing time, or message delivery (they have similar effects)
was (Author: yangyangyyy):
there could be a problem with trying to rely on forcing compaction order to make counter expiration work:
if you base the intended order on max timestamp of each sstable, the timestamp is not trustworthy, because a single malicious client request can bump up its timestamp to the future, and arbitrarily change the order of compaction, thus rendering the approach in 2735 useless.
you can't base the order on the physical sstable flush time either, since different nodes have different flush times.
overall I think trying to fix the compaction order is not the correct direction to attack this problem: the issue here is due to the changing order between individual counter adds/deletes (auto-expire is same as delete), this order can be different between different counters, so you have to fix the order between the updates within each counter, not the order between ensembles of counters. such ensembles of counters do not guarantee any orders at all, due to randomness in flushing time, or message delivery (they have similar effects)
> Timestamp Based Compaction Strategy
> -----------------------------------
>
> Key: CASSANDRA-2735
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2735
> Project: Cassandra
> Issue Type: New Feature
> Components: Core
> Reporter: Alan Liang
> Assignee: Alan Liang
> Priority: Minor
> Labels: compaction
> Attachments: 0004-timestamp-bucketed-compaction-strategy.patch
>
>
> Compaction strategy implementation based on max timestamp ordering of the sstables while satisfying max sstable size, min and max compaction thresholds. It also handles expiration of sstables based on a timestamp.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-2735) Timestamp Based Compaction
Strategy
Posted by "Alan Liang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Alan Liang updated CASSANDRA-2735:
----------------------------------
Attachment: 0004-timestamp-bucketed-compaction-strategy.patch
New patch has code just for timestamp compaction strategy.
> Timestamp Based Compaction Strategy
> -----------------------------------
>
> Key: CASSANDRA-2735
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2735
> Project: Cassandra
> Issue Type: New Feature
> Components: Core
> Reporter: Alan Liang
> Assignee: Alan Liang
> Priority: Minor
> Labels: compaction
> Attachments: 0004-timestamp-bucketed-compaction-strategy.patch
>
>
> Compaction strategy implementation based on max timestamp ordering of the sstables while satisfying max sstable size, min and max compaction thresholds. It also handles expiration of sstables based on a timestamp.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-2735) Timestamp Based Compaction
Strategy
Posted by "Alan Liang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Alan Liang updated CASSANDRA-2735:
----------------------------------
Comment: was deleted
(was: Removed unused/duplicate imports.)
> Timestamp Based Compaction Strategy
> -----------------------------------
>
> Key: CASSANDRA-2735
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2735
> Project: Cassandra
> Issue Type: New Feature
> Components: Core
> Reporter: Alan Liang
> Assignee: Alan Liang
> Priority: Minor
> Labels: compaction
> Attachments: 0003-implemented-timestamp-bucketed-compaction-strategy-a.patch
>
>
> Compaction strategy implementation based on max timestamp ordering of the sstables while satisfying max sstable size, min and max compaction thresholds. It also handles expiration of sstables based on a timestamp.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-2735) Timestamp Based Compaction
Strategy
Posted by "Alan Liang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Alan Liang updated CASSANDRA-2735:
----------------------------------
Attachment: 0003-implemented-timestamp-bucketed-compaction-strategy-a.patch
> Timestamp Based Compaction Strategy
> -----------------------------------
>
> Key: CASSANDRA-2735
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2735
> Project: Cassandra
> Issue Type: New Feature
> Components: Core
> Reporter: Alan Liang
> Assignee: Alan Liang
> Priority: Minor
> Attachments: 0003-implemented-timestamp-bucketed-compaction-strategy-a.patch
>
>
> Compaction strategy implementation based on max timestamp ordering of the sstables while satisfying max sstable size, min and max compaction thresholds. It also handles expiration of sstables based on a timestamp.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-2735) Timestamp Based Compaction
Strategy
Posted by "Alan Liang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Alan Liang updated CASSANDRA-2735:
----------------------------------
Attachment: (was: 0003-implemented-timestamp-bucketed-compaction-strategy-a.patch)
> Timestamp Based Compaction Strategy
> -----------------------------------
>
> Key: CASSANDRA-2735
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2735
> Project: Cassandra
> Issue Type: New Feature
> Components: Core
> Reporter: Alan Liang
> Assignee: Alan Liang
> Priority: Minor
> Labels: compaction
> Attachments: 0004-timestamp-bucketed-compaction-strategy.patch
>
>
> Compaction strategy implementation based on max timestamp ordering of the sstables while satisfying max sstable size, min and max compaction thresholds. It also handles expiration of sstables based on a timestamp.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-2735) Timestamp Based Compaction
Strategy
Posted by "Alan Liang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Alan Liang updated CASSANDRA-2735:
----------------------------------
Attachment: (was: 0003-implemented-timestamp-bucketed-compaction-strategy-a.patch)
> Timestamp Based Compaction Strategy
> -----------------------------------
>
> Key: CASSANDRA-2735
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2735
> Project: Cassandra
> Issue Type: New Feature
> Components: Core
> Reporter: Alan Liang
> Assignee: Alan Liang
> Priority: Minor
> Labels: compaction
> Attachments: 0003-implemented-timestamp-bucketed-compaction-strategy-a.patch
>
>
> Compaction strategy implementation based on max timestamp ordering of the sstables while satisfying max sstable size, min and max compaction thresholds. It also handles expiration of sstables based on a timestamp.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2735) Timestamp Based Compaction
Strategy
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13051347#comment-13051347 ]
Jonathan Ellis commented on CASSANDRA-2735:
-------------------------------------------
I don't think that's worth worrying about; there's lots worse things malicious clients can do, than prevent you from throwing away data that's expired. :)
> Timestamp Based Compaction Strategy
> -----------------------------------
>
> Key: CASSANDRA-2735
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2735
> Project: Cassandra
> Issue Type: New Feature
> Components: Core
> Reporter: Alan Liang
> Assignee: Alan Liang
> Priority: Minor
> Labels: compaction
> Attachments: 0004-timestamp-bucketed-compaction-strategy.patch
>
>
> Compaction strategy implementation based on max timestamp ordering of the sstables while satisfying max sstable size, min and max compaction thresholds. It also handles expiration of sstables based on a timestamp.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-2735) Timestamp Based Compaction
Strategy
Posted by "Alan Liang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Alan Liang updated CASSANDRA-2735:
----------------------------------
Attachment: (was: 0003-implemented-timestamp-bucketed-compaction-strategy-a.patch)
> Timestamp Based Compaction Strategy
> -----------------------------------
>
> Key: CASSANDRA-2735
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2735
> Project: Cassandra
> Issue Type: New Feature
> Components: Core
> Reporter: Alan Liang
> Assignee: Alan Liang
> Priority: Minor
> Labels: compaction
> Attachments: 0003-implemented-timestamp-bucketed-compaction-strategy-a.patch
>
>
> Compaction strategy implementation based on max timestamp ordering of the sstables while satisfying max sstable size, min and max compaction thresholds. It also handles expiration of sstables based on a timestamp.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2735) Timestamp Based Compaction
Strategy
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13046771#comment-13046771 ]
Jonathan Ellis commented on CASSANDRA-2735:
-------------------------------------------
... that would be CASSANDRA-2753.
> Timestamp Based Compaction Strategy
> -----------------------------------
>
> Key: CASSANDRA-2735
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2735
> Project: Cassandra
> Issue Type: New Feature
> Components: Core
> Reporter: Alan Liang
> Assignee: Alan Liang
> Priority: Minor
> Labels: compaction
> Attachments: 0002-timestamp-bucketed-compaction-strategy.patch, 0003-implemented-timestamp-bucketed-compaction-strategy-a.patch
>
>
> Compaction strategy implementation based on max timestamp ordering of the sstables while satisfying max sstable size, min and max compaction thresholds. It also handles expiration of sstables based on a timestamp.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2735) Timestamp Based Compaction
Strategy
Posted by "Yang Yang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13051345#comment-13051345 ]
Yang Yang commented on CASSANDRA-2735:
--------------------------------------
there could be a problem with trying to rely on forcing compaction order to make counter expiration work:
if you base the intended order on max timestamp of each sstable, the timestamp is not trustworthy, because a single malicious client request can bump up its timestamp to the future, and arbitrarily change the order of compaction, thus rendering the approach in 2735 useless.
you can't base the order on the physical sstable flush time either, since different nodes have different flush times.
overall I think trying to fix the compaction order is not the correct direction to attack this problem: the issue here is due to the changing order between individual counter adds/deletes (auto-expire is same as delete), this order can be different between different counters, so you have to fix the order between the updates within each counter, not the order between ensembles of counters. such ensembles of counters do not guarantee any orders at all, due to randomness in flushing time, or message delivery (they have similar effects)
> Timestamp Based Compaction Strategy
> -----------------------------------
>
> Key: CASSANDRA-2735
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2735
> Project: Cassandra
> Issue Type: New Feature
> Components: Core
> Reporter: Alan Liang
> Assignee: Alan Liang
> Priority: Minor
> Labels: compaction
> Attachments: 0004-timestamp-bucketed-compaction-strategy.patch
>
>
> Compaction strategy implementation based on max timestamp ordering of the sstables while satisfying max sstable size, min and max compaction thresholds. It also handles expiration of sstables based on a timestamp.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-2735) Timestamp Based Compaction
Strategy
Posted by "Alan Liang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Alan Liang updated CASSANDRA-2735:
----------------------------------
Attachment: 0003-implemented-timestamp-bucketed-compaction-strategy-a.patch
Removed unused/duplicate imports.
> Timestamp Based Compaction Strategy
> -----------------------------------
>
> Key: CASSANDRA-2735
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2735
> Project: Cassandra
> Issue Type: New Feature
> Components: Core
> Reporter: Alan Liang
> Assignee: Alan Liang
> Priority: Minor
> Labels: compaction
> Attachments: 0003-implemented-timestamp-bucketed-compaction-strategy-a.patch, 0003-implemented-timestamp-bucketed-compaction-strategy-a.patch
>
>
> Compaction strategy implementation based on max timestamp ordering of the sstables while satisfying max sstable size, min and max compaction thresholds. It also handles expiration of sstables based on a timestamp.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2735) Timestamp Based Compaction
Strategy
Posted by "Viktor Jevdokimov (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13108467#comment-13108467 ]
Viktor Jevdokimov commented on CASSANDRA-2735:
----------------------------------------------
When this patch will be included in any public release?
> Timestamp Based Compaction Strategy
> -----------------------------------
>
> Key: CASSANDRA-2735
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2735
> Project: Cassandra
> Issue Type: New Feature
> Components: Core
> Reporter: Alan Liang
> Assignee: Alan Liang
> Priority: Minor
> Labels: compaction
> Attachments: 0001-timestamp-bucketed-compaction-strategy-V2.patch, 0001-timestamp-bucketed-compaction-strategy.patch
>
>
> Compaction strategy implementation based on max timestamp ordering of the sstables while satisfying max sstable size, min and max compaction thresholds. It also handles expiration of sstables based on a timestamp.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2735) Timestamp Based Compaction
Strategy
Posted by "Yang Yang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13051350#comment-13051350 ]
Yang Yang commented on CASSANDRA-2735:
--------------------------------------
true, I don't really care about expired data, I'm happy as long as we have an expiring counter that "mostly" works or works with certain cautions.
but it seems that the "changing order" can come not only from compaction (which is fixed for realistic scenarios here),
but also from effects of message drops.
"compact( compact (Add1, delete), Add2) " is the same as receiving Add1, delete , Add2 in messages.
but we know that messages can be easily dropped.
so let's say the delete is dropped, then we replay it later (through repair for example), the same issue appears.
I think the latter issue can be fixed by changing the TTL reconcile rule so that reconciled death time is the older death time, not timestamp+new_TTL.
anyhow I think we users of the counters api need to understand that placing a delete shortly after your last update, or place an update shortly after delete is most likely not going to work. this patch fixes half of the issue, but it still remains.
> Timestamp Based Compaction Strategy
> -----------------------------------
>
> Key: CASSANDRA-2735
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2735
> Project: Cassandra
> Issue Type: New Feature
> Components: Core
> Reporter: Alan Liang
> Assignee: Alan Liang
> Priority: Minor
> Labels: compaction
> Attachments: 0004-timestamp-bucketed-compaction-strategy.patch
>
>
> Compaction strategy implementation based on max timestamp ordering of the sstables while satisfying max sstable size, min and max compaction thresholds. It also handles expiration of sstables based on a timestamp.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (CASSANDRA-2735) Timestamp Based
Compaction Strategy
Posted by "Yang Yang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13051350#comment-13051350 ]
Yang Yang edited comment on CASSANDRA-2735 at 6/17/11 10:06 PM:
----------------------------------------------------------------
true, I don't really care about expired data, I'm happy as long as we have an expiring counter that "mostly" works or works with certain cautions.
but it seems that the "changing order" can come not only from compaction (which is fixed for realistic scenarios here),
but also from effects of message drops.
"compact( compact (Add1, delete), Add2) " is the same as receiving Add1, delete , Add2 in messages.
but we know that messages can be easily dropped.
so let's say the delete is dropped, then we replay it later (through repair for example), so we have Add1, Add2, delete, the same issue appears.
I think the latter issue can be fixed by changing the TTL reconcile rule so that reconciled death time is the older death time, not timestamp+new_TTL.
anyhow I think we users of the counters api need to understand that placing a delete shortly after your last update, or place an update shortly after delete is most likely not going to work. this patch fixes half of the issue, but it still remains.
was (Author: yangyangyyy):
true, I don't really care about expired data, I'm happy as long as we have an expiring counter that "mostly" works or works with certain cautions.
but it seems that the "changing order" can come not only from compaction (which is fixed for realistic scenarios here),
but also from effects of message drops.
"compact( compact (Add1, delete), Add2) " is the same as receiving Add1, delete , Add2 in messages.
but we know that messages can be easily dropped.
so let's say the delete is dropped, then we replay it later (through repair for example), the same issue appears.
I think the latter issue can be fixed by changing the TTL reconcile rule so that reconciled death time is the older death time, not timestamp+new_TTL.
anyhow I think we users of the counters api need to understand that placing a delete shortly after your last update, or place an update shortly after delete is most likely not going to work. this patch fixes half of the issue, but it still remains.
> Timestamp Based Compaction Strategy
> -----------------------------------
>
> Key: CASSANDRA-2735
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2735
> Project: Cassandra
> Issue Type: New Feature
> Components: Core
> Reporter: Alan Liang
> Assignee: Alan Liang
> Priority: Minor
> Labels: compaction
> Attachments: 0004-timestamp-bucketed-compaction-strategy.patch
>
>
> Compaction strategy implementation based on max timestamp ordering of the sstables while satisfying max sstable size, min and max compaction thresholds. It also handles expiration of sstables based on a timestamp.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-2735) Timestamp Based Compaction
Strategy
Posted by "Alan Liang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Alan Liang updated CASSANDRA-2735:
----------------------------------
Attachment: (was: 0002-timestamp-bucketed-compaction-strategy.patch)
> Timestamp Based Compaction Strategy
> -----------------------------------
>
> Key: CASSANDRA-2735
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2735
> Project: Cassandra
> Issue Type: New Feature
> Components: Core
> Reporter: Alan Liang
> Assignee: Alan Liang
> Priority: Minor
> Labels: compaction
> Attachments: 0004-timestamp-bucketed-compaction-strategy.patch
>
>
> Compaction strategy implementation based on max timestamp ordering of the sstables while satisfying max sstable size, min and max compaction thresholds. It also handles expiration of sstables based on a timestamp.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2735) Timestamp Based Compaction
Strategy
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13266826#comment-13266826 ]
Jonathan Ellis commented on CASSANDRA-2735:
-------------------------------------------
I wonder if CASSANDRA-2498 (and CASSANDRA-2503) would help here. Since you get some help at read time, I would imagine you wouldn't need to be so aggressive at actual compactions, which should mitigate the extra I/O Alan mentions.
> Timestamp Based Compaction Strategy
> -----------------------------------
>
> Key: CASSANDRA-2735
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2735
> Project: Cassandra
> Issue Type: New Feature
> Components: Core
> Reporter: Alan Liang
> Assignee: Alan Liang
> Priority: Minor
> Labels: compaction
> Attachments: 0001-timestamp-bucketed-compaction-strategy-V2.patch, 0001-timestamp-bucketed-compaction-strategy.patch
>
>
> Compaction strategy implementation based on max timestamp ordering of the sstables while satisfying max sstable size, min and max compaction thresholds. It also handles expiration of sstables based on a timestamp.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira