You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Jonathan Ellis (Created) (JIRA)" <ji...@apache.org> on 2011/11/02 04:36:32 UTC
[jira] [Created] (CASSANDRA-3442) TTL histogram for sstable
metadata
TTL histogram for sstable metadata
----------------------------------
Key: CASSANDRA-3442
URL: https://issues.apache.org/jira/browse/CASSANDRA-3442
Project: Cassandra
Issue Type: Improvement
Components: Core
Reporter: Jonathan Ellis
Priority: Minor
Under size-tiered compaction, you can generate large sstables that compact infrequently. With expiring columns mixed in, we could waste a lot of space in this situation.
If we kept a TTL EstimatedHistogram in the sstable metadata, we could do a single-sstable compaction aginst sstables with over 20% (?) expired data.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3442) TTL histogram for sstable
metadata
Posted by "Sylvain Lebresne (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13142006#comment-13142006 ]
Sylvain Lebresne commented on CASSANDRA-3442:
---------------------------------------------
I think the idea has merits.
We have to keep in mind that unless the expired data is also gcable, this would only transform expiring columns to tombstones, so for those that have very small columns (~4 bytes) this would be useless. I would also maybe bump that 20% up to be sure we don't rewrite too ofen sstables that are good candidate for compaction anyway. But on principle this is a good idea I believe.
> TTL histogram for sstable metadata
> ----------------------------------
>
> Key: CASSANDRA-3442
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3442
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Reporter: Jonathan Ellis
> Assignee: Sylvain Lebresne
> Priority: Minor
> Labels: compaction
>
> Under size-tiered compaction, you can generate large sstables that compact infrequently. With expiring columns mixed in, we could waste a lot of space in this situation.
> If we kept a TTL EstimatedHistogram in the sstable metadata, we could do a single-sstable compaction aginst sstables with over 20% (?) expired data.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3442) TTL histogram for sstable
metadata
Posted by "Sylvain Lebresne (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13218057#comment-13218057 ]
Sylvain Lebresne commented on CASSANDRA-3442:
---------------------------------------------
Agreed on targeting for 1.2. There is now a branch for it though if that's ready (the patches probably need rebase though).
> TTL histogram for sstable metadata
> ----------------------------------
>
> Key: CASSANDRA-3442
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3442
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Reporter: Jonathan Ellis
> Assignee: Yuki Morishita
> Priority: Minor
> Labels: compaction
> Fix For: 1.2
>
> Attachments: cassandra-1.1-3442.txt
>
>
> Under size-tiered compaction, you can generate large sstables that compact infrequently. With expiring columns mixed in, we could waste a lot of space in this situation.
> If we kept a TTL EstimatedHistogram in the sstable metadata, we could do a single-sstable compaction aginst sstables with over 20% (?) expired data.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3442) TTL histogram for sstable
metadata
Posted by "Jonathan Ellis (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13218226#comment-13218226 ]
Jonathan Ellis commented on CASSANDRA-3442:
-------------------------------------------
Yes, I'm working on an updated patch.
> TTL histogram for sstable metadata
> ----------------------------------
>
> Key: CASSANDRA-3442
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3442
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Reporter: Jonathan Ellis
> Assignee: Yuki Morishita
> Priority: Minor
> Labels: compaction
> Fix For: 1.2
>
> Attachments: cassandra-1.1-3442.txt
>
>
> Under size-tiered compaction, you can generate large sstables that compact infrequently. With expiring columns mixed in, we could waste a lot of space in this situation.
> If we kept a TTL EstimatedHistogram in the sstable metadata, we could do a single-sstable compaction aginst sstables with over 20% (?) expired data.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3442) TTL histogram for sstable
metadata
Posted by "Jonathan Ellis (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3442?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-3442:
--------------------------------------
Reviewer: jbellis (was: slebresne)
> TTL histogram for sstable metadata
> ----------------------------------
>
> Key: CASSANDRA-3442
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3442
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Reporter: Jonathan Ellis
> Assignee: Yuki Morishita
> Priority: Minor
> Labels: compaction
> Fix For: 1.1
>
> Attachments: 3442.txt
>
>
> Under size-tiered compaction, you can generate large sstables that compact infrequently. With expiring columns mixed in, we could waste a lot of space in this situation.
> If we kept a TTL EstimatedHistogram in the sstable metadata, we could do a single-sstable compaction aginst sstables with over 20% (?) expired data.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3442) TTL histogram for sstable
metadata
Posted by "B. Todd Burruss (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13493439#comment-13493439 ]
B. Todd Burruss commented on CASSANDRA-3442:
--------------------------------------------
Will this patch force clean up *any* tombstoned data, or only because of columns that have expired because of their TTL?
> TTL histogram for sstable metadata
> ----------------------------------
>
> Key: CASSANDRA-3442
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3442
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Reporter: Jonathan Ellis
> Assignee: Yuki Morishita
> Priority: Minor
> Labels: compaction
> Fix For: 1.2.0 beta 1
>
> Attachments: 3442-track-tombstones.txt, 3442-v3.txt, cassandra-1.1-3442.txt
>
>
> Under size-tiered compaction, you can generate large sstables that compact infrequently. With expiring columns mixed in, we could waste a lot of space in this situation.
> If we kept a TTL EstimatedHistogram in the sstable metadata, we could do a single-sstable compaction aginst sstables with over 20% (?) expired data.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (CASSANDRA-3442) TTL histogram for sstable
metadata
Posted by "Jonathan Ellis (Assigned) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3442?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis reassigned CASSANDRA-3442:
-----------------------------------------
Assignee: Yuki Morishita (was: Sylvain Lebresne)
We can get average column size from row size / column count, so we could make it "If column size > N bytes or sstable is older than gc_grace_period."
> TTL histogram for sstable metadata
> ----------------------------------
>
> Key: CASSANDRA-3442
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3442
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Reporter: Jonathan Ellis
> Assignee: Yuki Morishita
> Priority: Minor
> Labels: compaction
>
> Under size-tiered compaction, you can generate large sstables that compact infrequently. With expiring columns mixed in, we could waste a lot of space in this situation.
> If we kept a TTL EstimatedHistogram in the sstable metadata, we could do a single-sstable compaction aginst sstables with over 20% (?) expired data.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3442) TTL histogram for sstable
metadata
Posted by "Jonathan Ellis (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13205649#comment-13205649 ]
Jonathan Ellis commented on CASSANDRA-3442:
-------------------------------------------
I'm going to wait to apply this because Sylvain asked us to use trunk for 1.1.1 for now, and IMO this should go in 1.2.
> TTL histogram for sstable metadata
> ----------------------------------
>
> Key: CASSANDRA-3442
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3442
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Reporter: Jonathan Ellis
> Assignee: Yuki Morishita
> Priority: Minor
> Labels: compaction
> Fix For: 1.2
>
> Attachments: cassandra-1.1-3442.txt
>
>
> Under size-tiered compaction, you can generate large sstables that compact infrequently. With expiring columns mixed in, we could waste a lot of space in this situation.
> If we kept a TTL EstimatedHistogram in the sstable metadata, we could do a single-sstable compaction aginst sstables with over 20% (?) expired data.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3442) TTL histogram for sstable
metadata
Posted by "Yuki Morishita (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3442?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Yuki Morishita updated CASSANDRA-3442:
--------------------------------------
Attachment: cassandra-1.1-3442.txt
I modified the patch to perform single sstable compaction when nothing else can be compacted.
I added new single_compaction_interval option which defaults to 24 hours in order to prevent recursive compaction on the same sstable.
I also added unit test to CompactionsTest.
> TTL histogram for sstable metadata
> ----------------------------------
>
> Key: CASSANDRA-3442
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3442
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Reporter: Jonathan Ellis
> Assignee: Yuki Morishita
> Priority: Minor
> Labels: compaction
> Fix For: 1.1.1
>
> Attachments: cassandra-1.1-3442.txt
>
>
> Under size-tiered compaction, you can generate large sstables that compact infrequently. With expiring columns mixed in, we could waste a lot of space in this situation.
> If we kept a TTL EstimatedHistogram in the sstable metadata, we could do a single-sstable compaction aginst sstables with over 20% (?) expired data.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3442) TTL histogram for sstable
metadata
Posted by "Yuki Morishita (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13493442#comment-13493442 ]
Yuki Morishita commented on CASSANDRA-3442:
-------------------------------------------
the former.
> TTL histogram for sstable metadata
> ----------------------------------
>
> Key: CASSANDRA-3442
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3442
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Reporter: Jonathan Ellis
> Assignee: Yuki Morishita
> Priority: Minor
> Labels: compaction
> Fix For: 1.2.0 beta 1
>
> Attachments: 3442-track-tombstones.txt, 3442-v3.txt, cassandra-1.1-3442.txt
>
>
> Under size-tiered compaction, you can generate large sstables that compact infrequently. With expiring columns mixed in, we could waste a lot of space in this situation.
> If we kept a TTL EstimatedHistogram in the sstable metadata, we could do a single-sstable compaction aginst sstables with over 20% (?) expired data.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3442) TTL histogram for sstable
metadata
Posted by "Jonathan Ellis (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3442?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-3442:
--------------------------------------
Reviewer: slebresne
Fix Version/s: 1.1
> TTL histogram for sstable metadata
> ----------------------------------
>
> Key: CASSANDRA-3442
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3442
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Reporter: Jonathan Ellis
> Assignee: Yuki Morishita
> Priority: Minor
> Labels: compaction
> Fix For: 1.1
>
> Attachments: 3442.txt
>
>
> Under size-tiered compaction, you can generate large sstables that compact infrequently. With expiring columns mixed in, we could waste a lot of space in this situation.
> If we kept a TTL EstimatedHistogram in the sstable metadata, we could do a single-sstable compaction aginst sstables with over 20% (?) expired data.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3442) TTL histogram for sstable
metadata
Posted by "Jonathan Ellis (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13154250#comment-13154250 ]
Jonathan Ellis commented on CASSANDRA-3442:
-------------------------------------------
Should we do single-sstable compactions *after* the bucket compactions? Doing them first means we might compact them twice, when the bucket-based compaction would have been adequate.
It looks like this will never stop recompacting sstables with high expiring column counts, until they finally expire and are expunged. I think we need to address this somehow, possibly by waiting until some fraction of gc_grace_seconds has elapsed since sstable creation (which we can just get from mtime).
If we can reasonably test this in CompactionsTest I'd like to add that.
> TTL histogram for sstable metadata
> ----------------------------------
>
> Key: CASSANDRA-3442
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3442
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Reporter: Jonathan Ellis
> Assignee: Yuki Morishita
> Priority: Minor
> Labels: compaction
> Fix For: 1.1
>
> Attachments: 3442.txt
>
>
> Under size-tiered compaction, you can generate large sstables that compact infrequently. With expiring columns mixed in, we could waste a lot of space in this situation.
> If we kept a TTL EstimatedHistogram in the sstable metadata, we could do a single-sstable compaction aginst sstables with over 20% (?) expired data.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3442) TTL histogram for sstable
metadata
Posted by "Yuki Morishita (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3442?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Yuki Morishita updated CASSANDRA-3442:
--------------------------------------
Attachment: (was: 3442.txt)
> TTL histogram for sstable metadata
> ----------------------------------
>
> Key: CASSANDRA-3442
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3442
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Reporter: Jonathan Ellis
> Assignee: Yuki Morishita
> Priority: Minor
> Labels: compaction
> Fix For: 1.1.1
>
>
> Under size-tiered compaction, you can generate large sstables that compact infrequently. With expiring columns mixed in, we could waste a lot of space in this situation.
> If we kept a TTL EstimatedHistogram in the sstable metadata, we could do a single-sstable compaction aginst sstables with over 20% (?) expired data.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3442) TTL histogram for sstable
metadata
Posted by "Yuki Morishita (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13218762#comment-13218762 ]
Yuki Morishita commented on CASSANDRA-3442:
-------------------------------------------
{quote}
I switched from checking instanceof ExipiringColumn, to instanceof DeletedColumn, ...
{quote}
I don't see this switch in your patch, but as you described, tracking tombstones makes more sense.
I think it's doable to create histogram of dropping time when writing sstable and use that for single sstable compaction.
So we can trigger compaction on single sstable which contains say 20% of ExpiringColumns and DeletedColumns and 50% of them can be dropped.
{quote}
Minor note: the new test seems fairly involved – what would we lose by just testing compaction of a single sstable w/ tombstones?
{quote}
Well, nothing :)
single sstable compaction test is fine.
> TTL histogram for sstable metadata
> ----------------------------------
>
> Key: CASSANDRA-3442
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3442
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Reporter: Jonathan Ellis
> Assignee: Yuki Morishita
> Priority: Minor
> Labels: compaction
> Fix For: 1.2
>
> Attachments: 3442-v3.txt, cassandra-1.1-3442.txt
>
>
> Under size-tiered compaction, you can generate large sstables that compact infrequently. With expiring columns mixed in, we could waste a lot of space in this situation.
> If we kept a TTL EstimatedHistogram in the sstable metadata, we could do a single-sstable compaction aginst sstables with over 20% (?) expired data.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3442) TTL histogram for sstable
metadata
Posted by "Yuki Morishita (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3442?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Yuki Morishita updated CASSANDRA-3442:
--------------------------------------
Attachment: 3442.txt
Patch is against 1.0 branch. Added TTL histogram and logic to determine whether to perform compaction over sstables that have expiring columns more than threshold.
Threshold is default to 20% (0.2) but you can specify via compaction_strategy_options.
> TTL histogram for sstable metadata
> ----------------------------------
>
> Key: CASSANDRA-3442
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3442
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Reporter: Jonathan Ellis
> Assignee: Yuki Morishita
> Priority: Minor
> Labels: compaction
> Attachments: 3442.txt
>
>
> Under size-tiered compaction, you can generate large sstables that compact infrequently. With expiring columns mixed in, we could waste a lot of space in this situation.
> If we kept a TTL EstimatedHistogram in the sstable metadata, we could do a single-sstable compaction aginst sstables with over 20% (?) expired data.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3442) TTL histogram for sstable
metadata
Posted by "Jonathan Ellis (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3442?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-3442:
--------------------------------------
Fix Version/s: (was: 1.1.1)
1.2
> TTL histogram for sstable metadata
> ----------------------------------
>
> Key: CASSANDRA-3442
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3442
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Reporter: Jonathan Ellis
> Assignee: Yuki Morishita
> Priority: Minor
> Labels: compaction
> Fix For: 1.2
>
> Attachments: cassandra-1.1-3442.txt
>
>
> Under size-tiered compaction, you can generate large sstables that compact infrequently. With expiring columns mixed in, we could waste a lot of space in this situation.
> If we kept a TTL EstimatedHistogram in the sstable metadata, we could do a single-sstable compaction aginst sstables with over 20% (?) expired data.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3442) TTL histogram for sstable
metadata
Posted by "Yuki Morishita (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3442?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Yuki Morishita updated CASSANDRA-3442:
--------------------------------------
Attachment: 3442-track-tombstones.txt
Patch attached to track tombstones by creating its drop time histogram.
Size tiered compaction strategy uses this to calculate fraction of droppable tombstones at compaction and perform single sstable compaction if the fraction exceeds threshold.
Note that original patch overcounted ExpiringColumn inside SuperColumn. Overall column count is done at SuperColumn level, so tombstone count should be done at the same level. Newer patch counts tombstones by simply checking its local deletion time < Integer.MAX_VALUE.
I also rewrite unit test to simply create one sstable with tombstones and let it get compacted.
> TTL histogram for sstable metadata
> ----------------------------------
>
> Key: CASSANDRA-3442
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3442
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Reporter: Jonathan Ellis
> Assignee: Yuki Morishita
> Priority: Minor
> Labels: compaction
> Fix For: 1.2
>
> Attachments: 3442-track-tombstones.txt, 3442-v3.txt, cassandra-1.1-3442.txt
>
>
> Under size-tiered compaction, you can generate large sstables that compact infrequently. With expiring columns mixed in, we could waste a lot of space in this situation.
> If we kept a TTL EstimatedHistogram in the sstable metadata, we could do a single-sstable compaction aginst sstables with over 20% (?) expired data.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3442) TTL histogram for sstable
metadata
Posted by "Jonathan Ellis (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3442?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-3442:
--------------------------------------
Attachment: 3442-v3.txt
v3 attached. It's a bit of two steps forward, one step back:
- renames RowStats -> ColumnStats; separates row size computation from column/tombstone counts, moves ColumnStats computation out of serializer and into AbstractCompactedRow
- I switched from checking instanceof ExipiringColumn, to instanceof DeletedColumn, since an ExpiringColumn just means it will expire eventually (at which point it turns into a DeletedColumn), whereas a DeletedColumn is a tombstone that will be eligible for dropping after gc_grace_seconds. A common use case for TTL is to expire all data in a row after N days; if we're just going by "this column has a TTL" we'll compact these sstables daily even if none of the data has actually expired yet. Switching to checking for DC instead mitigates this a little.
However, the more I think about it, the more I think what we *really* want to track is a histogram of *when tombstones are eligible to be dropped*, relative to the sstable creation time. So, if I had a column that expired after 30 days, and a gc_grace_seconds of 10 days, I'd add an entry for 40 days to the histogram. If I had a new manual delete, I'd add an entry for 10 days.
This would allow us to have a good estimate of *how much of the sstable could actually be cleaned out by compaction*, and we could drop the single_compaction_interval code entirely.
What do you think?
Minor note: the new test seems fairly involved -- what would we lose by just testing compaction of a single sstable w/ tombstones?
> TTL histogram for sstable metadata
> ----------------------------------
>
> Key: CASSANDRA-3442
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3442
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Reporter: Jonathan Ellis
> Assignee: Yuki Morishita
> Priority: Minor
> Labels: compaction
> Fix For: 1.2
>
> Attachments: 3442-v3.txt, cassandra-1.1-3442.txt
>
>
> Under size-tiered compaction, you can generate large sstables that compact infrequently. With expiring columns mixed in, we could waste a lot of space in this situation.
> If we kept a TTL EstimatedHistogram in the sstable metadata, we could do a single-sstable compaction aginst sstables with over 20% (?) expired data.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (CASSANDRA-3442) TTL histogram for sstable
metadata
Posted by "Jonathan Ellis (Assigned) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3442?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis reassigned CASSANDRA-3442:
-----------------------------------------
Assignee: Sylvain Lebresne
Sylvain, what do you think of this idea? (If it has merit, we can assign to Yuki for implementation.)
> TTL histogram for sstable metadata
> ----------------------------------
>
> Key: CASSANDRA-3442
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3442
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Reporter: Jonathan Ellis
> Assignee: Sylvain Lebresne
> Priority: Minor
> Labels: compaction
>
> Under size-tiered compaction, you can generate large sstables that compact infrequently. With expiring columns mixed in, we could waste a lot of space in this situation.
> If we kept a TTL EstimatedHistogram in the sstable metadata, we could do a single-sstable compaction aginst sstables with over 20% (?) expired data.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3442) TTL histogram for sstable
metadata
Posted by "Yuki Morishita (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13205640#comment-13205640 ]
Yuki Morishita commented on CASSANDRA-3442:
-------------------------------------------
Patch also can apply to trunk at this moment.
> TTL histogram for sstable metadata
> ----------------------------------
>
> Key: CASSANDRA-3442
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3442
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Reporter: Jonathan Ellis
> Assignee: Yuki Morishita
> Priority: Minor
> Labels: compaction
> Fix For: 1.2
>
> Attachments: cassandra-1.1-3442.txt
>
>
> Under size-tiered compaction, you can generate large sstables that compact infrequently. With expiring columns mixed in, we could waste a lot of space in this situation.
> If we kept a TTL EstimatedHistogram in the sstable metadata, we could do a single-sstable compaction aginst sstables with over 20% (?) expired data.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira