You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Andy Burgess <an...@rbsworldpay.com> on 2011/01/19 12:36:44 UTC

Time-to-live on SuperColumns

Hi,

I'm trying to use the time-to-live functionality, which I've patched 
into Cassandra 0.6.6. My problem is that my sstables are not emptying. 
When my data expires, they do reduce in size, but only by a fraction of 
what I expected. I've come to realise that this must be because I've set 
the TTL on my Columns, but actually, I need the SuperColumns to expire 
too - I believe that my sstables remain large because I (probably) have 
a lot of empty SuperColumns. Unfortunately, I can't see a way to set a 
TTL on a SuperColumn.

Does anyone have any suggestions for what I can do here?

Actually, I've also just been playing with a few tests of "manually" 
deleting a SuperColumn, rather than using TTL - i.e. using Hector's 
Deletion with setSuper_column. Whilst this appears to work as far as 
making the data unavailable to the client is concerned, it only appears 
to reduce sstable size by about three quarters. This is in the case 
where I know that the sstable contains only my test inserts, which I 
have completely deleted. Why doesn't the sstable reduce in size to 
(almost) nothing?

Thanks,
Andy.

-- 
Andy Burgess
Principal Development Engineer
Application Delivery
WorldPay Ltd.
270-289 Science Park, Milton Road
Cambridge, CB4 0WE, United Kingdom (Depot Code: 024)
Office: +44 (0)1223 706 779| Mobile: +44 (0)7909 534 940
andy.burgess@worldpay.com

Re: Time-to-live on SuperColumns

Posted by Aaron Morton <aa...@thelastpickle.com>.
You can dump the contents of your SSTables using bin/sstable2json see http://wiki.apache.org/cassandra/Operations?#Import_.2BAC8_export

AFAIK deleting unneeded sstables is part of the GC process. If you trigger GC through JConsole you should see them disappear, the process logs the files that it deleted. 

Cheers
Aaron
 
On 20 Jan, 2011,at 03:21 AM, Andy Burgess <an...@rbsworldpay.com> wrote:

In reply to my own question, I'm clearly talking nonsense. SuperColumns are cleared out when empty (because really, what's the difference from the client point of view?) and so setting TTL at the Column level is plainly good enough. This was actually solved by dropping GCGraceSeconds right down to something very small - probably not appropriate for a production environment, but at least I know that the data will be deleted properly.

In a related question - is there any way to trigger a deletion of sstables which have already been compacted (SSTABLE-CLEANUP-TIMER thread)? These could potentially take up a lot of disk space, and are surely no longer being used. What triggers Cassandra to perform these deletions, and can I do it manually?

Thanks,
Andy.

On 19/01/11 11:36, Andy Burgess wrote:
Hi, 

I'm trying to use the time-to-live functionality, which I've patched into Cassandra 0.6.6. My problem is that my sstables are not emptying. When my data expires, they do reduce in size, but only by a fraction of what I expected. I've come to realise that this must be because I've set the TTL on my Columns, but actually, I need the SuperColumns to expire too - I believe that my sstables remain large because I (probably) have a lot of empty SuperColumns. Unfortunately, I can't see a way to set a TTL on a SuperColumn. 

Does anyone have any suggestions for what I can do here? 

Actually, I've also just been playing with a few tests of "manually" deleting a SuperColumn, rather than using TTL - i.e. using Hector's Deletion with setSuper_column. Whilst this appears to work as far as making the data unavailable to the client is concerned, it only appears to reduce sstable size by about three quarters. This is in the case where I know that the sstable contains only my test inserts, which I have completely deleted. Why doesn't the sstable reduce in size to (almost) nothing? 

Thanks, 
Andy. 


WorldPay (UK) Limited, Company No. 07316500. Registered Office: 55 Mansell Street, London E1 8AN

Authorised and regulated by the Financial Services Authority.

‘WorldPay Group’ means WorldPay (UK) Limited and its affiliates from time to time.  A reference to an “affiliate” means any Subsidiary Undertaking, any Parent Undertaking and any Subsidiary Undertaking of any such Parent Undertaking and reference to a “Parent Undertaking” or a “Subsidiary Undertaking” is to be construed in accordance with section 1162 of the Companies Act 2006, as amended.

DISCLAIMER: This email and any files transmitted with it, including replies and forwarded copies (which may contain alterations) subsequently transmitted from the WorldPay Group, are confidential and solely for the use of the intended recipient. If you are not the intended recipient (or authorised to receive for the intended recipient), you have received this email in error and any review, use, distribution or disclosure of its content is strictly prohibited. If you have received this email in error please notify the sender immediately by replying to this message. Please then delete this email and destroy any copies of it.

Messages sent to and from the WorldPay Group may be monitored to ensure compliance with internal policies and to protect our business.  Emails are not necessarily secure.  The WorldPay Group does not accept responsibility for changes made to this message after it was sent. Please note that neither the WorldPay Group nor the sender accepts any responsibility for viruses and it is the responsibility of the recipient to ensure that the onward transmission, opening or use of this message and any attachments will not adversely affect its systems or data. Anyone who communicates with us by email is taken to accept these risks. Opinions, conclusions and other information contained in this message that do not relate to the official business of the WorldPay Group shall not be understood as endorsed or given by it.

-- 
Andy Burgess
Principal Development Engineer
Application Delivery
WorldPay Ltd.
270-289 Science Park, Milton Road
Cambridge, CB4 0WE, United Kingdom (Depot Code: 024)
Office: +44 (0)1223 706 779| Mobile: +44 (0)7909 534 940
andy.burgess@worldpay.com
WorldPay (UK) Limited, Company No. 07316500. Registered Office: 55 Mansell Street, London E1 8AN

Authorised and regulated by the Financial Services Authority.

‘WorldPay Group’ means WorldPay (UK) Limited and its affiliates from time to time. A reference to an “affiliate” means any Subsidiary Undertaking, any Parent Undertaking and any Subsidiary Undertaking of any such Parent Undertaking and reference to a “Parent Undertaking” or a “Subsidiary Undertaking” is to be construed in accordance with section 1162 of the Companies Act 2006, as amended.

DISCLAIMER: This email and any files transmitted with it, including replies and forwarded copies (which may contain alterations) subsequently transmitted from the WorldPay Group, are confidential and solely for the use of the intended recipient. If you are not the intended recipient (or authorised to receive for the intended recipient), you have received this email in error and any review, use, distribution or disclosure of its content is strictly prohibited. If you have received this email in error please notify the sender immediately by replying to this message. Please then delete this email and destroy any copies of it.

Messages sent to and from the WorldPay Group may be monitored to ensure compliance with internal policies and to protect our business. Emails are not necessarily secure. The WorldPay Group does not accept responsibility for changes made to this message after it was sent. Please note that neither the WorldPay Group nor the sender accepts any responsibility for viruses and it is the responsibility of the recipient to ensure that the onward transmission, opening or use of this message and any attachments will not adversely affect its systems or data. Anyone who communicates with us by email is taken to accept these risks. Opinions, conclusions and other information contained in this message that do not relate to the official business of the WorldPay Group shall not be understood as endorsed or given by it.

Re: Time-to-live on SuperColumns

Posted by Andy Burgess <an...@rbsworldpay.com>.
WorldPay (UK) Limited, Company No. 07316500. Registered Office: 55 Mansell Street, London E1 8AN

Authorised and regulated by the Financial Services Authority.

‘WorldPay Group’ means WorldPay (UK) Limited and its affiliates from time to time.  A reference to an “affiliate” means any Subsidiary Undertaking, any Parent Undertaking and any Subsidiary Undertaking of any such Parent Undertaking and reference to a “Parent Undertaking” or a “Subsidiary Undertaking” is to be construed in accordance with section 1162 of the Companies Act 2006, as amended.

DISCLAIMER: This email and any files transmitted with it, including replies and forwarded copies (which may contain alterations) subsequently transmitted from the WorldPay Group, are confidential and solely for the use of the intended recipient. If you are not the intended recipient (or authorised to receive for the intended recipient), you have received this email in error and any review, use, distribution or disclosure of its content is strictly prohibited. If you have received this email in error please notify the sender immediately by replying to this message. Please then delete this email and destroy any copies of it.

Messages sent to and from the WorldPay Group may be monitored to ensure compliance with internal policies and to protect our business.  Emails are not necessarily secure.  The WorldPay Group does not accept responsibility for changes made to this message after it was sent. Please note that neither the WorldPay Group nor the sender accepts any responsibility for viruses and it is the responsibility of the recipient to ensure that the onward transmission, opening or use of this message and any attachments will not adversely affect its systems or data. Anyone who communicates with us by email is taken to accept these risks. Opinions, conclusions and other information contained in this message that do not relate to the official business of the WorldPay Group shall not be understood as endorsed or given by it.