RE: [EXTERNAL] Re: Default TTL on CF

["Durity, Sean R" <SE...@homedepot.com>]

I spent a month of my life on similar problem... There wasn't an easy answer, but this is what I did

#1 - Stop the problem from growing further. Get new inserts using a TTL (or set the default on the table so they get it). App team had to do this one.
#2 - Delete any data  that should already be expired.
- In my case the partition key included a date in the composite string they had built. So I could know from the partition key if the data needed to be deleted. I used sstablekeys to get the keys into files on each host. Then I parsed the files and created deletes for only those expired records. Then I executed the deletes. Then I had to do some compaction to actually create disk space. A long process with hundreds of billions of records...
#3 - Add TTL to data that should live. I gave this to the app team. Using the extracted keys I gave them, they could calculate the proper TTL. They read the data with the key, calculated TTL, and rewrote the data with TTL. Long, boring, etc. but they did it.



Sean Durity

-----Original Message-----
From: Jeff Jirsa <jj...@gmail.com>
Sent: Thursday, March 14, 2019 9:30 AM
To: user@cassandra.apache.org
Subject: [EXTERNAL] Re: Default TTL on CF

SSTableReader and CQLSSTableWriter if you’re comfortable with Java


--
Jeff Jirsa


> On Mar 14, 2019, at 1:28 PM, Nick Hatfield <ni...@metricly.com> wrote:
>
> Bummer but, reasonable. Any cool tricks I could use to make that process
> easier? I have many TB of data on a live cluster and was hoping to
> starting cleaning out the earlier bad habits of data housekeeping
>
>> On 3/14/19, 9:24 AM, "Jeff Jirsa" <jj...@gmail.com> wrote:
>>
>> It does not impact existing data
>>
>> The data gets an expiration time stamp when you write it. Changing the
>> default only impacts newly written data
>>
>> If you need to change the expiration time on existing data, you must
>> update it
>>
>>
>> --
>> Jeff Jirsa
>>
>>
>>> On Mar 14, 2019, at 1:16 PM, Nick Hatfield <ni...@metricly.com>
>>> wrote:
>>>
>>> Hello,
>>>
>>> Can anyone tell me if setting a default TTL will affect existing data?
>>> I would like to enable a default TTL and have cassandra add that TTL to
>>> any rows that don¹t currently have a TTL set.
>>>
>>> Thanks,
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
>> For additional commands, e-mail: user-help@cassandra.apache.org
>>
>>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
> For additional commands, e-mail: user-help@cassandra.apache.org
>

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
For additional commands, e-mail: user-help@cassandra.apache.org


________________________________

The information in this Internet Email is confidential and may be legally privileged. It is intended solely for the addressee. Access to this Email by anyone else is unauthorized. If you are not the intended recipient, any disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it, is prohibited and may be unlawful. When addressed to our clients any opinions or advice contained in this Email are subject to the terms and conditions expressed in any applicable governing The Home Depot terms of business or client engagement letter. The Home Depot disclaims all responsibility and liability for the accuracy and content of this attachment and for any damages or losses arising from any inaccuracies, errors, viruses, e.g., worms, trojan horses, etc., or other items of a destructive nature, which may be contained in this attachment and shall not be liable for direct, indirect, consequential or special damages in connection with this e-mail message or its attachment.

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
For additional commands, e-mail: user-help@cassandra.apache.org