You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by AJ <aj...@dude.podzone.net> on 2011/06/07 07:03:46 UTC

Backups, Snapshots, SSTable Data Files, Compaction

Hi,

I am working on a backup strategy and am trying to understand what is 
going on in the data directory.

I notice that after a write to a CF and then flush, a new set of data 
files are created with an index number incremented in their names, such as:

Initially:
Users-e-1-Filter.db
Users-e-1-Index.db
Users-e-1-Statistics.db

Then, after a write to the Users CF, followed by a flush:
Users-e-2-Filter.db
Users-e-2-Index.db
Users-e-2-Statistics.db

Currently, my data dir has about 16 sets.  I thought that compaction 
(with nodetool) would clean-up these files, but it doesn't.  Neither 
does cleanup or repair.

Q1: Should the files with the lower index #'s (under the data/{keyspace} 
directory) be manually deleted?  Or, do ALL of the files in this 
directory need to be backed-up?

Q2: Can someone elaborate on the structure of these files and if they 
are interrelated?  I'm guessing that maybe each incremental set is like 
an incremental or differential backup of the SSTable, but I'm not sure.  
The reason I ask is because I hope that each set isn't a full copy of 
the data, eg, if my data set size for a given CF is 1 TB, I will not end 
up with 16 TB worth of data files after 16 calls to flush... I suspect 
not, but I'm just double-checking ;o)

Q3: When and how are these extra files removed or reduced?

Thanks!

Re: Backups, Snapshots, SSTable Data Files, Compaction

Posted by Benjamin Coverston <be...@datastax.com>.
Aaron makes a good point, the happiest customers in my opinion are the 
ones that choose nodes on the smaller side, and more of them.

Regarding the working set, I am referring to the OS cache. On linux, 
with JNA, Cassadra utilizes, to great effectiveness, memory mapped files 
and this is where I would expect most of your working set to reside.

The smaller the data set on each node the higher the proportion of CPU 
cycles, disk IO, network bandwidth, and memory you can dedicate to 
working with that data and making it work within your use case.

Ben

On 6/7/11 2:15 PM, aaron morton wrote:
> I'd also say consider what happens during maintenance and failure scenarios. Moving 10's TB around takes a lot longer than 100's GB.
>
> Cheers
>
> -----------------
> Aaron Morton
> Freelance Cassandra Developer
> @aaronmorton
> http://www.thelastpickle.com
>
> On 8 Jun 2011, at 06:40, AJ wrote:
>
>> Thanks to everyone who responded thus far.
>>
>>
>> On 6/7/2011 10:16 AM, Benjamin Coverston wrote:
>> <snip>
>>> Not to say that there aren't workloads where having many TB/Node doesn't work, but if you're planning to read from the data you're writing you do want to ensure that your working set is stored in memory.
>>>
>> Thank you Ben.  Can you elaborate some more on the above point?  Are you referring to the OS's working set or the Cassandra caches?  Why exactly do I need to ensure this?
>>
>> I am also wondering if there is any reason I should segregate my frequently write/read smallish data set (such as usage statistics) from my bulk mostly read-only data set (static content) into separate CFs if the schema allows it.  Would this be of any benefit?

-- 
Ben Coverston
Director of Operations
DataStax -- The Apache Cassandra Company
http://www.datastax.com/


Re: Backups, Snapshots, SSTable Data Files, Compaction

Posted by aaron morton <aa...@thelastpickle.com>.
I'd also say consider what happens during maintenance and failure scenarios. Moving 10's TB around takes a lot longer than 100's GB. 

Cheers

-----------------
Aaron Morton
Freelance Cassandra Developer
@aaronmorton
http://www.thelastpickle.com

On 8 Jun 2011, at 06:40, AJ wrote:

> Thanks to everyone who responded thus far.
> 
> 
> On 6/7/2011 10:16 AM, Benjamin Coverston wrote:
> <snip>
>> Not to say that there aren't workloads where having many TB/Node doesn't work, but if you're planning to read from the data you're writing you do want to ensure that your working set is stored in memory.
>> 
> 
> Thank you Ben.  Can you elaborate some more on the above point?  Are you referring to the OS's working set or the Cassandra caches?  Why exactly do I need to ensure this?
> 
> I am also wondering if there is any reason I should segregate my frequently write/read smallish data set (such as usage statistics) from my bulk mostly read-only data set (static content) into separate CFs if the schema allows it.  Would this be of any benefit?


Re: Backups, Snapshots, SSTable Data Files, Compaction

Posted by AJ <aj...@dude.podzone.net>.
Thanks to everyone who responded thus far.


On 6/7/2011 10:16 AM, Benjamin Coverston wrote:
<snip>
> Not to say that there aren't workloads where having many TB/Node 
> doesn't work, but if you're planning to read from the data you're 
> writing you do want to ensure that your working set is stored in memory.
>

Thank you Ben.  Can you elaborate some more on the above point?  Are you 
referring to the OS's working set or the Cassandra caches?  Why exactly 
do I need to ensure this?

I am also wondering if there is any reason I should segregate my 
frequently write/read smallish data set (such as usage statistics) from 
my bulk mostly read-only data set (static content) into separate CFs if 
the schema allows it.  Would this be of any benefit?

Re: Backups, Snapshots, SSTable Data Files, Compaction

Posted by Benjamin Coverston <be...@datastax.com>.
Hi AJ,

Unfortunately, for storage capacity planning it's a bit of a guessing 
game. Until you run your load against it and profile the usage you just 
are not going to know for sure. I have seen cases where planning to have 
50% excess capacity/node was plenty, and I have seen other extreme cases 
where 3x planned capacity was not enough when replica counts and entropy 
levels are high.

Cassandra will _try_ to work within the resource restrictions that you 
give it, but keep in mind that if it has excess resources in terms of 
disk space it may be a bit more lazy than you would expect in getting 
rid of some of the extra files that are sitting around waiting to be 
deleted. You know if they are scheduled to be deleted have a .compacted 
marker. If you want to actually SEE this happen use the stress.java or 
stress.py tools and do several test runs with different workloads. I 
think actually watching it happen would be enlightening for you.

Lastly while I have seen a few instances where people have chosen to use 
node sizes with 10's of TB, it is an unusual case. Most node sizing I 
have seen falls in the range of 20-250GB. Not to say that there aren't 
workloads where having many TB/Node doesn't work, but if you're planning 
to read from the data you're writing you do want to ensure that your 
working set is stored in memory.

HTH,
Ben


On 6/7/11 9:14 AM, AJ wrote:
> On 6/7/2011 2:29 AM, Maki Watanabe wrote:
>> You can find useful information in:
>> http://www.datastax.com/docs/0.8/operations/scheduled_tasks
>>
>> sstables are immutable. Once it written to disk, it won't be updated.
>> When you take snapshot, the tool makes hard links to sstable files.
>> After certain time, you will have some times of memtable flushs, so
>> your sstable files will be merged, and obsolete sstable files will be
>> removed. But snapshot set will remains on your disk, for backup.
>>
>
> Thanks for the doc source.  I will be experimenting with 0.8.0 since 
> it has many features I've been waiting for.
>
> But, still, if the snapshots don't link to all of the previous sets of 
> .db files, then those unlinked previous file sets MUST be safe to 
> manually delete.  But, they aren't deleted until later after a GC.  
> It's a bit confusing why they are kept after compaction up until GC 
> when they seem to not be needed.  We have Big Data plans... one node 
> can have 10's of TBs, so I'm trying to get an idea of how much disk 
> space will be required and whether or not I can free-up some disk space.
>
> Hopefully someone can still elaborate on this.
>
>

-- 
Ben Coverston
Director of Operations
DataStax -- The Apache Cassandra Company
http://www.datastax.com/


RE: Backups, Snapshots, SSTable Data Files, Compaction

Posted by Jeremiah Jordan <JE...@morningstar.com>.
Don't manually delete things.  Let Cassandra do it.  Force a garbage
collection or restart your instance and Cassandra will delete the unused
files.

-----Original Message-----
From: AJ [mailto:aj@dude.podzone.net] 
Sent: Tuesday, June 07, 2011 10:15 AM
To: user@cassandra.apache.org
Subject: Re: Backups, Snapshots, SSTable Data Files, Compaction

On 6/7/2011 2:29 AM, Maki Watanabe wrote:
> You can find useful information in:
> http://www.datastax.com/docs/0.8/operations/scheduled_tasks
>
> sstables are immutable. Once it written to disk, it won't be updated.
> When you take snapshot, the tool makes hard links to sstable files.
> After certain time, you will have some times of memtable flushs, so 
> your sstable files will be merged, and obsolete sstable files will be 
> removed. But snapshot set will remains on your disk, for backup.
>

Thanks for the doc source.  I will be experimenting with 0.8.0 since it
has many features I've been waiting for.

But, still, if the snapshots don't link to all of the previous sets of
.db files, then those unlinked previous file sets MUST be safe to
manually delete.  But, they aren't deleted until later after a GC.  It's
a bit confusing why they are kept after compaction up until GC when they
seem to not be needed.  We have Big Data plans... one node can have 10's
of TBs, so I'm trying to get an idea of how much disk space will be
required and whether or not I can free-up some disk space.

Hopefully someone can still elaborate on this.



Re: Backups, Snapshots, SSTable Data Files, Compaction

Posted by AJ <aj...@dude.podzone.net>.
On 6/7/2011 2:29 AM, Maki Watanabe wrote:
> You can find useful information in:
> http://www.datastax.com/docs/0.8/operations/scheduled_tasks
>
> sstables are immutable. Once it written to disk, it won't be updated.
> When you take snapshot, the tool makes hard links to sstable files.
> After certain time, you will have some times of memtable flushs, so
> your sstable files will be merged, and obsolete sstable files will be
> removed. But snapshot set will remains on your disk, for backup.
>

Thanks for the doc source.  I will be experimenting with 0.8.0 since it 
has many features I've been waiting for.

But, still, if the snapshots don't link to all of the previous sets of 
.db files, then those unlinked previous file sets MUST be safe to 
manually delete.  But, they aren't deleted until later after a GC.  It's 
a bit confusing why they are kept after compaction up until GC when they 
seem to not be needed.  We have Big Data plans... one node can have 10's 
of TBs, so I'm trying to get an idea of how much disk space will be 
required and whether or not I can free-up some disk space.

Hopefully someone can still elaborate on this.



Re: Backups, Snapshots, SSTable Data Files, Compaction

Posted by Maki Watanabe <wa...@gmail.com>.
You can find useful information in:
http://www.datastax.com/docs/0.8/operations/scheduled_tasks

sstables are immutable. Once it written to disk, it won't be updated.
When you take snapshot, the tool makes hard links to sstable files.
After certain time, you will have some times of memtable flushs, so
your sstable files will be merged, and obsolete sstable files will be
removed. But snapshot set will remains on your disk, for backup.

Assume you have sstable: A B C D E F,
When you take snapshot, you will have hard links A B C D E F under
snapshots subdirectory.
These hard links = files will not removed even after you run
major/minor compaction.

maki

2011/6/7 AJ <aj...@dude.podzone.net>:
> On 6/6/2011 11:25 PM, Benjamin Coverston wrote:
>>>
>>> Currently, my data dir has about 16 sets.  I thought that compaction
>>> (with nodetool) would clean-up these files, but it doesn't.  Neither does
>>> cleanup or repair.
>>
>> You're not even talking about snapshots using nodetool snapshot yet. Also
>> nodetool compact does compact all of the live files, however the compacted
>> SSTables will not be cleaned up until a garbage collection is triggered, or
>> a capacity threshold is met.
>
> Ok, so after a compaction, Cass is still not done with the older sets of .db
> files and I should let Cass delete them?  But, I thought one of the main
> purposes of compaction was to reclaim disk storage resources.  I'm only
> playing around with a small data set so I can't tell how fast the data
> grows.  I'm trying to plan my storage requirements.  Is each newly-generated
> set as large in size as the previous?
>
> The reason I ask is it seems a snapshot is...
>
>>> Q1: Should the files with the lower index #'s (under the data/{keyspace}
>>> directory) be manually deleted?  Or, do ALL of the files in this directory
>>> need to be backed-up?
>>
>> Do not ever delete files in your data directory if you care about data on
>> that replica, unless they are from a column family that no longer exists on
>> that server. There may be some duplicate data in the files, but if the files
>> are in the data directory, as a general rule, they are there because they
>> contain some set of data that is in none of the other SSTables.
>
> ... It seems a snapshot is implemented, unsurprisingly,  as just a link to
> the latest (highest indexed) set; not the previous sets.  So, obviously,
> only the latest *.db files will get backed-up.  Therefore, the previous sets
> must be worthless.
>
>



-- 
w3m

Re: Backups, Snapshots, SSTable Data Files, Compaction

Posted by AJ <aj...@dude.podzone.net>.
On 6/6/2011 11:25 PM, Benjamin Coverston wrote:
>> Currently, my data dir has about 16 sets.  I thought that compaction 
>> (with nodetool) would clean-up these files, but it doesn't.  Neither 
>> does cleanup or repair.
>
> You're not even talking about snapshots using nodetool snapshot yet. 
> Also nodetool compact does compact all of the live files, however the 
> compacted SSTables will not be cleaned up until a garbage collection 
> is triggered, or a capacity threshold is met.

Ok, so after a compaction, Cass is still not done with the older sets of 
.db files and I should let Cass delete them?  But, I thought one of the 
main purposes of compaction was to reclaim disk storage resources.  I'm 
only playing around with a small data set so I can't tell how fast the 
data grows.  I'm trying to plan my storage requirements.  Is each 
newly-generated set as large in size as the previous?

The reason I ask is it seems a snapshot is...

>> Q1: Should the files with the lower index #'s (under the 
>> data/{keyspace} directory) be manually deleted?  Or, do ALL of the 
>> files in this directory need to be backed-up?
> Do not ever delete files in your data directory if you care about data 
> on that replica, unless they are from a column family that no longer 
> exists on that server. There may be some duplicate data in the files, 
> but if the files are in the data directory, as a general rule, they 
> are there because they contain some set of data that is in none of the 
> other SSTables.

... It seems a snapshot is implemented, unsurprisingly,  as just a link 
to the latest (highest indexed) set; not the previous sets.  So, 
obviously, only the latest *.db files will get backed-up.  Therefore, 
the previous sets must be worthless.


Re: Backups, Snapshots, SSTable Data Files, Compaction

Posted by Benjamin Coverston <be...@datastax.com>.
Hi AJ,

inline:

On 6/6/11 11:03 PM, AJ wrote:
> Hi,
>
> I am working on a backup strategy and am trying to understand what is 
> going on in the data directory.
>
> I notice that after a write to a CF and then flush, a new set of data 
> files are created with an index number incremented in their names, 
> such as:
>
> Initially:
> Users-e-1-Filter.db
> Users-e-1-Index.db
> Users-e-1-Statistics.db
>
> Then, after a write to the Users CF, followed by a flush:
> Users-e-2-Filter.db
> Users-e-2-Index.db
> Users-e-2-Statistics.db
>
> Currently, my data dir has about 16 sets.  I thought that compaction 
> (with nodetool) would clean-up these files, but it doesn't.  Neither 
> does cleanup or repair.
You're not even talking about snapshots using nodetool snapshot yet. 
Also nodetool compact does compact all of the live files, however the 
compacted SSTables will not be cleaned up until a garbage collection is 
triggered, or a capacity threshold is met.
> Q1: Should the files with the lower index #'s (under the 
> data/{keyspace} directory) be manually deleted?  Or, do ALL of the 
> files in this directory need to be backed-up?
Do not ever delete files in your data directory if you care about data 
on that replica, unless they are from a column family that no longer 
exists on that server. There may be some duplicate data in the files, 
but if the files are in the data directory, as a general rule, they are 
there because they contain some set of data that is in none of the other 
SSTables.
> Q2: Can someone elaborate on the structure of these files and if they 
> are interrelated?  I'm guessing that maybe each incremental set is 
> like an incremental or differential backup of the SSTable, but I'm not 
> sure.  The reason I ask is because I hope that each set isn't a full 
> copy of the data, eg, if my data set size for a given CF is 1 TB, I 
> will not end up with 16 TB worth of data files after 16 calls to 
> flush... I suspect not, but I'm just double-checking ;o)
>
The are interrelated only in the sense that they contain data associated 
with the same column family. Each set may contain a complete, partial, 
or entirely independent set of the data depending on your write pattern 
and the frequency of minor compactions.
> Q3: When and how are these extra files removed or reduced?
A GC or a threshold.
>
> Thanks!

-- 
Ben Coverston
Director of Operations
DataStax -- The Apache Cassandra Company
http://www.datastax.com/