Re: [Marketing Mail] Cassandra 2.1: Snapshot data changing while transferring

[Paul Dunkler <pa...@uplex.de>]

Hi Mike,

> Hi Paul, what is the value of the snapshot_before_compaction property in your cassandra.yaml?

snapshot_before_compaction: false

> Say if another snapshot is being taken (because compaction kicked in and snapshot_before_compaction property is set to TRUE) and at this moment you're tarring the snapshot folders......

Okay, totally understand. But this feature is currently disabled on our side.
> Maybe can take a look at the records in system.compaction:
> 
> select * from system.compaction_history;
> 
I did so and found a snapshot exactly starting at 01:30 (roughly at the same time the snapshot starts).
Name of the snapshotted table matches the .db-File, tar is complaining about.

What happened here is that we are saving incremental backups every 10 minutes. In that process we do a manual nodetool flush.
This nodetool flush seems to trigger compactions. Just checked the cassandra log and compactions always take place at every 10 minutes.

We are using the SizeTieredCompactionStrategy for all tables.
Is it true that - using that strategy - compaction is triggered exactly at the point where a flush (automatically or manually) is done?

Probably it would be a better idea to not do manual flushes when saving the incremental_backups (because then compactions won't happen at same time with snapshot), right?
> Regards,
> 
> Mike Yeap
> 
> 
> 
> 
> On Tue, May 31, 2016 at 5:21 PM, Paul Dunkler <paul@uplex.de <ma...@uplex.de>> wrote:
> And - as an addition:
> 
> Shoudln't that be documented that even snapshot files can change?
> 
>> I guess this might come from the incremental repairs...
>>> The repair time is stored in the sstable (RepairedAt timestamp metadata).
>> 
>> ok, that sounds interesting.
>> Could that also happen to incremental backup files as well? I had another case where incremental backup files were totally deleted automagically.
>> 
>> And - what is the suggested way to solve that problem? Should i try again tar-ing the snapshot until it doesn't happen anymore that something changes in between?
>> Or is there a way to "pause" the incremental repairs?
>> 
>> 
>>> Cheers,
>>> Reynald
>>> 
>>> On 31/05/2016 11:03, Paul Dunkler wrote:
>>>> Hi there,
>>>> 
>>>> i am sometimes running in very strange errors while backing up snapshots from a cassandra cluster.
>>>> 
>>>> Cassandra version:
>>>> 2.1.11
>>>> 
>>>> What i basically do:
>>>> 1. nodetool snapshot
>>>> 2. tar all snapshot folders into one file
>>>> 3. transfer them to another server
>>>> 
>>>> What happens is that tar just sometimes give the error message "file changed as we read it" while its adding a .db-file from the folder of the previously created snapshot.
>>>> If i understand everything correct, this SHOULD never happen. Snapshots should be totally immutable, right?
>>>> 
>>>> Am i maybe hitting a bug or is there some rare case with running repair operations or what-so-ever which can change snapshotted data?
>>>> I already searched through cassandra jira but couldn't find a bug which looks related to this behaviour.
>>>> 
>>>> Would love to get some help on this.
>>>> 
>>>> —
>>>> Paul Dunkler
>>> 
>> 
>> —
>> Paul Dunkler
> 
> —
> Paul Dunkler

—
Paul Dunkler