You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Roshan <co...@gmail.com> on 2012/04/28 15:04:06 UTC
Cassandra backup queston regarding commitlogs
Hi
Currently I am taking daily snapshot on my keyspace in production and
already enable the incremental backups as well.
According to the documentation, the incremental backup option will create an
hard-link to the backup folder when new sstable is flushed. Snapshot will
copy all the data/index/etc. files to a new folder.
*Question:*
What will happen (with enabling the incremental backup) when crash (due to
any reason) the Cassandra before flushing the data as a SSTable (inserted
data still in commitlog). In this case how can I backup/restore data?
Do I need to backup the commitlogs as well and and replay during the server
start to restore the data in commitlog files?
Thanks.
--
View this message in context: http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Cassandra-backup-queston-regarding-commitlogs-tp7508823.html
Sent from the cassandra-user@incubator.apache.org mailing list archive at Nabble.com.
Re: Cassandra backup queston regarding commitlogs
Posted by aaron morton <aa...@thelastpickle.com>.
> 1. If I already have a Cassandra cluster running, would changing the incremental_backups parameter in the cassandra.yaml of each node, and then restart it do the trick?
Yes it is a per node setting.
> 2. Assuming I am creating a daily snapshot, what is the gain from setting incremental backup to true?
Better point in time recovery on a node.
Cheers
-----------------
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com
On 29/04/2012, at 6:41 PM, Tamar Fraenkel wrote:
> I want to add a couple of questions regrading incremental backups:
> 1. If I already have a Cassandra cluster running, would changing the incremental_backups parameter in the cassandra.yaml of each node, and then restart it do the trick?
> 2. Assuming I am creating a daily snapshot, what is the gain from setting incremental backup to true?
>
> Thanks,
> Tamar
>
> Tamar Fraenkel
> Senior Software Engineer, TOK Media
>
> <tokLogo.png>
>
> tamar@tok-media.com
> Tel: +972 2 6409736
> Mob: +972 54 8356490
> Fax: +972 2 5612956
>
>
>
>
>
> On Sat, Apr 28, 2012 at 4:04 PM, Roshan <co...@gmail.com> wrote:
> Hi
>
> Currently I am taking daily snapshot on my keyspace in production and
> already enable the incremental backups as well.
>
> According to the documentation, the incremental backup option will create an
> hard-link to the backup folder when new sstable is flushed. Snapshot will
> copy all the data/index/etc. files to a new folder.
>
> *Question:*
> What will happen (with enabling the incremental backup) when crash (due to
> any reason) the Cassandra before flushing the data as a SSTable (inserted
> data still in commitlog). In this case how can I backup/restore data?
>
> Do I need to backup the commitlogs as well and and replay during the server
> start to restore the data in commitlog files?
>
> Thanks.
>
>
>
> --
> View this message in context: http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Cassandra-backup-queston-regarding-commitlogs-tp7508823.html
> Sent from the cassandra-user@incubator.apache.org mailing list archive at Nabble.com.
>
Re: Cassandra backup queston regarding commitlogs
Posted by Roshan <co...@gmail.com>.
Tamar
Please don't jump to other users discussions. If you want to ask any issue,
create a new one, please.
Thanks.
--
View this message in context: http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Cassandra-backup-question-regarding-commitlogs-tp7508823p7511913.html
Sent from the cassandra-user@incubator.apache.org mailing list archive at Nabble.com.
Re: Cassandra backup queston regarding commitlogs
Posted by Tamar Fraenkel <ta...@tok-media.com>.
I want to add a couple of questions regrading incremental backups:
1. If I already have a Cassandra cluster running, would changing the i
ncremental_backups parameter in the cassandra.yaml of each node, and then
restart it do the trick?
2. Assuming I am creating a daily snapshot, what is the gain from setting
incremental backup to true?
Thanks,
Tamar
*Tamar Fraenkel *
Senior Software Engineer, TOK Media
[image: Inline image 1]
tamar@tok-media.com
Tel: +972 2 6409736
Mob: +972 54 8356490
Fax: +972 2 5612956
On Sat, Apr 28, 2012 at 4:04 PM, Roshan <co...@gmail.com> wrote:
> Hi
>
> Currently I am taking daily snapshot on my keyspace in production and
> already enable the incremental backups as well.
>
> According to the documentation, the incremental backup option will create
> an
> hard-link to the backup folder when new sstable is flushed. Snapshot will
> copy all the data/index/etc. files to a new folder.
>
> *Question:*
> What will happen (with enabling the incremental backup) when crash (due to
> any reason) the Cassandra before flushing the data as a SSTable (inserted
> data still in commitlog). In this case how can I backup/restore data?
>
> Do I need to backup the commitlogs as well and and replay during the server
> start to restore the data in commitlog files?
>
> Thanks.
>
>
>
> --
> View this message in context:
> http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Cassandra-backup-queston-regarding-commitlogs-tp7508823.html
> Sent from the cassandra-user@incubator.apache.org mailing list archive at
> Nabble.com.
>
Re: Cassandra backup queston regarding commitlogs
Posted by Roshan <co...@gmail.com>.
Many thanks Aaron. I will post a support issue for them. But will keep the
snapshot + incremental backups + commitlogs to recover any failure
situation.
--
View this message in context: http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Cassandra-backup-queston-regarding-commitlogs-tp7508823p7518866.html
Sent from the cassandra-user@incubator.apache.org mailing list archive at Nabble.com.
Re: Cassandra backup queston regarding commitlogs
Posted by aaron morton <aa...@thelastpickle.com>.
If you delete the commit logs you are rolling back to exactly what was in the snapshot. When you take a snapshot it flushes the memtables first, so there is nothing in the commit log that is not in the snapshot. Rolling back to a snapshot is rollback to that point in time.
If you want to restore to any point in time you need snapshots + incremental snapshot + commit log (for things that have not made it to sstables). Otherwise there is a potential loss of data that has not been flushed to disk. This is different to what the DS docs are talking about. I'm not sure why they are saying delete the commit log, try asking on their forum http://www.datastax.com/support-forums/
Cheers
-----------------
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com
On 2/05/2012, at 12:02 PM, Roshan wrote:
> Any help regarding this is appreciated.
>
> --
> View this message in context: http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Cassandra-backup-queston-regarding-commitlogs-tp7508823p7518544.html
> Sent from the cassandra-user@incubator.apache.org mailing list archive at Nabble.com.
Re: Cassandra backup queston regarding commitlogs
Posted by Roshan <co...@gmail.com>.
Any help regarding this is appreciated.
--
View this message in context: http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Cassandra-backup-queston-regarding-commitlogs-tp7508823p7518544.html
Sent from the cassandra-user@incubator.apache.org mailing list archive at Nabble.com.
Re: Cassandra backup queston regarding commitlogs
Posted by Roshan <co...@gmail.com>.
http://www.datastax.com/docs/1.0/operations/backup_restore
--
View this message in context: http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Cassandra-backup-queston-regarding-commitlogs-tp7508823p7515679.html
Sent from the cassandra-user@incubator.apache.org mailing list archive at Nabble.com.
Re: Cassandra backup queston regarding commitlogs
Posted by aaron morton <aa...@thelastpickle.com>.
Can you provide a link to that page ?
Cheers
-----------------
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com
On 1/05/2012, at 10:12 AM, Roshan wrote:
> Many Thanks Aaron.
>
> According to the datastax restore documentation, they ask to remove the
> commitlogs before restoring (Clear all files the
> /var/lib/cassandra/commitlog (by default)).
>
> In that case better not to follow this step in a server rash situation.
>
> Thanks
>
> /Roshan
>
> --
> View this message in context: http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Cassandra-backup-queston-regarding-commitlogs-tp7508823p7515217.html
> Sent from the cassandra-user@incubator.apache.org mailing list archive at Nabble.com.
Re: Cassandra backup queston regarding commitlogs
Posted by Roshan <co...@gmail.com>.
Many Thanks Aaron.
According to the datastax restore documentation, they ask to remove the
commitlogs before restoring (Clear all files the
/var/lib/cassandra/commitlog (by default)).
In that case better not to follow this step in a server rash situation.
Thanks
/Roshan
--
View this message in context: http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Cassandra-backup-queston-regarding-commitlogs-tp7508823p7515217.html
Sent from the cassandra-user@incubator.apache.org mailing list archive at Nabble.com.
Re: Cassandra backup queston regarding commitlogs
Posted by aaron morton <aa...@thelastpickle.com>.
When the server starts it reads the SSTables then applies the Commit Logs.
There is nothing you need to do other than leave the commit logs where they are.
Cheers
-----------------
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com
On 30/04/2012, at 6:02 PM, Roshan wrote:
> Hi Aaron
>
> Thanks for the comments. Yes for the durability will keep them in a safe
> place. But such crash situation, how can I restore the data (because those
> are not in a SSTable and only in commit log).
>
> Do I need to replay only that commit log when server starts after crash?
> Will it override the same keys with values?
>
> Appreciate your reply on this.
>
> Kind Regards
>
> /Roshan
>
> --
> View this message in context: http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/deleted-tp7508823p7512499.html
> Sent from the cassandra-user@incubator.apache.org mailing list archive at Nabble.com.
Re: Cassandra backup queston regarding commitlogs
Posted by Roshan <co...@gmail.com>.
Hi Aaron
Thanks for the comments. Yes for the durability will keep them in a safe
place. But such crash situation, how can I restore the data (because those
are not in a SSTable and only in commit log).
Do I need to replay only that commit log when server starts after crash?
Will it override the same keys with values?
Appreciate your reply on this.
Kind Regards
/Roshan
--
View this message in context: http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/deleted-tp7508823p7512499.html
Sent from the cassandra-user@incubator.apache.org mailing list archive at Nabble.com.
Re: Cassandra backup queston regarding commitlogs
Posted by aaron morton <aa...@thelastpickle.com>.
Each mutation is applied to the commit log before being applied to the memtable. On server start the SSTables are read before replaying the commit logs. This is part of the crash only software design and happens for every start.
AFAIk there is no facility to snapshot commit log files as they are closed. The best advice would be to to keep them on a mirror set for durability.
Cheers
-----------------
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com
On 29/04/2012, at 1:04 AM, Roshan wrote:
> Hi
>
> Currently I am taking daily snapshot on my keyspace in production and
> already enable the incremental backups as well.
>
> According to the documentation, the incremental backup option will create an
> hard-link to the backup folder when new sstable is flushed. Snapshot will
> copy all the data/index/etc. files to a new folder.
>
> *Question:*
> What will happen (with enabling the incremental backup) when crash (due to
> any reason) the Cassandra before flushing the data as a SSTable (inserted
> data still in commitlog). In this case how can I backup/restore data?
>
> Do I need to backup the commitlogs as well and and replay during the server
> start to restore the data in commitlog files?
>
> Thanks.
>
>
>
> --
> View this message in context: http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Cassandra-backup-queston-regarding-commitlogs-tp7508823.html
> Sent from the cassandra-user@incubator.apache.org mailing list archive at Nabble.com.