You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@kvrocks.apache.org by GitBox <gi...@apache.org> on 2023/03/26 13:06:54 UTC
[GitHub] [incubator-kvrocks] GoodManWEN created a discussion: About backup and recovery
GitHub user GoodManWEN created a discussion: About backup and recovery
I would like to ask about the recommended backup and recovery methods. According to the information in [](https://kvrocks.apache.org/docs/backup/) , it is recommended to use the "bgsave" command to export the full database, but it does not provide a method for importing. Does this mean that the "bgsave" command will generate a "backup" folder, and copying the contents of this folder to a third-party location represents a backup? To restore data, means overwrite the contents of the "db" folder with the contents of the "backup" folder and then restart the kvrocks service. Not sure if I understand correctly.
GitHub link: https://github.com/apache/incubator-kvrocks/discussions/1357
----
This is an automatically sent email for issues@kvrocks.apache.org.
To unsubscribe, please send an email to: issues-unsubscribe@kvrocks.apache.org
Re: [D] About backup and recovery [kvrocks]
Posted by "ltagliamonte-dd (via GitHub)" <gi...@apache.org>.
GitHub user ltagliamonte-dd edited a comment on the discussion: About backup and recovery
@git-hulk thanks for the insight, I'm Interested as well in this, what happens in terms of performances if we set `rocksdb.wal_size_limit_mb` to 1GB of example?
My use case is: load parquet files in rocks db (~100GB), compact the data, make a snapshot, upload it on s3.
GitHub link: https://github.com/apache/kvrocks/discussions/1357#discussioncomment-9950454
----
This is an automatically sent email for issues@kvrocks.apache.org.
To unsubscribe, please send an email to: issues-unsubscribe@kvrocks.apache.org
Re: [D] About backup and recovery [kvrocks]
Posted by "ltagliamonte-dd (via GitHub)" <gi...@apache.org>.
GitHub user ltagliamonte-dd added a comment to the discussion: About backup and recovery
Hi there!
I'm trying to understand better how the incremental backup works, can i please get an example on how to implement it?
when `BGSAVE` a full backup is created, how can i later create only the incremental part?
GitHub link: https://github.com/apache/kvrocks/discussions/1357#discussioncomment-9347080
----
This is an automatically sent email for issues@kvrocks.apache.org.
To unsubscribe, please send an email to: issues-unsubscribe@kvrocks.apache.org
Re: [D] About backup and recovery [kvrocks]
Posted by "ltagliamonte-dd (via GitHub)" <gi...@apache.org>.
GitHub user ltagliamonte-dd edited a comment on the discussion: About backup and recovery
@git-hulk thanks for the insight, I'm Interested as well in this, what happens in terms of performances if we set `rocksdb.wal_size_limit_mb` to 1GB of example?
I wonder if a wal size this low will slow down my use case:
I don't do replication, just single instance kvrocks, I load parquet files in kvrocks (~100GB), compact the data, make a snapshot, upload it on s3.
GitHub link: https://github.com/apache/kvrocks/discussions/1357#discussioncomment-9950454
----
This is an automatically sent email for issues@kvrocks.apache.org.
To unsubscribe, please send an email to: issues-unsubscribe@kvrocks.apache.org
Re: [D] About backup and recovery [kvrocks]
Posted by "kinoute (via GitHub)" <gi...@apache.org>.
GitHub user kinoute added a comment to the discussion: About backup and recovery
@git-hulk Thanks a lot, I will reduce this. Does it have impact on performance if I reduce this number, to lets say, a few Gbs? My rocksdb.write_buffer_size is set to the default, 64Mb (I'm reading the kvrocks.conf comments at the moment).
GitHub link: https://github.com/apache/kvrocks/discussions/1357#discussioncomment-9949788
----
This is an automatically sent email for issues@kvrocks.apache.org.
To unsubscribe, please send an email to: issues-unsubscribe@kvrocks.apache.org
[GitHub] [incubator-kvrocks] git-hulk added a comment to the discussion: About backup and recovery
Posted by GitBox <gi...@apache.org>.
GitHub user git-hulk added a comment to the discussion: About backup and recovery
> Does this mean that the "bgsave" command will generate a "backup" folder, and copying the contents of this folder to a third-party location represents a backup? To restore data, means overwrite the contents of the "db" folder with the contents of the "backup" folder and then restart the kvrocks service. Not sure if I understand correctly.
That's right that the "bgsave" command would only produce a snapshot in the backup dir. And you need to copy it to the db dir and restart the Kvrocks. It's trivial and inconvenient, but we plan to improve this by supporting the restore command.
> If I want to perform incremental backup, I need to compare the updated parts in all files myself, which seems to require reading all files at least once (if using methods like calculating the MD5 of files), I am not sure if this is the recommended way
RocksDB allows incremental backup indeed, but we're using checkpoint now. Maybe we can use the backup mechanism when creating the backup.
> Additionally, I would like to ask if the "backup" folder can be set to a different path. If it is in the same directory as the "db" folder, it seems to require reserving twice the space on the physical hardware for backup.
Yes, we recommend storing the backup in different disks from the DB.
GitHub link: https://github.com/apache/incubator-kvrocks/discussions/1357#discussioncomment-5438741
----
This is an automatically sent email for issues@kvrocks.apache.org.
To unsubscribe, please send an email to: issues-unsubscribe@kvrocks.apache.org
Re: [D] About backup and recovery [kvrocks]
Posted by "git-hulk (via GitHub)" <gi...@apache.org>.
GitHub user git-hulk added a comment to the discussion: About backup and recovery
@ltagliamonte-dd It wouldn't affect the performance except during the replication time. So you can reduce it to 0 if you didn't use the replication.
GitHub link: https://github.com/apache/kvrocks/discussions/1357#discussioncomment-9953816
----
This is an automatically sent email for issues@kvrocks.apache.org.
To unsubscribe, please send an email to: issues-unsubscribe@kvrocks.apache.org
Re: [D] About backup and recovery [kvrocks]
Posted by "ltagliamonte-dd (via GitHub)" <gi...@apache.org>.
GitHub user ltagliamonte-dd added a comment to the discussion: About backup and recovery
@git-hulk thanks for the insight, I'm Interested as well in this, what happens in terms of performances if we set `rocksdb.wal_size_limit_mb` to 1GB of example?
GitHub link: https://github.com/apache/kvrocks/discussions/1357#discussioncomment-9950454
----
This is an automatically sent email for issues@kvrocks.apache.org.
To unsubscribe, please send an email to: issues-unsubscribe@kvrocks.apache.org
Re: [D] About backup and recovery [kvrocks]
Posted by "git-hulk (via GitHub)" <gi...@apache.org>.
GitHub user git-hulk added a comment to the discussion: About backup and recovery
@kinoute It only affects the possibility of the full sync since we use the `rocksdb.wal_size_limit_mb` which is similiar to the replication buffer. That said if it's bigger then the bigger lag was allowed between the master and replicas. And the write buffer is generally depended on size of num_of_write_buffer * write_buffer_size.
GitHub link: https://github.com/apache/kvrocks/discussions/1357#discussioncomment-9953799
----
This is an automatically sent email for issues@kvrocks.apache.org.
To unsubscribe, please send an email to: issues-unsubscribe@kvrocks.apache.org
Re: [D] About backup and recovery [kvrocks]
Posted by "ltagliamonte-dd (via GitHub)" <gi...@apache.org>.
GitHub user ltagliamonte-dd edited a comment on the discussion: About backup and recovery
Hi there!
I'm trying to understand better how the incremental backup works, can i please get an example on how to implement it?
when `BGSAVE` a full backup is created, how can i later create only the incremental part?
after `BGSAVE` I get:
```
ls -1 backup/
000015.sst
000016.log
000020.sst
CURRENT
MANIFEST-000017
OPTIONS-000019
```
add more data and run `BGSAVE` again i get:
```
ls -1 backup
000015.sst
000020.sst
000022.sst
CURRENT
MANIFEST-000017
OPTIONS-000019
```
`000016.log` is gone and `000022.sst` got created how am i supposed to generate a "diff"?
is there a way to force compaction and generate all sst tables?
I guess I have to execute `COMPACT` and then `BGSAVE`?
GitHub link: https://github.com/apache/kvrocks/discussions/1357#discussioncomment-9347080
----
This is an automatically sent email for issues@kvrocks.apache.org.
To unsubscribe, please send an email to: issues-unsubscribe@kvrocks.apache.org
Re: [D] About backup and recovery [kvrocks]
Posted by "ltagliamonte-dd (via GitHub)" <gi...@apache.org>.
GitHub user ltagliamonte-dd edited a comment on the discussion: About backup and recovery
Hi there!
I'm trying to understand better how the incremental backup works, can i please get an example on how to implement it?
when `BGSAVE` a full backup is created, how can i later create only the incremental part?
after `BGSAVE` I get:
```
ls -1 backup/
000015.sst
000016.log
000020.sst
CURRENT
MANIFEST-000017
OPTIONS-000019
```
add more data and run `BGSAVE` again i get:
```
ls -1 backup
000015.sst
000020.sst
000022.sst
CURRENT
MANIFEST-000017
OPTIONS-000019
```
`000016.log` is gone and `000022.sst` got created how am i supposed to generate a "diff"?
GitHub link: https://github.com/apache/kvrocks/discussions/1357#discussioncomment-9347080
----
This is an automatically sent email for issues@kvrocks.apache.org.
To unsubscribe, please send an email to: issues-unsubscribe@kvrocks.apache.org
[GitHub] [incubator-kvrocks] GoodManWEN added a comment to the discussion: About backup and recovery
Posted by GitBox <gi...@apache.org>.
GitHub user GoodManWEN added a comment to the discussion: About backup and recovery
Thank you for your reply. I did not find the corresponding option in the `kvrocks.conf` configuration file regarding changing the default bgsave output directory. Is it possible to modify it? Or do I need to transfer it to a third-party device using some techniques like symbolic links?
GitHub link: https://github.com/apache/incubator-kvrocks/discussions/1357#discussioncomment-5447615
----
This is an automatically sent email for issues@kvrocks.apache.org.
To unsubscribe, please send an email to: issues-unsubscribe@kvrocks.apache.org
[GitHub] [incubator-kvrocks] GoodManWEN added a comment to the discussion: About backup and recovery
Posted by GitBox <gi...@apache.org>.
GitHub user GoodManWEN added a comment to the discussion: About backup and recovery
Upd: I noticed that the wiki describes the checkpoint backup method using hard links as not taking up additional storage space and the backups should be deleted in a timely manner. Therefore, I don’t think it’s necessary to change the default output path.
GitHub link: https://github.com/apache/incubator-kvrocks/discussions/1357#discussioncomment-5447881
----
This is an automatically sent email for issues@kvrocks.apache.org.
To unsubscribe, please send an email to: issues-unsubscribe@kvrocks.apache.org
Re: [D] About backup and recovery [kvrocks]
Posted by "ltagliamonte-dd (via GitHub)" <gi...@apache.org>.
GitHub user ltagliamonte-dd edited a comment on the discussion: About backup and recovery
Hi there!
I'm trying to understand better how the incremental backup works, can i please get an example on how to implement it?
when `BGSAVE` a full backup is created, how can i later create only the incremental part?
after `BGSAVE` I get:
```
ls -1 backup/
000015.sst
000016.log
000020.sst
CURRENT
MANIFEST-000017
OPTIONS-000019
```
add more data and run `BGSAVE` again i get:
```
ls -1 backup
000015.sst
000020.sst
000022.sst
CURRENT
MANIFEST-000017
OPTIONS-000019
```
`000016.log` is gone and `000022.sst` got created how am i supposed to generate a "diff"?
is there a way to force compaction and generate all sst tables?
GitHub link: https://github.com/apache/kvrocks/discussions/1357#discussioncomment-9347080
----
This is an automatically sent email for issues@kvrocks.apache.org.
To unsubscribe, please send an email to: issues-unsubscribe@kvrocks.apache.org
Re: [D] About backup and recovery [kvrocks]
Posted by "ltagliamonte-dd (via GitHub)" <gi...@apache.org>.
GitHub user ltagliamonte-dd added a comment to the discussion: About backup and recovery
After few tests my understanding is that:
- both `BGSAVE` and `COMPACT` are async and that I can use `INFO rocksdb` to poll and check.
However when i execute `COMPACT` it generates every time a new set of sstables so it seems hard to create any diff policy between the prev uploaded sstables and the new one.
First COMPACT + BGSAVE:
```
ll backup
total 512
-rw-r--r-- 2 luigi.tagliamonte staff 182123 May 7 17:04 000040.sst
-rw-r--r-- 2 luigi.tagliamonte staff 10994 May 7 17:04 000042.sst
-rw-r--r-- 1 luigi.tagliamonte staff 16 May 7 17:05 CURRENT
-rw-r--r-- 1 luigi.tagliamonte staff 6636 May 7 17:05 MANIFEST-000011
-rw-r--r-- 2 luigi.tagliamonte staff 36036 May 7 17:00 OPTIONS-000013
ll db
total 9384
-rw-r--r-- 1 luigi.tagliamonte staff 0 May 7 17:04 000029.log
-rw-r--r-- 2 luigi.tagliamonte staff 182123 May 7 17:04 000040.sst
-rw-r--r-- 2 luigi.tagliamonte staff 10994 May 7 17:04 000042.sst
-rw-r--r-- 1 luigi.tagliamonte staff 16 May 7 17:00 CURRENT
-rw-r--r-- 1 luigi.tagliamonte staff 36 May 7 17:00 IDENTITY
-rw-r--r-- 1 luigi.tagliamonte staff 0 May 7 17:00 LOCK
-rw-r--r-- 1 luigi.tagliamonte staff 203660 May 7 17:05 LOG
-rw-r--r-- 1 luigi.tagliamonte staff 89766 May 7 17:00 LOG.old.1715126427310129
-rw-r--r-- 1 luigi.tagliamonte staff 6636 May 7 17:04 MANIFEST-000011
-rw-r--r-- 1 luigi.tagliamonte staff 35454 May 7 17:00 OPTIONS-000009
-rw-r--r-- 2 luigi.tagliamonte staff 36036 May 7 17:00 OPTIONS-000013
drwxr-xr-x 4 luigi.tagliamonte staff 128 May 7 17:04 archive
```
second COMPACT + BGSAVE:
```
ll backup
total 528
-rw-r--r-- 2 luigi.tagliamonte staff 182123 May 7 17:06 000052.sst
-rw-r--r-- 2 luigi.tagliamonte staff 10994 May 7 17:06 000054.sst
-rw-r--r-- 1 luigi.tagliamonte staff 16 May 7 17:06 CURRENT
-rw-r--r-- 1 luigi.tagliamonte staff 9305 May 7 17:06 MANIFEST-000011
-rw-r--r-- 2 luigi.tagliamonte staff 36036 May 7 17:00 OPTIONS-000013
ll db
total 9520
-rw-r--r-- 1 luigi.tagliamonte staff 0 May 7 17:04 000029.log
-rw-r--r-- 2 luigi.tagliamonte staff 182123 May 7 17:06 000052.sst
-rw-r--r-- 2 luigi.tagliamonte staff 10994 May 7 17:06 000054.sst
-rw-r--r-- 1 luigi.tagliamonte staff 16 May 7 17:00 CURRENT
-rw-r--r-- 1 luigi.tagliamonte staff 36 May 7 17:00 IDENTITY
-rw-r--r-- 1 luigi.tagliamonte staff 0 May 7 17:00 LOCK
-rw-r--r-- 1 luigi.tagliamonte staff 253768 May 7 17:06 LOG
-rw-r--r-- 1 luigi.tagliamonte staff 89766 May 7 17:00 LOG.old.1715126427310129
-rw-r--r-- 1 luigi.tagliamonte staff 9305 May 7 17:06 MANIFEST-000011
-rw-r--r-- 1 luigi.tagliamonte staff 35454 May 7 17:00 OPTIONS-000009
-rw-r--r-- 2 luigi.tagliamonte staff 36036 May 7 17:00 OPTIONS-000013
drwxr-xr-x 4 luigi.tagliamonte staff 128 May 7 17:04 archive
```
Am I missing something?
GitHub link: https://github.com/apache/kvrocks/discussions/1357#discussioncomment-9348183
----
This is an automatically sent email for issues@kvrocks.apache.org.
To unsubscribe, please send an email to: issues-unsubscribe@kvrocks.apache.org
Re: [D] About backup and recovery [kvrocks]
Posted by "ltagliamonte-dd (via GitHub)" <gi...@apache.org>.
GitHub user ltagliamonte-dd edited a comment on the discussion: About backup and recovery
@git-hulk thanks for the insight, I'm Interested as well in this, what happens in terms of performances if we set `rocksdb.wal_size_limit_mb` to 1GB of example?
Bet the a wal size this low will slow down my use case:
I don't do replication, just single instance kvrocks, I load parquet files in kvrocks (~100GB), compact the data, make a snapshot, upload it on s3.
GitHub link: https://github.com/apache/kvrocks/discussions/1357#discussioncomment-9950454
----
This is an automatically sent email for issues@kvrocks.apache.org.
To unsubscribe, please send an email to: issues-unsubscribe@kvrocks.apache.org
Re: [D] About backup and recovery [kvrocks]
Posted by "ltagliamonte-dd (via GitHub)" <gi...@apache.org>.
GitHub user ltagliamonte-dd edited a comment on the discussion: About backup and recovery
@git-hulk thanks for the insight, I'm Interested as well in this, what happens in terms of performances if we set `rocksdb.wal_size_limit_mb` to 1GB of example?
Bet the a wal size this low will slow down my use case:
load parquet files in rocks db (~100GB), compact the data, make a snapshot, upload it on s3.
GitHub link: https://github.com/apache/kvrocks/discussions/1357#discussioncomment-9950454
----
This is an automatically sent email for issues@kvrocks.apache.org.
To unsubscribe, please send an email to: issues-unsubscribe@kvrocks.apache.org
Re: [D] About backup and recovery [kvrocks]
Posted by "kinoute (via GitHub)" <gi...@apache.org>.
GitHub user kinoute added a comment to the discussion: About backup and recovery
https://github.com/apache/kvrocks/issues/2365 might be related I don't know
GitHub link: https://github.com/apache/kvrocks/discussions/1357#discussioncomment-9949687
----
This is an automatically sent email for issues@kvrocks.apache.org.
To unsubscribe, please send an email to: issues-unsubscribe@kvrocks.apache.org
Re: [D] About backup and recovery [kvrocks]
Posted by "kinoute (via GitHub)" <gi...@apache.org>.
GitHub user kinoute added a comment to the discussion: About backup and recovery
Sorry to hijack this discussion but we encountered a weird behavior today related to the `archive` folder. We detected that our disk usage was 100%. Indeed, in a matter of hours, a lot of log files were put in the `archive` folder, more than 7Gb (our disk is 10Gb).

What could be the cause of this and how to prevent it? I deleted the archive folder, hopefully I didn't lose any data.
Thanks in advance
GitHub link: https://github.com/apache/kvrocks/discussions/1357#discussioncomment-9949551
----
This is an automatically sent email for issues@kvrocks.apache.org.
To unsubscribe, please send an email to: issues-unsubscribe@kvrocks.apache.org
Re: [D] About backup and recovery [kvrocks]
Posted by "git-hulk (via GitHub)" <gi...@apache.org>.
GitHub user git-hulk edited a comment on the discussion: About backup and recovery
@kinoute It’s fine to delete the archive dir and won’t lost any data. And you can reduce rocksdb.wal_size_limit_mb if you want.
GitHub link: https://github.com/apache/kvrocks/discussions/1357#discussioncomment-9949692
----
This is an automatically sent email for issues@kvrocks.apache.org.
To unsubscribe, please send an email to: issues-unsubscribe@kvrocks.apache.org
Re: [D] About backup and recovery [kvrocks]
Posted by "git-hulk (via GitHub)" <gi...@apache.org>.
GitHub user git-hulk added a comment to the discussion: About backup and recovery
@kinoute It’s fine to delete the archive dir and won’t lost any data.
GitHub link: https://github.com/apache/kvrocks/discussions/1357#discussioncomment-9949692
----
This is an automatically sent email for issues@kvrocks.apache.org.
To unsubscribe, please send an email to: issues-unsubscribe@kvrocks.apache.org
Re: [D] About backup and recovery [kvrocks]
Posted by "git-hulk (via GitHub)" <gi...@apache.org>.
GitHub user git-hulk added a comment to the discussion: About backup and recovery
@ltagliamonte-dd Yes, you're right. It only supports generating the full DB for now, so you can keep the latest one or two.
GitHub link: https://github.com/apache/kvrocks/discussions/1357#discussioncomment-9348498
----
This is an automatically sent email for issues@kvrocks.apache.org.
To unsubscribe, please send an email to: issues-unsubscribe@kvrocks.apache.org
Re: [D] About backup and recovery [kvrocks]
Posted by "ltagliamonte-dd (via GitHub)" <gi...@apache.org>.
GitHub user ltagliamonte-dd edited a comment on the discussion: About backup and recovery
Hi there!
I'm trying to understand better how the incremental backup works, can i please get an example on how to implement it?
when `BGSAVE` a full backup is created, how can i later create only the incremental part?
after `BGSAVE` I get:
```
ls -1 backup/
000015.sst
000016.log
000020.sst
CURRENT
MANIFEST-000017
OPTIONS-000019
```
add more data and run `BGSAVE` again i get:
```
ls -1 backup
000015.sst
000020.sst
000022.sst
CURRENT
MANIFEST-000017
OPTIONS-000019
```
`000016.log` is gone and `000022.sst` got created how am i supposed to generate a "diff"?
is there a way to force compaction and generate all sst tables?
I guess I have to execute `COMPACT` and then `BGSAVE`?
is `COMPACT` sync or async? how can i monitor the completion?
GitHub link: https://github.com/apache/kvrocks/discussions/1357#discussioncomment-9347080
----
This is an automatically sent email for issues@kvrocks.apache.org.
To unsubscribe, please send an email to: issues-unsubscribe@kvrocks.apache.org
[GitHub] [incubator-kvrocks] GoodManWEN edited a discussion: About backup and recovery
Posted by GitBox <gi...@apache.org>.
GitHub user GoodManWEN edited a discussion: About backup and recovery
I would like to ask about the recommended backup and recovery methods. According to the information in [](https://kvrocks.apache.org/docs/backup/) , it is recommended to use the "bgsave" command to export the full database, but it does not provide a method for importing. Does this mean that the "bgsave" command will generate a "backup" folder, and copying the contents of this folder to a third-party location represents a backup? To restore data, means overwrite the contents of the "db" folder with the contents of the "backup" folder and then restart the kvrocks service. Not sure if I understand correctly.
If I want to perform incremental backup, I need to compare the updated parts in all files myself, which seems to require reading all files at least once (if using methods like calculating the MD5 of files), I am not sure if this is the recommended way
Additionally, I would like to ask if the "backup" folder can be set to a different path. If it is in the same directory as the "db" folder, it seems to require reserving twice the space on the physical hardware for backup.
GitHub link: https://github.com/apache/incubator-kvrocks/discussions/1357
----
This is an automatically sent email for issues@kvrocks.apache.org.
To unsubscribe, please send an email to: issues-unsubscribe@kvrocks.apache.org