You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@kvrocks.apache.org by "git-hulk (via GitHub)" <gi...@apache.org> on 2023/05/29 15:57:26 UTC
[GitHub] [incubator-kvrocks] git-hulk opened a new issue, #1478: Allow using S3 to backup the Kvrocks DB
git-hulk opened a new issue, #1478:
URL: https://github.com/apache/incubator-kvrocks/issues/1478
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/incubator-kvrocks/issues) and found no similar issues.
### Motivation
Most users demand a backup of the DB dir, but we can only support backup in the local file system. And it may cause trouble if we didn't reserve enough disk space. It would be better if we can put the backup on cloud storage like S3/GCS/...
### Solution
_No response_
### Are you willing to submit a PR?
- [ ] I'm willing to submit a PR!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kvrocks] git-hulk commented on issue #1478: Allow using S3 to backup the Kvrocks DB
Posted by "git-hulk (via GitHub)" <gi...@apache.org>.
git-hulk commented on issue #1478:
URL: https://github.com/apache/incubator-kvrocks/issues/1478#issuecomment-1584244309
@torwig Thanks a lot. For this issue, I am not sure if it's good to compress the db into a single object and then upload it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Allow using S3 to backup the Kvrocks DB [kvrocks]
Posted by "git-hulk (via GitHub)" <gi...@apache.org>.
git-hulk commented on issue #1478:
URL: https://github.com/apache/kvrocks/issues/1478#issuecomment-2029126666
> But yes, the fact that the backup is first generated on the same volume can be problematic (lack of space etc).
Kvrocks allows changing the backup dir via `config set backup-dir`. And it's now using the rocksdb checkpoint as the backup which will use the hard link when copying files. Perhaps you can remove the backup after syncing to S3?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Allow using S3 to backup the Kvrocks DB [kvrocks]
Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
mapleFU commented on issue #1478:
URL: https://github.com/apache/kvrocks/issues/1478#issuecomment-1756847828
Initialize S3/GCS etc would be a bit tricky, maybe opendal C SDK would help: https://github.com/apache/incubator-opendal . It would be also ok for testing in local machine. Other tools in C++ is also welcomed. Since s3 credit config is a bit tricky, I think we'd better use thirdparty library at first.
Also, the dependency would be a bit complex for using object SDK, we'd better make clear what the config would like. You can try to investigate how other system does that:
1. https://tikv.org/docs/6.5/concepts/explore-tikv-features/backup-restore-cn/
2. https://www.cockroachlabs.com/docs/stable/backup
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kvrocks] git-hulk commented on issue #1478: Allow using S3 to backup the Kvrocks DB
Posted by "git-hulk (via GitHub)" <gi...@apache.org>.
git-hulk commented on issue #1478:
URL: https://github.com/apache/incubator-kvrocks/issues/1478#issuecomment-1584263901
🆒 Thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Allow using S3 to backup the Kvrocks DB [kvrocks]
Posted by "PragmaTwice (via GitHub)" <gi...@apache.org>.
PragmaTwice commented on issue #1478:
URL: https://github.com/apache/kvrocks/issues/1478#issuecomment-2653682216
@chrisxu333 Are you working on it? If not we can make it a GSOC project.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Allow using S3 to backup the Kvrocks DB [kvrocks]
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on issue #1478:
URL: https://github.com/apache/kvrocks/issues/1478#issuecomment-2045436910
Hi, I'm Xuanwo from the [OpenDAL](https://github.com/apache/opendal) communiy. I'm watching on development of kvrocks for sometime and find this issue interesting.
As you may know, OpenDAL offers a unified data access layer, empowering users to seamlessly and efficiently retrieve data from diverse storage services. I feel like opendal will be a good fit for kvrocks to implement backup to/from storage services like s3/gcs/azblob/...
Since kvrocks code base is mainly cpp, their are two ways to integrate with opendal:
- Implement a rust module in kvrocks and expose FFI to exsiting code.
- benefits: opendal rust core is mature and adopted by many projects
- shortcome: kvrocks should have rust code inside.
- Integrate opendal-cpp in kvrocks directly.
- benefits: work with cpp natively.
- shortcome: opendal-cpp is still under developement
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Allow using S3 to backup the Kvrocks DB [kvrocks]
Posted by "PragmaTwice (via GitHub)" <gi...@apache.org>.
PragmaTwice commented on issue #1478:
URL: https://github.com/apache/kvrocks/issues/1478#issuecomment-2653685336
Let's make it a GSOC project.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Allow using S3 to backup the Kvrocks DB [kvrocks]
Posted by "chrisxu333 (via GitHub)" <gi...@apache.org>.
chrisxu333 commented on issue #1478:
URL: https://github.com/apache/kvrocks/issues/1478#issuecomment-1756474000
Hi @torwig are you still working on this issue? If not @git-hulk could I take it up?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Allow using S3 to backup the Kvrocks DB [kvrocks]
Posted by "git-hulk (via GitHub)" <gi...@apache.org>.
git-hulk commented on issue #1478:
URL: https://github.com/apache/kvrocks/issues/1478#issuecomment-1756851678
To be honest, I didn't think clearly about whether this feature should be put inside Kvrocks. Perhaps implementing a new dedicated tool for the backup like ClickHouse is a good idea.
Refer: https://github.com/Altinity/clickhouse-backup
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Allow using S3 to backup the Kvrocks DB [kvrocks]
Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
mapleFU commented on issue #1478:
URL: https://github.com/apache/kvrocks/issues/1478#issuecomment-2102258591
@Xuanwo Here I think the performance is not the critical reason and we may not enable some advance feature about threading, I think opendal as a backend of RocksDB Env would be a goodway for solving both this and backup to hdfs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Allow using S3 to backup the Kvrocks DB [kvrocks]
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on issue #1478:
URL: https://github.com/apache/kvrocks/issues/1478#issuecomment-2102278912
> opendal as a backend of RocksDB Env
It looks like a good idea. I don't have much understadning of RocksDB Env so I don't know if it's possible with a simple wrapper.
My friend @leiysky told me that rocksdb env requires `append` support which is not widely supported by object storage services (at least s3 doesn't). And even for services that support append, It might be not good for append many small chunks. This could be an issue.
Note: OpenDAL itself does support `append` but s3 doesn't.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Allow using S3 to backup the Kvrocks DB [kvrocks]
Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
mapleFU commented on issue #1478:
URL: https://github.com/apache/kvrocks/issues/1478#issuecomment-1756853491
🤔 ClickHouse can read from remote S3, so I think it's able to dosomething here.
However, TiKV only supports a br here. (See: https://tikv.org/docs/6.5/concepts/explore-tikv-features/backup-restore-cn/ ). Maybe we can considering using the sameway. It can also not bring any size amplify to our binary and hide the risk of unmature implemention.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kvrocks] torwig commented on issue #1478: Allow using S3 to backup the Kvrocks DB
Posted by "torwig (via GitHub)" <gi...@apache.org>.
torwig commented on issue #1478:
URL: https://github.com/apache/incubator-kvrocks/issues/1478#issuecomment-1584253812
@git-hulk Thank you for your tip. I'm going to think about the whole process and suggest something like "high-level design" and "possible implementation(s)" before actually starting implementing so we can discuss all key things.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Allow using S3 to backup the Kvrocks DB [kvrocks]
Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
mapleFU commented on issue #1478:
URL: https://github.com/apache/kvrocks/issues/1478#issuecomment-2102313172
After some discussion, maybe design some new syntax and using another thread / process to upload Backup in Local FileSystem to HDFS/S3 is also a way. This avoid the complex logic of intereact with `rocksdb::Env`, and could be done in a separate way
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Allow using S3 to backup the Kvrocks DB [kvrocks]
Posted by "torwig (via GitHub)" <gi...@apache.org>.
torwig commented on issue #1478:
URL: https://github.com/apache/kvrocks/issues/1478#issuecomment-1756828299
@chrisxu333 Currently, I can't dedicate my time to this issue. If you wish to run it, @git-hulk will reassign it to you.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Allow using S3 to backup the Kvrocks DB [kvrocks]
Posted by "git-hulk (via GitHub)" <gi...@apache.org>.
git-hulk commented on issue #1478:
URL: https://github.com/apache/kvrocks/issues/1478#issuecomment-1756854878
@mapleFU Thanks for your great references!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Allow using S3 to backup the Kvrocks DB [kvrocks]
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on issue #1478:
URL: https://github.com/apache/kvrocks/issues/1478#issuecomment-2653910464
Let's make it a GSOC *(Great Scalable OpenDAL-powered Cloud)* project.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Allow using S3 to backup the Kvrocks DB [kvrocks]
Posted by "PragmaTwice (via GitHub)" <gi...@apache.org>.
PragmaTwice commented on issue #1478:
URL: https://github.com/apache/kvrocks/issues/1478#issuecomment-2658625634
Hi @LindaSummer, for GSoC projects, we need to follow the official process and we cannot determine the candidates outside the GSoC official.
You can refer to the official page (https://summerofcode.withgoogle.com/programs/2025-ao) if you want to sign up : )
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Allow using S3 to backup the Kvrocks DB [kvrocks]
Posted by "LindaSummer (via GitHub)" <gi...@apache.org>.
LindaSummer commented on issue #1478:
URL: https://github.com/apache/kvrocks/issues/1478#issuecomment-2658601069
Hi Team,
I'm experienced in AWS and want to dive deep in rocksdb and S3. 😁
Could I take this gsoc task and make some contribution?
Best Regards,
Edward
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Allow using S3 to backup the Kvrocks DB [kvrocks]
Posted by "vlourme (via GitHub)" <gi...@apache.org>.
vlourme commented on issue #1478:
URL: https://github.com/apache/kvrocks/issues/1478#issuecomment-3939622114
Hi! I started using kvrocks recently and I really needed automated backups (and restore to local dev for debugging) so I made a tool for that: https://github.com/SneakersAPI/kvrocks-backup
It only supports S3 for now. No incremental backup and no compression.
Also I have some concerns about `restore` process, it basically shutdown the server and replace `db` content from an S3 backup, it works but doesn't feel safe to me. I would recommend using my tool for backup only and restore on safe environment, not production.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Allow using S3 to backup the Kvrocks DB [kvrocks]
Posted by "LindaSummer (via GitHub)" <gi...@apache.org>.
LindaSummer commented on issue #1478:
URL: https://github.com/apache/kvrocks/issues/1478#issuecomment-2658693324
Hi @PragmaTwice ,
Thanks for your warm reminder. 😊
Got it! I will follow the gsoc's official process and try my best.
Best Regards,
Edward
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Allow using S3 to backup the Kvrocks DB [kvrocks]
Posted by "kinoute (via GitHub)" <gi...@apache.org>.
kinoute commented on issue #1478:
URL: https://github.com/apache/kvrocks/issues/1478#issuecomment-2028838344
Encryption of the backup file(s) will be nice too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Allow using S3 to backup the Kvrocks DB [kvrocks]
Posted by "asad-awadia (via GitHub)" <gi...@apache.org>.
asad-awadia commented on issue #1478:
URL: https://github.com/apache/kvrocks/issues/1478#issuecomment-1987580609
>if it's good to compress the db into a single object and then upload it.
Why not?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kvrocks] torwig commented on issue #1478: Allow using S3 to backup the Kvrocks DB
Posted by "torwig (via GitHub)" <gi...@apache.org>.
torwig commented on issue #1478:
URL: https://github.com/apache/incubator-kvrocks/issues/1478#issuecomment-1584238293
@git-hulk Let me try to implement this feature.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org