You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@kvrocks.apache.org by "git-hulk (via GitHub)" <gi...@apache.org> on 2023/05/29 15:57:26 UTC
[GitHub] [incubator-kvrocks] git-hulk opened a new issue, #1478: Allow using S3 to backup the Kvrocks DB
git-hulk opened a new issue, #1478:
URL: https://github.com/apache/incubator-kvrocks/issues/1478
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/incubator-kvrocks/issues) and found no similar issues.
### Motivation
Most users demand a backup of the DB dir, but we can only support backup in the local file system. And it may cause trouble if we didn't reserve enough disk space. It would be better if we can put the backup on cloud storage like S3/GCS/...
### Solution
_No response_
### Are you willing to submit a PR?
- [ ] I'm willing to submit a PR!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kvrocks] git-hulk commented on issue #1478: Allow using S3 to backup the Kvrocks DB
Posted by "git-hulk (via GitHub)" <gi...@apache.org>.
git-hulk commented on issue #1478:
URL: https://github.com/apache/incubator-kvrocks/issues/1478#issuecomment-1584263901
🆒 Thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Allow using S3 to backup the Kvrocks DB [kvrocks]
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on issue #1478:
URL: https://github.com/apache/kvrocks/issues/1478#issuecomment-2045436910
Hi, I'm Xuanwo from the [OpenDAL](https://github.com/apache/opendal) communiy. I'm watching on development of kvrocks for sometime and find this issue interesting.
As you may know, OpenDAL offers a unified data access layer, empowering users to seamlessly and efficiently retrieve data from diverse storage services. I feel like opendal will be a good fit for kvrocks to implement backup to/from storage services like s3/gcs/azblob/...
Since kvrocks code base is mainly cpp, their are two ways to integrate with opendal:
- Implement a rust module in kvrocks and expose FFI to exsiting code.
- benefits: opendal rust core is mature and adopted by many projects
- shortcome: kvrocks should have rust code inside.
- Integrate opendal-cpp in kvrocks directly.
- benefits: work with cpp natively.
- shortcome: opendal-cpp is still under developement
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kvrocks] git-hulk commented on issue #1478: Allow using S3 to backup the Kvrocks DB
Posted by "git-hulk (via GitHub)" <gi...@apache.org>.
git-hulk commented on issue #1478:
URL: https://github.com/apache/incubator-kvrocks/issues/1478#issuecomment-1584244309
@torwig Thanks a lot. For this issue, I am not sure if it's good to compress the db into a single object and then upload it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Allow using S3 to backup the Kvrocks DB [kvrocks]
Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
mapleFU commented on issue #1478:
URL: https://github.com/apache/kvrocks/issues/1478#issuecomment-1756847828
Initialize S3/GCS etc would be a bit tricky, maybe opendal C SDK would help: https://github.com/apache/incubator-opendal . It would be also ok for testing in local machine. Other tools in C++ is also welcomed. Since s3 credit config is a bit tricky, I think we'd better use thirdparty library at first.
Also, the dependency would be a bit complex for using object SDK, we'd better make clear what the config would like. You can try to investigate how other system does that:
1. https://tikv.org/docs/6.5/concepts/explore-tikv-features/backup-restore-cn/
2. https://www.cockroachlabs.com/docs/stable/backup
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Allow using S3 to backup the Kvrocks DB [kvrocks]
Posted by "git-hulk (via GitHub)" <gi...@apache.org>.
git-hulk commented on issue #1478:
URL: https://github.com/apache/kvrocks/issues/1478#issuecomment-2029126666
> But yes, the fact that the backup is first generated on the same volume can be problematic (lack of space etc).
Kvrocks allows changing the backup dir via `config set backup-dir`. And it's now using the rocksdb checkpoint as the backup which will use the hard link when copying files. Perhaps you can remove the backup after syncing to S3?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Allow using S3 to backup the Kvrocks DB [kvrocks]
Posted by "git-hulk (via GitHub)" <gi...@apache.org>.
git-hulk commented on issue #1478:
URL: https://github.com/apache/kvrocks/issues/1478#issuecomment-1756851678
To be honest, I didn't think clearly about whether this feature should be put inside Kvrocks. Perhaps implementing a new dedicated tool for the backup like ClickHouse is a good idea.
Refer: https://github.com/Altinity/clickhouse-backup
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Allow using S3 to backup the Kvrocks DB [kvrocks]
Posted by "chrisxu333 (via GitHub)" <gi...@apache.org>.
chrisxu333 commented on issue #1478:
URL: https://github.com/apache/kvrocks/issues/1478#issuecomment-1756474000
Hi @torwig are you still working on this issue? If not @git-hulk could I take it up?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kvrocks] torwig commented on issue #1478: Allow using S3 to backup the Kvrocks DB
Posted by "torwig (via GitHub)" <gi...@apache.org>.
torwig commented on issue #1478:
URL: https://github.com/apache/incubator-kvrocks/issues/1478#issuecomment-1584253812
@git-hulk Thank you for your tip. I'm going to think about the whole process and suggest something like "high-level design" and "possible implementation(s)" before actually starting implementing so we can discuss all key things.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Allow using S3 to backup the Kvrocks DB [kvrocks]
Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
mapleFU commented on issue #1478:
URL: https://github.com/apache/kvrocks/issues/1478#issuecomment-1756853491
🤔 ClickHouse can read from remote S3, so I think it's able to dosomething here.
However, TiKV only supports a br here. (See: https://tikv.org/docs/6.5/concepts/explore-tikv-features/backup-restore-cn/ ). Maybe we can considering using the sameway. It can also not bring any size amplify to our binary and hide the risk of unmature implemention.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Allow using S3 to backup the Kvrocks DB [kvrocks]
Posted by "torwig (via GitHub)" <gi...@apache.org>.
torwig commented on issue #1478:
URL: https://github.com/apache/kvrocks/issues/1478#issuecomment-1756828299
@chrisxu333 Currently, I can't dedicate my time to this issue. If you wish to run it, @git-hulk will reassign it to you.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Allow using S3 to backup the Kvrocks DB [kvrocks]
Posted by "git-hulk (via GitHub)" <gi...@apache.org>.
git-hulk commented on issue #1478:
URL: https://github.com/apache/kvrocks/issues/1478#issuecomment-1756854878
@mapleFU Thanks for your great references!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kvrocks] torwig commented on issue #1478: Allow using S3 to backup the Kvrocks DB
Posted by "torwig (via GitHub)" <gi...@apache.org>.
torwig commented on issue #1478:
URL: https://github.com/apache/incubator-kvrocks/issues/1478#issuecomment-1584238293
@git-hulk Let me try to implement this feature.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Allow using S3 to backup the Kvrocks DB [kvrocks]
Posted by "asad-awadia (via GitHub)" <gi...@apache.org>.
asad-awadia commented on issue #1478:
URL: https://github.com/apache/kvrocks/issues/1478#issuecomment-1987580609
>if it's good to compress the db into a single object and then upload it.
Why not?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Allow using S3 to backup the Kvrocks DB [kvrocks]
Posted by "kinoute (via GitHub)" <gi...@apache.org>.
kinoute commented on issue #1478:
URL: https://github.com/apache/kvrocks/issues/1478#issuecomment-2028838344
Encryption of the backup file(s) will be nice too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org