You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@kvrocks.apache.org by "caipengbo (via GitHub)" <gi...@apache.org> on 2023/08/03 02:10:05 UTC

[GitHub] [kvrocks] caipengbo added a comment to the discussion: Some talk about bulk load feature

GitHub user caipengbo added a comment to the discussion: Some talk about bulk load feature

> 1. Which way should we create the SST file?

The amount of imported data is usually very large, and it may be an inefficient job to use kvrocks to do this. I think we should provide external tools or big data programs to directly generate the final Key-Value data.

> 2. What fields should our `exchange format` have?

For importing information, I think you can use the command like: `submit bulkload-scheme scheme1 file file1_local_path checksum xxxxxxx` (the name of the command is just an example).
This command can currently pass a local path, in the future we can pass the address, asking kvrocks to download these files asynchronously, and we can provide display scheme, delete scheme, etc. 
In my opinion, this approach is more consistent with the use of redis protocol and easier to extend.

> 3. during the process of ingesting SST files, there is a step that will block the write operations of RocksDB, and we need to evaluate how long this time will be.**

The common use cases I can think of for bulk loading are cold starts, migrating to kvrocks from other databases, or periodically ingesting data. Blocking writes should be an acceptable behavior for the user. In general, ingesting data should be fast(I think tens of seconds is enough for big data), and what's slower is generating and downloading the data.


GitHub link: https://github.com/apache/kvrocks/discussions/1628#discussioncomment-6622687

----
This is an automatically sent email for issues@kvrocks.apache.org.
To unsubscribe, please send an email to: issues-unsubscribe@kvrocks.apache.org