You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@kvrocks.apache.org by "MarcoLugo (via GitHub)" <gi...@apache.org> on 2024/11/19 21:33:06 UTC
[D] Recommended configuration for large database? [kvrocks]
GitHub user MarcoLugo created a discussion: Recommended configuration for large database?
Thanks for creating kvrocks, it's a wonderful use of rocksdb. I do have a question as I may not yet understand every single kvrocks/rocksdb parameter there is.
I've been testing kvrocks and obtaining great results with the default configuration so far, and `workers` set to the number of CPU cores. Excellent throughput basically saturating the NVMe and pretty low latency for `GET` (i.e. P99 below 1ms). This is for small keys (32 bytes or less) and values of 4kb or less. This is for a cluster of 3 master nodes.
Now, up to about 200 million keys, all that performance holds up well but at 400 million keys things start to deteriorate: throughput is cut in more than half and latency doubles. I think the data directory is a little over 100GB at that point. Maybe there is some additional tuning that is required at this scale and above? Is there a recommended configuration for billions of keys?
FWIW, I tried these two parameters but it didn't change anything https://github.com/apache/kvrocks/issues/366#issuecomment-937794183.
Thank you.
GitHub link: https://github.com/apache/kvrocks/discussions/2673
----
This is an automatically sent email for issues@kvrocks.apache.org.
To unsubscribe, please send an email to: issues-unsubscribe@kvrocks.apache.org
Re: [D] Recommended configuration for large database? [kvrocks]
Posted by "MarcoLugo (via GitHub)" <gi...@apache.org>.
GitHub user MarcoLugo added a comment to the discussion: Recommended configuration for large database?
Thank you @git-hulk . What I have observed so far is that with say, below 60GB of files, it uses all CPUs (32) to about 50-55% (with that being divided between normal threads and kernel threads) and I/O is high enough to be saturated. I should have mentioned, this is a read-intensive workload.
Now, when above 100GB or so, for the same workload, CPU utilization in the same instance actually goes down and so does the I/O throughput and the database OPS and latency go down accordingly.
I did try setting `rocksdb.block_cache_size` to 16384 and even to `32768` and saw no changes. I'll try again later to double check.
GitHub link: https://github.com/apache/kvrocks/discussions/2673#discussioncomment-11321049
----
This is an automatically sent email for issues@kvrocks.apache.org.
To unsubscribe, please send an email to: issues-unsubscribe@kvrocks.apache.org
Re: [D] Recommended configuration for large database? [kvrocks]
Posted by "git-hulk (via GitHub)" <gi...@apache.org>.
GitHub user git-hulk added a comment to the discussion: Recommended configuration for large database?
@MarcoLugo What's your Kvrocks version?
GitHub link: https://github.com/apache/kvrocks/discussions/2673#discussioncomment-11321670
----
This is an automatically sent email for issues@kvrocks.apache.org.
To unsubscribe, please send an email to: issues-unsubscribe@kvrocks.apache.org
Re: [D] Recommended configuration for large database? [kvrocks]
Posted by "MarcoLugo (via GitHub)" <gi...@apache.org>.
GitHub user MarcoLugo added a comment to the discussion: Recommended configuration for large database?
@git-hulk Interesting! I was using `2.10.1` in both cases. I'll try `2.8` today or tomorrow and report back.
GitHub link: https://github.com/apache/kvrocks/discussions/2673#discussioncomment-11325808
----
This is an automatically sent email for issues@kvrocks.apache.org.
To unsubscribe, please send an email to: issues-unsubscribe@kvrocks.apache.org
Re: [D] Recommended configuration for large database? [kvrocks]
Posted by "git-hulk (via GitHub)" <gi...@apache.org>.
GitHub user git-hulk added a comment to the discussion: Recommended configuration for large database?
Hi @MarcoLugo
Want to know most of your operations are `GET`? I cannot tell what configurations could be helpful with the current information. But I think you can check the usage of CPU/Disk IO to see if they're close to the bottleneck. And for the `GET` operation, you can increase the value of `rocksdb.block_cache_size` if your memory is still enough which the default is 4096(4GiB).
GitHub link: https://github.com/apache/kvrocks/discussions/2673#discussioncomment-11317985
----
This is an automatically sent email for issues@kvrocks.apache.org.
To unsubscribe, please send an email to: issues-unsubscribe@kvrocks.apache.org
Re: [D] Recommended configuration for large database? [kvrocks]
Posted by "MarcoLugo (via GitHub)" <gi...@apache.org>.
GitHub user MarcoLugo added a comment to the discussion: Recommended configuration for large database?
@git-hulk , I am using:
```
~# ./kvrocks --version
kvrocks version 2.10.1 (commit dc4d5303)
```
GitHub link: https://github.com/apache/kvrocks/discussions/2673#discussioncomment-11322073
----
This is an automatically sent email for issues@kvrocks.apache.org.
To unsubscribe, please send an email to: issues-unsubscribe@kvrocks.apache.org
Re: [D] Recommended configuration for large database? [kvrocks]
Posted by "git-hulk (via GitHub)" <gi...@apache.org>.
GitHub user git-hulk added a comment to the discussion: Recommended configuration for large database?
@MarcoLugo Thank you! Are you also using this version before? we got the feedback from another user that 2.10.1(rocksdb 9.x) performed worse than 2.8(rocksdb is 8.x).
GitHub link: https://github.com/apache/kvrocks/discussions/2673#discussioncomment-11323551
----
This is an automatically sent email for issues@kvrocks.apache.org.
To unsubscribe, please send an email to: issues-unsubscribe@kvrocks.apache.org
Re: [D] Recommended configuration for large database? [kvrocks]
Posted by "MarcoLugo (via GitHub)" <gi...@apache.org>.
GitHub user MarcoLugo added a comment to the discussion: Recommended configuration for large database?
@git-hulk I tried with `kvrocks version 2.8.0 (commit 311398a2)` this time and the result was roughly the same, unfortunately. Using the following non-default parameters:
```
--workers 32 --rocksdb.block_cache_size 16384 --rocksdb.subkey_block_cache_size 10240 --rocksdb.share_metadata_and_subkey_block_cache yes
```
Any other suggestions?
GitHub link: https://github.com/apache/kvrocks/discussions/2673#discussioncomment-11327591
----
This is an automatically sent email for issues@kvrocks.apache.org.
To unsubscribe, please send an email to: issues-unsubscribe@kvrocks.apache.org
Re: [D] Recommended configuration for large database? [kvrocks]
Posted by "git-hulk (via GitHub)" <gi...@apache.org>.
GitHub user git-hulk added a comment to the discussion: Recommended configuration for large database?
@MarcoLugo Yes, that's not related to it if you're using 2.10 in both cases. What's the instance throughput and P99 latency for now?
You can remove ` --rocksdb.subkey_block_cache_size 10240 --rocksdb.share_metadata_and_subkey_block_cache yes` since they are deprecated parameters. And you can try to decrease worker threads to `16`.
You can try profiling it on one of your instance:
Step 1: Enabled the profiling
```
redis-cli > CONFIG SET profiling-sample-commands "GET"
redis-cli > CONFIG SET profiling-sample-ratio 1
redis-cli > CONFIG SET profiling-sample-record-threshold-ms 100
```
Step 2: Investigate the profiling logs
```
# Get latest 10 profiling logs
redis-cli > perflog get 10
```
GitHub link: https://github.com/apache/kvrocks/discussions/2673#discussioncomment-11332246
----
This is an automatically sent email for issues@kvrocks.apache.org.
To unsubscribe, please send an email to: issues-unsubscribe@kvrocks.apache.org