You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@kvrocks.apache.org by GitBox <gi...@apache.org> on 2023/03/24 23:36:16 UTC
[GitHub] [incubator-kvrocks] GoodManWEN created a discussion: How to configure row cache value to improve read performance?
GitHub user GoodManWEN created a discussion: How to configure row cache value to improve read performance?
I read in the configuration file that if the use case is mostly point-like kv search, then enabling row cache may help improve performance. This happens to be my use case exactly, but unfortunately, there is no example given in the configuration file, so I don’t know how to enable this option appropriately.
I have 64GB of physical memory and have already allocated 24GB of `rocksdb.subkey_block_cache_size` and 8GB of `rocksdb.metadata_block_cache_size`. I’m not sure what scenarios they are responsible for and how to allocate the proportions. Any suggestions from our fellow community?
GitHub link: https://github.com/apache/incubator-kvrocks/discussions/1353
----
This is an automatically sent email for issues@kvrocks.apache.org.
To unsubscribe, please send an email to: issues-unsubscribe@kvrocks.apache.org
[GitHub] [incubator-kvrocks] GoodManWEN added a comment to the discussion: How to configure row cache value to improve read performance?
Posted by GitBox <gi...@apache.org>.
GitHub user GoodManWEN added a comment to the discussion: How to configure row cache value to improve read performance?
Yes, this option is enabled. So from your reply, I deduce that after enabling this option, the sum of the two(metadata & subkey cache size) should be close to the memory occupied by the application under normal circumstances, my situation is a bit incorrect?
GitHub link: https://github.com/apache/incubator-kvrocks/discussions/1353#discussioncomment-5424912
----
This is an automatically sent email for issues@kvrocks.apache.org.
To unsubscribe, please send an email to: issues-unsubscribe@kvrocks.apache.org
[GitHub] [incubator-kvrocks] GoodManWEN added a comment to the discussion: How to configure row cache value to improve read performance?
Posted by GitBox <gi...@apache.org>.
GitHub user GoodManWEN added a comment to the discussion: How to configure row cache value to improve read performance?
Thank you @git-hulk . So actually it doesn’t matter what their allocation ratio is, as long as the total amount is the same, is that what it means?
In the test I conducted a few days ago, I tested about 100 million keys of random data, and each value was about 16KB in length. I configured a total of 32GB maximum cache of 64GB for the above two items. During global random reading test, the memory usage has been maintained at around 8G, which I think is a relatively low level and makes me doubt whether the memory has been fully utilized (of course, the system cache has almost filled up all 64GB). According to the optimization suggestions I saw in on of the rocksdb tuning document before, it suggests setting the block cache to one-third of the physical memory and leaving the rest to the system cache. I am not sure if his wording refers to setting the upper limit in the setting item to one-third or whether one-third of physical memory should be occupied during actual operation. It would be very helpful if you could provide some additional explanations.
GitHub link: https://github.com/apache/incubator-kvrocks/discussions/1353#discussioncomment-5424686
----
This is an automatically sent email for issues@kvrocks.apache.org.
To unsubscribe, please send an email to: issues-unsubscribe@kvrocks.apache.org
[GitHub] [incubator-kvrocks] GoodManWEN added a comment to the discussion: How to configure row cache value to improve read performance?
Posted by GitBox <gi...@apache.org>.
GitHub user GoodManWEN added a comment to the discussion: How to configure row cache value to improve read performance?
Btw, could you please provide a configuration example of the option in this link [https://github.com/apache/incubator-kvrocks/blob/unstable/kvrocks.conf#L537](https://github.com/apache/incubator-kvrocks/blob/unstable/kvrocks.conf#L537). Due to the lack of examples, I am not sure what unit this option is in. Should I set it to 1, or several bytes, or several GB? In addition, about row cache, it seems that this is not a native rocksDB function, but a feature of kvrocks. I don’t know if there is a recommended size for setting it. What is the appropriate physical memory usage ratio for multi-read and few-write scenarios?
GitHub link: https://github.com/apache/incubator-kvrocks/discussions/1353#discussioncomment-5424926
----
This is an automatically sent email for issues@kvrocks.apache.org.
To unsubscribe, please send an email to: issues-unsubscribe@kvrocks.apache.org
[GitHub] [incubator-kvrocks] git-hulk added a comment to the discussion: How to configure row cache value to improve read performance?
Posted by GitBox <gi...@apache.org>.
GitHub user git-hulk added a comment to the discussion: How to configure row cache value to improve read performance?
Can you check if [rocksdb.share_metadata_and_subkey_block_cache](https://github.com/apache/incubator-kvrocks/blob/unstable/kvrocks.conf#L530) was enabled?
GitHub link: https://github.com/apache/incubator-kvrocks/discussions/1353#discussioncomment-5424827
----
This is an automatically sent email for issues@kvrocks.apache.org.
To unsubscribe, please send an email to: issues-unsubscribe@kvrocks.apache.org
[GitHub] [incubator-kvrocks] git-hulk added a comment to the discussion: How to configure row cache value to improve read performance?
Posted by GitBox <gi...@apache.org>.
GitHub user git-hulk added a comment to the discussion: How to configure row cache value to improve read performance?
> Yes, this option is enabled. So from your reply, I deduce that after enabling this option, the sum of the two(metadata & subkey cache size) should be close to the memory occupied by the application under normal circumstances, my situation is a bit incorrect?
That's right. The block size should roughly reach the sum of those two configurations if the dataset is large enough.
GitHub link: https://github.com/apache/incubator-kvrocks/discussions/1353#discussioncomment-5425032
----
This is an automatically sent email for issues@kvrocks.apache.org.
To unsubscribe, please send an email to: issues-unsubscribe@kvrocks.apache.org
[GitHub] [incubator-kvrocks] git-hulk added a comment to the discussion: How to configure row cache value to improve read performance?
Posted by GitBox <gi...@apache.org>.
GitHub user git-hulk added a comment to the discussion: How to configure row cache value to improve read performance?
> Due to the lack of examples, I am not sure what unit this option is in. Should I set it to 1, or several bytes, or several GB?
Good catch, we should describe the unit in the configuration file, the unit is MiB indeed.
GitHub link: https://github.com/apache/incubator-kvrocks/discussions/1353#discussioncomment-5425027
----
This is an automatically sent email for issues@kvrocks.apache.org.
To unsubscribe, please send an email to: issues-unsubscribe@kvrocks.apache.org
[GitHub] [incubator-kvrocks] git-hulk added a comment to the discussion: How to configure row cache value to improve read performance?
Posted by GitBox <gi...@apache.org>.
GitHub user git-hulk added a comment to the discussion: How to configure row cache value to improve read performance?
Hi @GoodManWEN
We have enabled the configuration `https://github.com/apache/incubator-kvrocks/blob/unstable/kvrocks.conf#L530` by default which means those two configurations will share the memory cache. And we still keep them instead of the single one for compatibility reasons.
GitHub link: https://github.com/apache/incubator-kvrocks/discussions/1353#discussioncomment-5424547
----
This is an automatically sent email for issues@kvrocks.apache.org.
To unsubscribe, please send an email to: issues-unsubscribe@kvrocks.apache.org