You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@kvrocks.apache.org by "stpp2 (via GitHub)" <gi...@apache.org> on 2024/11/24 16:36:17 UTC
[D] Expected use of Memory [kvrocks]
GitHub user stpp2 created a discussion: Expected use of Memory
Hey all,
Thank you for this great library!
I'm running a pretty basic `kvrocks` node with about 20GB of data on disk. Current memory usage is about 2GB.
I believe memory usage is correlated with the total size of the data stored on disk (due to indexes, perhaps?)
My question is - is there a way to limit the memory to, say, 4GB or 8GB and make sure it never uses more than that? (while hopefully keeping decent performance levels)
Thanks!
GitHub link: https://github.com/apache/kvrocks/discussions/2679
----
This is an automatically sent email for issues@kvrocks.apache.org.
To unsubscribe, please send an email to: issues-unsubscribe@kvrocks.apache.org
Re: [D] Expected use of Memory [kvrocks]
Posted by "stpp2 (via GitHub)" <gi...@apache.org>.
GitHub user stpp2 closed a discussion: Expected use of Memory
Hey all,
Thank you for this great library!
I'm running a pretty basic `kvrocks` node with about 20GB of data on disk. Current memory usage is about 2GB.
I believe memory usage is correlated with the total size of the data stored on disk (due to indexes, perhaps?)
My question is - is there a way to limit the memory to, say, 4GB or 8GB and make sure it never uses more than that? (while hopefully keeping decent performance levels)
Thanks!
GitHub link: https://github.com/apache/kvrocks/discussions/2679
----
This is an automatically sent email for issues@kvrocks.apache.org.
To unsubscribe, please send an email to: issues-unsubscribe@kvrocks.apache.org
Re: [D] Expected use of Memory [kvrocks]
Posted by "stpp2 (via GitHub)" <gi...@apache.org>.
GitHub user stpp2 added a comment to the discussion: Expected use of Memory
Got it, thanks!
So there's really shouldn't be a concern that memory would go way beyond those 4GB right? (assuming I'm running on 5-6GB of memory)
Just trying to confirm it won't result in crashing the server or anything like that.
GitHub link: https://github.com/apache/kvrocks/discussions/2679#discussioncomment-11376325
----
This is an automatically sent email for issues@kvrocks.apache.org.
To unsubscribe, please send an email to: issues-unsubscribe@kvrocks.apache.org
Re: [D] Expected use of Memory [kvrocks]
Posted by "stpp2 (via GitHub)" <gi...@apache.org>.
GitHub user stpp2 added a comment to the discussion: Expected use of Memory
Hi @git-hulk,
Thank you so much for the insight.
The thing I didn't understand, however, is if this `4096MB` value is the overall set limit, or if memory can be be higher than that.
For example, if I kept that at `4096` but also had 500GB of stored data - would the overall memory use still be around 4GB?
I fully understand how it can affect performance, but I'm also trying to make sure the server doesn't crash due to insufficient memory. (unless this case is already handled by the lib)
Thanks again!
GitHub link: https://github.com/apache/kvrocks/discussions/2679#discussioncomment-11371158
----
This is an automatically sent email for issues@kvrocks.apache.org.
To unsubscribe, please send an email to: issues-unsubscribe@kvrocks.apache.org
Re: [D] Expected use of Memory [kvrocks]
Posted by "git-hulk (via GitHub)" <gi...@apache.org>.
GitHub user git-hulk added a comment to the discussion: Expected use of Memory
Hi @stpp2
The memory usage is generally affected by the configuration: `rocksdb.block_cache_size` which the default value is `4096MiB`. You can reduce it to a smaller value, but keeping this value larger will benefit the read performance.
GitHub link: https://github.com/apache/kvrocks/discussions/2679#discussioncomment-11370042
----
This is an automatically sent email for issues@kvrocks.apache.org.
To unsubscribe, please send an email to: issues-unsubscribe@kvrocks.apache.org
Re: [D] Expected use of Memory [kvrocks]
Posted by "stpp2 (via GitHub)" <gi...@apache.org>.
GitHub user stpp2 added a comment to the discussion: Expected use of Memory
Thank you both @PragmaTwice and @git-hulk !!
Really appreciate your work on this repo, it's been great using it so far.
GitHub link: https://github.com/apache/kvrocks/discussions/2679#discussioncomment-11382477
----
This is an automatically sent email for issues@kvrocks.apache.org.
To unsubscribe, please send an email to: issues-unsubscribe@kvrocks.apache.org
Re: [D] Expected use of Memory [kvrocks]
Posted by "git-hulk (via GitHub)" <gi...@apache.org>.
GitHub user git-hulk added a comment to the discussion: Expected use of Memory
> For example, if I kept that at 4096 but also had 500GB of stored data - would the overall memory use still be around 4GB?
Yes, it will be around 4GB but it might be a bit larger than that. Because Kvrocks will use some memory for keeping connections and other states, but it won't eat too much memory.
GitHub link: https://github.com/apache/kvrocks/discussions/2679#discussioncomment-11371835
----
This is an automatically sent email for issues@kvrocks.apache.org.
To unsubscribe, please send an email to: issues-unsubscribe@kvrocks.apache.org
Re: [D] Expected use of Memory [kvrocks]
Posted by "PragmaTwice (via GitHub)" <gi...@apache.org>.
GitHub user PragmaTwice added a comment to the discussion: Expected use of Memory
Yeah, the major part of memory consumption of Kvrocks is cache mechanisms in RocksDB.
GitHub link: https://github.com/apache/kvrocks/discussions/2679#discussioncomment-11378597
----
This is an automatically sent email for issues@kvrocks.apache.org.
To unsubscribe, please send an email to: issues-unsubscribe@kvrocks.apache.org