[D] Indexes and filters in memory [kvrocks]

["bartek-siudeja (via GitHub)" <gi...@apache.org>]

GitHub user bartek-siudeja created a discussion: Indexes and filters in memory

kvrocks is quite impressive! Thanks for doing this. 

We are trying to build a KV store with extremely strict latency/throughput requirements. We are working with really large servers, with lots of memory. Our values are relatively big and keys are random and small. So it would make sense to store all indexes and big bloom filters in memory. It seems this can be achieved using `cache_index_and_filter_blocks: no` and `max_open_files: -1`. However, kvrocks enables partitioning (at least for filters, not sure about indexes):
https://github.com/apache/kvrocks/blob/2.10/src/storage/storage.cc#L132
And partitioned filters seem to always live inside block_cache, so they can be evicted.

Would it be possible to expose the partitioning option in the config file? Or maybe have a "meta option" to force filters/indexes to stay in memory. Along the same line bloom filter size could be increased from 10 to say 15 bits to really avoid hitting the disk at any cost. Really only when we are pretty much sure key is there, and value can be retrieved. But this option is also hardcoded (a line above filter partitioning).

GitHub link: https://github.com/apache/kvrocks/discussions/2675

----
This is an automatically sent email for issues@kvrocks.apache.org.
To unsubscribe, please send an email to: issues-unsubscribe@kvrocks.apache.org

Re: [D] Indexes and filters in memory [kvrocks]

["git-hulk (via GitHub)" <gi...@apache.org>]

GitHub user git-hulk edited a comment on the discussion: Indexes and filters in memory

> Would it be possible to expose the partitioning option in the config file? Or maybe have a "meta option" to force filters/indexes to stay in memory. 

@bartek-siudeja Yes, that makes sense. I will create an issue from this discussion to track this.

GitHub link: https://github.com/apache/kvrocks/discussions/2675#discussioncomment-11343050

----
This is an automatically sent email for issues@kvrocks.apache.org.
To unsubscribe, please send an email to: issues-unsubscribe@kvrocks.apache.org

Re: [D] Indexes and filters in memory [kvrocks]

["git-hulk (via GitHub)" <gi...@apache.org>]

GitHub user git-hulk added a comment to the discussion: Indexes and filters in memory

> Would it be possible to expose the partitioning option in the config file? Or maybe have a "meta option" to force filters/indexes to stay in memory. 

Yes, that makes sense. I will create an issue from this discussion to track this.

GitHub link: https://github.com/apache/kvrocks/discussions/2675#discussioncomment-11343050

----
This is an automatically sent email for issues@kvrocks.apache.org.
To unsubscribe, please send an email to: issues-unsubscribe@kvrocks.apache.org