You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@kvrocks.apache.org by "git-hulk (via GitHub)" <gi...@apache.org> on 2024/03/23 10:44:50 UTC
[I] Reduce the number of write buffers for infrequently used column families [kvrocks]
git-hulk opened a new issue, #2193:
URL: https://github.com/apache/kvrocks/issues/2193
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/kvrocks/issues) and found no similar issues.
### Motivation
Currently, all column families are shared the same number of write buffers
which is unnecessary for infrequently used column families including `pubsub`,
`propagte`, and `zset_score`. We can keep them to a const value like 1 to reduce
the memory usage in some scenarios.
### Solution
_No response_
### Are you willing to submit a PR?
- [ ] I'm willing to submit a PR!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Reduce the number of write buffers for infrequently used column families [kvrocks]
Posted by "git-hulk (via GitHub)" <gi...@apache.org>.
git-hulk commented on issue #2193:
URL: https://github.com/apache/kvrocks/issues/2193#issuecomment-2047504883
@LiuYuHui Thanks for your quick response. Welcome back while you get time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Reduce the number of write buffers for infrequently used column families [kvrocks]
Posted by "git-hulk (via GitHub)" <gi...@apache.org>.
git-hulk commented on issue #2193:
URL: https://github.com/apache/kvrocks/issues/2193#issuecomment-2048830578
@jjz921024 Yes, I'm good with the implementation. Would you like to submit a PR?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Reduce the number of write buffers for infrequently used column families [kvrocks]
Posted by "jjz921024 (via GitHub)" <gi...@apache.org>.
jjz921024 commented on issue #2193:
URL: https://github.com/apache/kvrocks/issues/2193#issuecomment-2047905920
Hi @git-hulk , May i question? How do we implement this feature?
1. For each column family add a config, specifying its number of write buffers. eg: `pubsub.max_write_buffer_number`、`propagte.max_write_buffer_number`、`zset_score.max_write_buffer_number` ...
2. Add a `lite` mode config, When this config is set to `true`, create a new `lite_table_opts` for infrequently used column families. like:
```java
rocksdb::BlockBasedTableOptions lite_table_opts = InitTableOptions();
rocksdb::ColumnFamilyOptions lite_opts(subkey_opts);
lite_opts.write_buffer_size = 32 * MiB;
lite_opts.max_write_buffer_number = 0;
// replace subkey_opts with lite_opts
column_families.emplace_back(kZSetScoreColumnFamilyName, lite_opts);
column_families.emplace_back(kPubSubColumnFamilyName, lite_opts);
column_families.emplace_back(kPropagateColumnFamilyName, lite_opts);
column_families.emplace_back(kStreamColumnFamilyName, lite_opts);```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Reduce the number of write buffers for infrequently used column families [kvrocks]
Posted by "jjz921024 (via GitHub)" <gi...@apache.org>.
jjz921024 commented on issue #2193:
URL: https://github.com/apache/kvrocks/issues/2193#issuecomment-2051732652
Yes, I want to try
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Reduce the number of write buffers for infrequently used column families [kvrocks]
Posted by "git-hulk (via GitHub)" <gi...@apache.org>.
git-hulk commented on issue #2193:
URL: https://github.com/apache/kvrocks/issues/2193#issuecomment-2044122796
Hi @LiuYuHui Are you still working on this issue?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Reduce the number of write buffers for infrequently used column families [kvrocks]
Posted by "git-hulk (via GitHub)" <gi...@apache.org>.
git-hulk commented on issue #2193:
URL: https://github.com/apache/kvrocks/issues/2193#issuecomment-2016523264
@LiuYuHui Thank you!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Reduce the number of write buffers for infrequently used column families [kvrocks]
Posted by "LiuYuHui (via GitHub)" <gi...@apache.org>.
LiuYuHui commented on issue #2193:
URL: https://github.com/apache/kvrocks/issues/2193#issuecomment-2016522579
Please assign it to me, thanks a lot :D
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Reduce the number of write buffers for infrequently used column families [kvrocks]
Posted by "git-hulk (via GitHub)" <gi...@apache.org>.
git-hulk commented on issue #2193:
URL: https://github.com/apache/kvrocks/issues/2193#issuecomment-2051775150
@jjz921024 Assigned, thank you!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Reduce the number of write buffers for infrequently used column families [kvrocks]
Posted by "LiuYuHui (via GitHub)" <gi...@apache.org>.
LiuYuHui commented on issue #2193:
URL: https://github.com/apache/kvrocks/issues/2193#issuecomment-2045410344
Hi @git-hulk, I have had no extra time bandwidth recently, please feel free to assign it to other people.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org