You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kvrocks.apache.org by Benoit TELLIER <bt...@linagora.com> on 2025/03/03 08:12:22 UTC
Apache James meets KVRocks
Hello folks,
I'm involved on the Apache James email server, and as part of Apache James we currently have a Redis integration.
Redis is used for:
- 1. Implementing rate limiting in a distributed fashion by using leaky bucket algorithm
- 2. Implementing sid storage as part of our OpenID connect backchannel logout (integrated in Apache APISIX - java plugin runner)
- 3. Use redis streams for backing apache James pub sub (~email notifications)
- 4. Store bayes database for our RSPAMD setup
Our setup needs to be using sentinel (some use case like 4. relies on commands not supplied from Redis cluster)
Our biggest pain is that, as we use a per user database for bayes the database grows significantly and storing this DB in memory is not financially viable.
In order to solve our concerns with 4. (and as we are not performance bound for Redis components) we are considering switching our use cases to KVrocks, and potentially makes KVrocks the default key value store / cache for Apache James.
I (in the name of the Apache James PMC and LINAGORA contributors) would happily report on how this integrations goes on.
--
Best regards,
Benoit TELLIER
General manager of Linagora VIETNAM.
Product owner for Team-Mail product.
Chairman of the Apache James project.
Mail: btellier@linagora.com
Tel: (0033) 6 77 26 04 58 (WhatsApp, Signal)
Re: Apache James meets KVRocks
Posted by hulk <hu...@gmail.com>.
Hi, Quan
I greatly appreciate your effort and detailed information.
> Maybe the gradual memory growth issue is more a Jemalloc's nature than Kvrocks issue itself?
I think so. From your information, it did look like most of the memory
are reclaimable cache since they cannot be returned to the OS
when running with memory-intensive processes
I will repost your post link in GitHub Discussions to let other users
have a try. Thank you!
On Thu, 8 May 2025 at 12:44, Hong Quan TRAN <hq...@linagora.com> wrote:
>
> Hi Hulk,
> I’d like to share some results from an experiment I ran related to Jemalloc memory tuning in Kvrocks, which may be helpful to the gradual memory growth over time.I read a Apache Doris jemalloc analysis article, where they seems to claim more or the the same issue as our. Jemalloc retained memory aggressively by default, leading to high memory usage that the OS could not easily reclaim. They suggest to restart Apache Doris after sometime to reclaim the memory, or use the retain:false Jemalloc option to force Jemalloc return the unused memory to OS.
>
> I tried the option with Kvrocks using the environment variable MALLOC_CONF=retain:false.
>
> After applying retain=false and stress tests Kvrocks using redis-benchmark a few dozen times, I observed:
>
>
> - Docker stats memory: consistently under 800MB (was more than 1.3GB if retain:true)
>
>
> - Kvrocks-reported RSS memory: consistently under 500MB
>
>
> - When running other memory-intensive processes, Kvrocks memory usage dropped by ~150MB, suggesting part of the usage was reclaimable cache.
>
>
>
> This aligns with Jemalloc behavior where disabling retain makes memory more readily returnable to the OS.
>
> Maybe the gradual memory growth issue is more a Jemalloc's nature than Kvrocks issue itself?
>
> Regarding performance metrics, I did not deeply dig into it yet. A quick benchmark on my side showed no significant difference between the two settings.
>
> Please let me know your thoughts on this matter.
>
> Quan
>
>
>
> On May 3, 2025 9:10 PM, from Hulk <hu...@gmail.com>Hi Quan,
>
> I find a gap in the memory usage in Docker stats, which is about double
> that of Kvrocks's RSS.
>
> You might have a check as well using the command `info memory` via
> redis-cli.
>
> Not sure if it's caused by the different algorithm when counting the memory
> usage.
>
>
> *docker stats a68a50cf4a82*
>
> a68a50cf4a82 tender_jackson 143.39% 635MiB / 14.63GiB 4.24%
> 22.8GB / 4.7GB 844kB / 21GB 18
>
> *hulk@instance-20241021-041032:~$ redis-cli -p 6666 info memory*
>
> # Memory
> used_memory_rss:247369728
> used_memory_rss_human:235.91M
> used_memory_lua:319488
> used_memory_lua_human:312.00K
> used_memory_startup:33468416
>
>
> On Sat, 3 May 2025 at 19:02, hulk <hu...@gmail.com> wrote:
>
> > Hi, Quan
> >
> > Thanks for your detailed information. I can reproduce it on my side. I
> > ran the benchmark a few times, and its memory is around the same(about
> > 1.3GiB),
> > but I haven't dived deep into the Jemalloc stats yet.
> >
> > On Fri, 2 May 2025 at 17:31, Hong Quan TRAN <hq...@linagora.com> wrote:
> > >
> > > Hi Hulk,
> > >
> > > Sorry for the late reply (we were on holiday).
> > >
> > > > I have no clue about why it's up to 1.3 GiB, will try to reproduce
> > > this on my side and analyze the memory in Jemalloc.
> > >
> > > We use Kvrocks as Rspamd's bayes training database. I would attach the
> > Kvrocks used command stats for more details.
> > >
> > > BTW on my side, I tried to run the redis-benchmark tool against Kvrocks
> > (still the same configuration). After a dozen times running
> > `redis-benchmark -n 2000000 -c 50` command, Kvrocks memory is upto around
> > 1.1GB and does not seems to be released automatically after a while.
> > >
> > > I hope this would help you ease reproducing the issue. I would be happy
> > to do more diagnostic and share the output to you if needed.
> > >
> > > Regards,
> > >
> > > Quan
> > >
> > >
> > > On Apr 27, 2025 4:27 PM, from Hulk <hu...@gmail.com>
> > >
> > > Hi Benoit, Quan
> > >
> > > Sorry for the late reply. I took a look at your configuration, and it
> > > all looks good.
> > >
> > > I have no clue about why it's up to 1.3 GiB, will try to reproduce
> > > this on my side and analyze the memory in Jemalloc.
> > >
> > > Will update you once we have any progress.
> > >
> > > On Mon, 21 Apr 2025 at 11:00, hulk <hu...@gmail.com> wrote:
> > > >
> > > > Thanks all for taking care of this, will have a look.
> > > >
> > > > On Mon, 21 Apr 2025 at 10:56, Hong Quan TRAN <hq...@linagora.com>
> > wrote:
> > > >>
> > > >> Hi,
> > > >>
> > > >> Somehow the format is still bad (although I tested sending it to my
> > personal mail).
> > > >>
> > > >> I would send the Kvrocks config output via an attachment this time.
> > > >>
> > > >> Quan
> > > >>
> > > >> Vào ngày thg 4 21, 2025 9:39 SA, từ Hong Quan Tran <
> > hqtran@linagora.com>
> > > >>
> > > >> Hi everyone,
> > > >> Sorry for the bad formatting output.
> > > >> I resend the config output. I hope it is fine this time.
> > > >>
> > > >> kvrocks@561e141f9b72:/$ redis-cli -p 6379 CONFIG GET '*' 1)
> > "auto-resize-block-and-sst" 2) "yes" 3) "backup-dir" 4)
> > "/var/lib/kvrocks/backup" 5) "bgsave-cron" 6) "" 7) "bind" 8) "0.0.0.0" 9)
> > "cluster-enabled" 10) "no" 11) "compact-cron" 12) "" 13)
> > "compaction-checker-cron" 14) "" 15) "compaction-checker-range" 16) "" 17)
> > "daemonize" 18) "no" 19) "db-name" 20) "change.me.db" 21)
> > "dbsize-scan-cron" 22) "" 23) "dir" 24) "/var/lib/kvrocks" 25)
> > "force-compact-file-age" 26) "172800" 27)
> > "force-compact-file-min-deleted-percentage" 28) "10" 29)
> > "fullsync-recv-file-delay" 30) "0" 31) "histogram-bucket-boundaries" 32) ""
> > 33) "json-max-nesting-depth" 34) "1024" 35) "json-storage-format" 36)
> > "json" 37) "log-dir" 38) "/var/lib/kvrocks" 39) "log-level" 40) "info" 41)
> > "log-retention-days" 42) "-1" 43) "master-use-repl-port" 44) "no" 45)
> > "masterauth" 46) "" 47) "max-backup-keep-hours" 48) "0" 49)
> > "max-backup-to-keep" 50) "1" 51) "max-bitmap-to-string-mb" 52) "16" 53)
> > "max-db-size" 54) "0" 55) "max-io-mb" 56) "0" 57) "max-replication-mb" 58)
> > "0" 59) "maxclients" 60) "10240" 61) "migrate-batch-rate-limit-mb" 62) "16"
> > 63) "migrate-batch-size-kb" 64) "16" 65) "migrate-pipeline-size" 66) "16"
> > 67) "migrate-sequence-gap" 68) "10000" 69) "migrate-speed" 70) "4096" 71)
> > "migrate-type" 72) "redis-command" 73) "persist-cluster-nodes-enabled" 74)
> > "yes" 75) "pidfile" 76) "/var/run/kvrocks/kvrocks.pid" 77) "port" 78)
> > "6379" 79) "profiling-sample-commands" 80) "" 81) "profiling-sample-ratio"
> > 82) "0" 83) "profiling-sample-record-max-len" 84) "256" 85)
> > "profiling-sample-record-threshold-ms" 86) "100" 87) "proto-max-bulk-len"
> > 88) "536870912" 89) "purge-backup-on-fullsync" 90) "no" 91)
> > "redis-cursor-compatible" 92) "yes" 93) "repl-namespace-enabled" 94) "no"
> > 95) "replica-announce-ip" 96) "" 97) "replica-announce-port" 98) "0" 99)
> > "replicaof"100) ""101) "replication-connect-timeout-ms"102) "3100"103)
> > "replication-recv-timeout-ms"104) "3200"105) "requirepass"106) ""107)
> > "resp3-enabled"108) "no"109) "rocksdb.avoid_unnecessary_blocking_io"110)
> > "yes"111) "rocksdb.blob_file_size"112) "268435456"113)
> > "rocksdb.blob_garbage_collection_age_cutoff"114) "25"115)
> > "rocksdb.block_cache_size"116) "256"117) "rocksdb.block_cache_type"118)
> > "lru"119) "rocksdb.block_size"120) "16384"121)
> > "rocksdb.cache_index_and_filter_blocks"122) "yes"123)
> > "rocksdb.compaction_readahead_size"124) "2097152"125)
> > "rocksdb.compression"126) "no"127) "rocksdb.compression_level"128)
> > "32767"129) "rocksdb.compression_start_level"130) "2"131)
> > "rocksdb.delayed_write_rate"132) "0"133)
> > "rocksdb.disable_auto_compactions"134) "no"135)
> > "rocksdb.dump_malloc_stats"136) "yes"137) "rocksdb.enable_blob_files"138)
> > "no"139) "rocksdb.enable_blob_garbage_collection"140) "yes"141)
> > "rocksdb.enable_pipelined_write"142) "no"143)
> > "rocksdb.level0_file_num_compaction_trigger"144) "4"145)
> > "rocksdb.level0_slowdown_writes_trigger"146) "20"147)
> > "rocksdb.level0_stop_writes_trigger"148) "40"149)
> > "rocksdb.level_compaction_dynamic_level_bytes"150) "no"151)
> > "rocksdb.max_background_compactions"152) "2"153)
> > "rocksdb.max_background_flushes"154) "2"155)
> > "rocksdb.max_background_jobs"156) "4"157)
> > "rocksdb.max_bytes_for_level_base"158) "268435456"159)
> > "rocksdb.max_bytes_for_level_multiplier"160) "10"161)
> > "rocksdb.max_open_files"162) "8096"163) "rocksdb.max_subcompactions"164)
> > "2"165) "rocksdb.max_total_wal_size"166) "256"167)
> > "rocksdb.max_write_buffer_number"168) "2"169)
> > "rocksdb.metadata_block_cache_size"170) "2048"171)
> > "rocksdb.min_blob_size"172) "4096"173) "rocksdb.partition_filters"174)
> > "yes"175) "rocksdb.rate_limiter_auto_tuned"176) "yes"177)
> > "rocksdb.read_options.async_io"178) "yes"179) "rocksdb.row_cache_size"180)
> > "0"181) "rocksdb.share_metadata_and_subkey_block_cache"182) "yes"183)
> > "rocksdb.stats_dump_period_sec"184) "0"185)
> > "rocksdb.subkey_block_cache_size"186) "2048"187)
> > "rocksdb.target_file_size_base"188) "128"189) "rocksdb.wal_compression"190)
> > "no"191) "rocksdb.wal_size_limit_mb"192) "16384"193)
> > "rocksdb.wal_ttl_seconds"194) "10800"195) "rocksdb.write_buffer_size"196)
> > "32"197) "rocksdb.write_options.disable_wal"198) "no"199)
> > "rocksdb.write_options.low_pri"200) "no"201)
> > "rocksdb.write_options.memtable_insert_hint_per_batch"202) "no"203)
> > "rocksdb.write_options.no_slowdown"204) "no"205)
> > "rocksdb.write_options.sync"206) "no"207)
> > "rocksdb.write_options.write_batch_max_bytes"208) "0"209)
> > "skip-block-cache-deallocation-on-close"210) "no"211)
> > "slave-empty-db-before-fullsync"212) "no"213) "slave-priority"214)
> > "100"215) "slave-read-only"216) "yes"217) "slave-serve-stale-data"218)
> > "yes"219) "slaveof"220) ""221) "slowlog-log-slower-than"222) "200000"223)
> > "slowlog-max-len"224) "128"225) "socket-fd"226) "-1"227) "supervised"228)
> > "no"229) "tcp-backlog"230) "511"231) "timeout"232) "0"233)
> > "tls-auth-clients"234) ""235) "tls-ca-cert-dir"236) ""237)
> > "tls-ca-cert-file"238) ""239) "tls-cert-file"240) ""241) "tls-ciphers"242)
> > ""243) "tls-ciphersuites"244) ""245) "tls-key-file"246) ""247)
> > "tls-key-file-pass"248) ""249) "tls-port"250) "0"251)
> > "tls-prefer-server-ciphers"252) "no"253) "tls-protocols"254) ""255)
> > "tls-replication"256) "no"257) "tls-session-cache-size"258) "20480"259)
> > "tls-session-cache-timeout"260) "300"261) "tls-session-caching"262)
> > "yes"263) "txn-context-enabled"264) "no"265) "unixsocket"266) ""267)
> > "unixsocketperm"268) "777"269) "use-rsid-psync"270) "no"271) "workers"272)
> > "8"
> > > >> Quan
> > > >> Vào ngày thg 4 21, 2025 1:52 SA, từ Benoit Tellier <
> > btellier@linagora.com>Careful
> > > >>
> > > >> Formating sucks
> > > >>
> > > >> Maybe resent it as a plain/text version to avoid this pitfall?
> > > >>
> > > >> --
> > > >>
> > > >> Best regards,
> > > >>
> > > >> Benoit TELLIER
> > > >>
> > > >> General manager of Linagora VIETNAM.
> > > >> Product owner for Team-Mail product.
> > > >> Chairman of the Apache James project.
> > > >>
> > > >> Mail: btellier@linagora.com
> > > >> Tel: (0033) 6 77 26 04 58 (WhatsApp, Signal)
> > > >>
> > > >>
> > > >> On Apr 20, 2025 3:43 PM, from Hong Quan Tran <hq...@linagora.com>Hi
> > Hulk,
> > > >> > Thanks for your information, could you please provide the output by
> > running the command in redis-cli `config get *`?
> > > >> Please see the output below.
> > > >>
> > > >> kvrocks@561e141f9b72:/$ redis-cli -p 6379 CONFIG GET '*' 1)
> > "auto-resize-block-and-sst" 2) "yes" 3) "backup-dir" 4)
> > "/var/lib/kvrocks/backup" 5) "bgsave-cron" 6) "" 7) "bind" 8) "0.0.0.0" 9)
> > "cluster-enabled" 10) "no" 11) "compact-cron" 12) "" 13)
> > "compaction-checker-cron" 14) "" 15) "compaction-checker-range" 16) "" 17)
> > "daemonize" 18) "no" 19) "db-name" 20) "change.me.db" 21)
> > "dbsize-scan-cron" 22) "" 23) "dir" 24) "/var/lib/kvrocks" 25)
> > "force-compact-file-age" 26) "172800" 27)
> > "force-compact-file-min-deleted-percentage" 28) "10" 29)
> > "fullsync-recv-file-delay" 30) "0" 31) "histogram-bucket-boundaries" 32) ""
> > 33) "json-max-nesting-depth" 34) "1024" 35) "json-storage-format" 36)
> > "json" 37) "log-dir" 38) "/var/lib/kvrocks" 39) "log-level" 40) "info" 41)
> > "log-retention-days" 42) "-1" 43) "master-use-repl-port" 44) "no" 45)
> > "masterauth" 46) "" 47) "max-backup-keep-hours" 48) "0" 49)
> > "max-backup-to-keep" 50) "1" 51) "max-bitmap-to-string-mb" 52) "16" 53)
> > "max-db-size" 54) "0" 55) "max-io-mb" 56) "0" 57) "max-replication-mb" 58)
> > "0" 59) "maxclients" 60) "10240" 61) "migrate-batch-rate-limit-mb" 62) "16"
> > 63) "migrate-batch-size-kb" 64) "16" 65) "migrate-pipeline-size" 66) "16"
> > 67) "migrate-sequence-gap" 68) "10000" 69) "migrate-speed" 70) "4096" 71)
> > "migrate-type" 72) "redis-command" 73) "persist-cluster-nodes-enabled" 74)
> > "yes" 75) "pidfile" 76) "/var/run/kvrocks/kvrocks.pid" 77) "port" 78)
> > "6379" 79) "profiling-sample-commands" 80) "" 81) "profiling-sample-ratio"
> > 82) "0" 83) "profiling-sample-record-max-len" 84) "256" 85)
> > "profiling-sample-record-threshold-ms" 86) "100" 87) "proto-max-bulk-len"
> > 88) "536870912" 89) "purge-backup-on-fullsync" 90) "no" 91)
> > "redis-cursor-compatible" 92) "yes" 93) "repl-namespace-enabled" 94) "no"
> > 95) "replica-announce-ip" 96) "" 97) "replica-announce-port" 98) "0" 99)
> > "replicaof"100) ""101) "replication-connect-timeout-ms"102) "3100"103)
> > "replication-recv-timeout-ms"104) "3200"105) "requirepass"106) ""107)
> > "resp3-enabled"108) "no"109) "rocksdb.avoid_unnecessary_blocking_io"110)
> > "yes"111) "rocksdb.blob_file_size"112) "268435456"113)
> > "rocksdb.blob_garbage_collection_age_cutoff"114) "25"115)
> > "rocksdb.block_cache_size"116) "256"117) "rocksdb.block_cache_type"118)
> > "lru"119) "rocksdb.block_size"120) "16384"121)
> > "rocksdb.cache_index_and_filter_blocks"122) "yes"123)
> > "rocksdb.compaction_readahead_size"124) "2097152"125)
> > "rocksdb.compression"126) "no"127) "rocksdb.compression_level"128)
> > "32767"129) "rocksdb.compression_start_level"130) "2"131)
> > "rocksdb.delayed_write_rate"132) "0"133)
> > "rocksdb.disable_auto_compactions"134) "no"135)
> > "rocksdb.dump_malloc_stats"136) "yes"137) "rocksdb.enable_blob_files"138)
> > "no"139) "rocksdb.enable_blob_garbage_collection"140) "yes"141)
> > "rocksdb.enable_pipelined_write"142) "no"143)
> > "rocksdb.level0_file_num_compaction_trigger"144) "4"145)
> > "rocksdb.level0_slowdown_writes_trigger"146) "20"147)
> > "rocksdb.level0_stop_writes_trigger"148) "40"149)
> > "rocksdb.level_compaction_dynamic_level_bytes"150) "no"151)
> > "rocksdb.max_background_compactions"152) "2"153)
> > "rocksdb.max_background_flushes"154) "2"155)
> > "rocksdb.max_background_jobs"156) "4"157)
> > "rocksdb.max_bytes_for_level_base"158) "268435456"159)
> > "rocksdb.max_bytes_for_level_multiplier"160) "10"161)
> > "rocksdb.max_open_files"162) "8096"163) "rocksdb.max_subcompactions"164)
> > "2"165) "rocksdb.max_total_wal_size"166) "256"167)
> > "rocksdb.max_write_buffer_number"168) "2"169)
> > "rocksdb.metadata_block_cache_size"170) "2048"171)
> > "rocksdb.min_blob_size"172) "4096"173) "rocksdb.partition_filters"174)
> > "yes"175) "rocksdb.rate_limiter_auto_tuned"176) "yes"177)
> > "rocksdb.read_options.async_io"178) "yes"179) "rocksdb.row_cache_size"180)
> > "0"181) "rocksdb.share_metadata_and_subkey_block_cache"182) "yes"183)
> > "rocksdb.stats_dump_period_sec"184) "0"185)
> > "rocksdb.subkey_block_cache_size"186) "2048"187)
> > "rocksdb.target_file_size_base"188) "128"189) "rocksdb.wal_compression"190)
> > "no"191) "rocksdb.wal_size_limit_mb"192) "16384"193)
> > "rocksdb.wal_ttl_seconds"194) "10800"195) "rocksdb.write_buffer_size"196)
> > "32"197) "rocksdb.write_options.disable_wal"198) "no"199)
> > "rocksdb.write_options.low_pri"200) "no"201)
> > "rocksdb.write_options.memtable_insert_hint_per_batch"202) "no"203)
> > "rocksdb.write_options.no_slowdown"204) "no"205)
> > "rocksdb.write_options.sync"206) "no"207)
> > "rocksdb.write_options.write_batch_max_bytes"208) "0"209)
> > "skip-block-cache-deallocation-on-close"210) "no"211)
> > "slave-empty-db-before-fullsync"212) "no"213) "slave-priority"214)
> > "100"215) "slave-read-only"216) "yes"217) "slave-serve-stale-data"218)
> > "yes"219) "slaveof"220) ""221) "slowlog-log-slower-than"222) "200000"223)
> > "slowlog-max-len"224) "128"225) "socket-fd"226) "-1"227) "supervised"228)
> > "no"229) "tcp-backlog"230) "511"231) "timeout"232) "0"233)
> > "tls-auth-clients"234) ""235) "tls-ca-cert-dir"236) ""237)
> > "tls-ca-cert-file"238) ""239) "tls-cert-file"240) ""241) "tls-ciphers"242)
> > ""243) "tls-ciphersuites"244) ""245) "tls-key-file"246) ""247)
> > "tls-key-file-pass"248) ""249) "tls-port"250) "0"251)
> > "tls-prefer-server-ciphers"252) "no"253) "tls-protocols"254) ""255)
> > "tls-replication"256) "no"257) "tls-session-cache-size"258) "20480"259)
> > "tls-session-cache-timeout"260) "300"261) "tls-session-caching"262)
> > "yes"263) "txn-context-enabled"264) "no"265) "unixsocket"266) ""267)
> > "unixsocketperm"268) "777"269) "use-rsid-psync"270) "no"271) "workers"272)
> > "8"
> > > >> Vào ngày thg 4 18, 2025 5:49 CH, từ Hulk <hu...@gmail.com>Hi
> > Quan,
> > > >>
> > > >> Thanks for your information, could you please provide the output by
> > running
> > > >> the command in redis-cli `config get *`?
> > > >>
> > > >> On Fri, 18 Apr 2025 at 18:24, Hong Quan TRAN <hq...@linagora.com>
> > wrote:
> > > >>
> > > >> > Hi the Kvrocks community,
> > > >> > We would like to update you our progress on the integration topic.
> > > >> > We have started to migrate a Redis deployment to Kvrocks for a
> > week. The
> > > >> > Kvrocks is running stable I would say. However, we have concern
> > after
> > > >> > observing the Kvrocks's memory comsuption as it keeps slowly
> > consume more
> > > >> > memory without releasing them back to the OS. At first the memory
> > usage was
> > > >> > 500MB, then it is up to 700MB, 1GB, 1.3GB...
> > > >> > Please note that we already tuning the memory usage using the
> > > >> > recommendations at github.com/apache/kvrocks/discussions/2767.
> > > >> > ```kvrocks.conflog-level
> > > >> > <github.com/apache/kvrocks/discussions/2767.kvrocks.conflog-level>
> > > >> > infoport 6379rocksdb.max_total_wal_size
> > 256rocksdb.max_write_buffer_number
> > > >> > 2rocksdb.write_buffer_size 32rocksdb.block_cache_size 256```
> > > >> > We expect the memory consumption would be low and stable with this
> > applied
> > > >> > configuration. However, it keeps slowly increasing, which makes us
> > worry.
> > > >> > Our memory stats so far:
> > > >> > ```docker statsCONTAINER ID NAME CPU % MEM USAGE /
> > > >> > LIMIT MEM % NET I/O BLOCK I/O PIDS561e141f9b72
> > > >> > kvrocks 0.14% 1.292GiB / 14.3GiB 9.03% 4.77GB /
> > > >> > 1.51GB 270kB / 8.08GB 19```
> > > >> > ```kvrocks@561e141f9b72:/$ redis-cli -p 6379 info memory#
> > > >> >
> > Memoryused_memory_rss:983244800used_memory_rss_human:937.70Mused_memory_lua:59392used_memory_lua_human:58.00Kused_memory_startup:63860736```
> > > >> > FYI we are using Kvrocks version 2.11.1.
> > > >> > Could you help provide insights and advice on the Kvrocks memory
> > > >> > management topic? How to limit the Kvrocks memory usage effectively?
> > > >> >
> > > >> > Best regards,
> > > >> > Quan
> > > >> >
> > > >> >
> > > >>
> > > >> --
> > > >> Best Regards,
> > > >> - *Hulk Lin*
> > > >
> > > >
> > > >
> > > > --
> > > > Best Regards,
> > > > - Hulk Lin
> > >
> > >
> > >
> > > --
> > > Best Regards,
> > > - Hulk Lin
> > >
> >
> >
> > --
> > Best Regards,
> > - Hulk Lin
> >
>
>
> --
> Best Regards,
> - *Hulk Lin*
--
Best Regards,
- Hulk Lin
Re: Apache James meets KVRocks
Posted by Hong Quan TRAN <hq...@linagora.com>.
Hi Hulk,
I’d like to share some results from an experiment I ran related to Jemalloc memory tuning in Kvrocks, which may be helpful to the gradual memory growth over time.I read a Apache Doris jemalloc analysis article, where they seems to claim more or the the same issue as our. Jemalloc retained memory aggressively by default, leading to high memory usage that the OS could not easily reclaim. They suggest to restart Apache Doris after sometime to reclaim the memory, or use the retain:false Jemalloc option to force Jemalloc return the unused memory to OS.
I tried the option with Kvrocks using the environment variable MALLOC_CONF=retain:false.
After applying retain=false and stress tests Kvrocks using redis-benchmark a few dozen times, I observed:
- Docker stats memory: consistently under 800MB (was more than 1.3GB if retain:true)
- Kvrocks-reported RSS memory: consistently under 500MB
- When running other memory-intensive processes, Kvrocks memory usage dropped by ~150MB, suggesting part of the usage was reclaimable cache.
This aligns with Jemalloc behavior where disabling retain makes memory more readily returnable to the OS.
Maybe the gradual memory growth issue is more a Jemalloc's nature than Kvrocks issue itself?
Regarding performance metrics, I did not deeply dig into it yet. A quick benchmark on my side showed no significant difference between the two settings.
Please let me know your thoughts on this matter.
Quan
On May 3, 2025 9:10 PM, from Hulk <hu...@gmail.com>Hi Quan,
I find a gap in the memory usage in Docker stats, which is about double
that of Kvrocks's RSS.
You might have a check as well using the command `info memory` via
redis-cli.
Not sure if it's caused by the different algorithm when counting the memory
usage.
*docker stats a68a50cf4a82*
a68a50cf4a82 tender_jackson 143.39% 635MiB / 14.63GiB 4.24%
22.8GB / 4.7GB 844kB / 21GB 18
*hulk@instance-20241021-041032:~$ redis-cli -p 6666 info memory*
# Memory
used_memory_rss:247369728
used_memory_rss_human:235.91M
used_memory_lua:319488
used_memory_lua_human:312.00K
used_memory_startup:33468416
On Sat, 3 May 2025 at 19:02, hulk <hu...@gmail.com> wrote:
> Hi, Quan
>
> Thanks for your detailed information. I can reproduce it on my side. I
> ran the benchmark a few times, and its memory is around the same(about
> 1.3GiB),
> but I haven't dived deep into the Jemalloc stats yet.
>
> On Fri, 2 May 2025 at 17:31, Hong Quan TRAN <hq...@linagora.com> wrote:
> >
> > Hi Hulk,
> >
> > Sorry for the late reply (we were on holiday).
> >
> > > I have no clue about why it's up to 1.3 GiB, will try to reproduce
> > this on my side and analyze the memory in Jemalloc.
> >
> > We use Kvrocks as Rspamd's bayes training database. I would attach the
> Kvrocks used command stats for more details.
> >
> > BTW on my side, I tried to run the redis-benchmark tool against Kvrocks
> (still the same configuration). After a dozen times running
> `redis-benchmark -n 2000000 -c 50` command, Kvrocks memory is upto around
> 1.1GB and does not seems to be released automatically after a while.
> >
> > I hope this would help you ease reproducing the issue. I would be happy
> to do more diagnostic and share the output to you if needed.
> >
> > Regards,
> >
> > Quan
> >
> >
> > On Apr 27, 2025 4:27 PM, from Hulk <hu...@gmail.com>
> >
> > Hi Benoit, Quan
> >
> > Sorry for the late reply. I took a look at your configuration, and it
> > all looks good.
> >
> > I have no clue about why it's up to 1.3 GiB, will try to reproduce
> > this on my side and analyze the memory in Jemalloc.
> >
> > Will update you once we have any progress.
> >
> > On Mon, 21 Apr 2025 at 11:00, hulk <hu...@gmail.com> wrote:
> > >
> > > Thanks all for taking care of this, will have a look.
> > >
> > > On Mon, 21 Apr 2025 at 10:56, Hong Quan TRAN <hq...@linagora.com>
> wrote:
> > >>
> > >> Hi,
> > >>
> > >> Somehow the format is still bad (although I tested sending it to my
> personal mail).
> > >>
> > >> I would send the Kvrocks config output via an attachment this time.
> > >>
> > >> Quan
> > >>
> > >> Vào ngày thg 4 21, 2025 9:39 SA, từ Hong Quan Tran <
> hqtran@linagora.com>
> > >>
> > >> Hi everyone,
> > >> Sorry for the bad formatting output.
> > >> I resend the config output. I hope it is fine this time.
> > >>
> > >> kvrocks@561e141f9b72:/$ redis-cli -p 6379 CONFIG GET '*' 1)
> "auto-resize-block-and-sst" 2) "yes" 3) "backup-dir" 4)
> "/var/lib/kvrocks/backup" 5) "bgsave-cron" 6) "" 7) "bind" 8) "0.0.0.0" 9)
> "cluster-enabled" 10) "no" 11) "compact-cron" 12) "" 13)
> "compaction-checker-cron" 14) "" 15) "compaction-checker-range" 16) "" 17)
> "daemonize" 18) "no" 19) "db-name" 20) "change.me.db" 21)
> "dbsize-scan-cron" 22) "" 23) "dir" 24) "/var/lib/kvrocks" 25)
> "force-compact-file-age" 26) "172800" 27)
> "force-compact-file-min-deleted-percentage" 28) "10" 29)
> "fullsync-recv-file-delay" 30) "0" 31) "histogram-bucket-boundaries" 32) ""
> 33) "json-max-nesting-depth" 34) "1024" 35) "json-storage-format" 36)
> "json" 37) "log-dir" 38) "/var/lib/kvrocks" 39) "log-level" 40) "info" 41)
> "log-retention-days" 42) "-1" 43) "master-use-repl-port" 44) "no" 45)
> "masterauth" 46) "" 47) "max-backup-keep-hours" 48) "0" 49)
> "max-backup-to-keep" 50) "1" 51) "max-bitmap-to-string-mb" 52) "16" 53)
> "max-db-size" 54) "0" 55) "max-io-mb" 56) "0" 57) "max-replication-mb" 58)
> "0" 59) "maxclients" 60) "10240" 61) "migrate-batch-rate-limit-mb" 62) "16"
> 63) "migrate-batch-size-kb" 64) "16" 65) "migrate-pipeline-size" 66) "16"
> 67) "migrate-sequence-gap" 68) "10000" 69) "migrate-speed" 70) "4096" 71)
> "migrate-type" 72) "redis-command" 73) "persist-cluster-nodes-enabled" 74)
> "yes" 75) "pidfile" 76) "/var/run/kvrocks/kvrocks.pid" 77) "port" 78)
> "6379" 79) "profiling-sample-commands" 80) "" 81) "profiling-sample-ratio"
> 82) "0" 83) "profiling-sample-record-max-len" 84) "256" 85)
> "profiling-sample-record-threshold-ms" 86) "100" 87) "proto-max-bulk-len"
> 88) "536870912" 89) "purge-backup-on-fullsync" 90) "no" 91)
> "redis-cursor-compatible" 92) "yes" 93) "repl-namespace-enabled" 94) "no"
> 95) "replica-announce-ip" 96) "" 97) "replica-announce-port" 98) "0" 99)
> "replicaof"100) ""101) "replication-connect-timeout-ms"102) "3100"103)
> "replication-recv-timeout-ms"104) "3200"105) "requirepass"106) ""107)
> "resp3-enabled"108) "no"109) "rocksdb.avoid_unnecessary_blocking_io"110)
> "yes"111) "rocksdb.blob_file_size"112) "268435456"113)
> "rocksdb.blob_garbage_collection_age_cutoff"114) "25"115)
> "rocksdb.block_cache_size"116) "256"117) "rocksdb.block_cache_type"118)
> "lru"119) "rocksdb.block_size"120) "16384"121)
> "rocksdb.cache_index_and_filter_blocks"122) "yes"123)
> "rocksdb.compaction_readahead_size"124) "2097152"125)
> "rocksdb.compression"126) "no"127) "rocksdb.compression_level"128)
> "32767"129) "rocksdb.compression_start_level"130) "2"131)
> "rocksdb.delayed_write_rate"132) "0"133)
> "rocksdb.disable_auto_compactions"134) "no"135)
> "rocksdb.dump_malloc_stats"136) "yes"137) "rocksdb.enable_blob_files"138)
> "no"139) "rocksdb.enable_blob_garbage_collection"140) "yes"141)
> "rocksdb.enable_pipelined_write"142) "no"143)
> "rocksdb.level0_file_num_compaction_trigger"144) "4"145)
> "rocksdb.level0_slowdown_writes_trigger"146) "20"147)
> "rocksdb.level0_stop_writes_trigger"148) "40"149)
> "rocksdb.level_compaction_dynamic_level_bytes"150) "no"151)
> "rocksdb.max_background_compactions"152) "2"153)
> "rocksdb.max_background_flushes"154) "2"155)
> "rocksdb.max_background_jobs"156) "4"157)
> "rocksdb.max_bytes_for_level_base"158) "268435456"159)
> "rocksdb.max_bytes_for_level_multiplier"160) "10"161)
> "rocksdb.max_open_files"162) "8096"163) "rocksdb.max_subcompactions"164)
> "2"165) "rocksdb.max_total_wal_size"166) "256"167)
> "rocksdb.max_write_buffer_number"168) "2"169)
> "rocksdb.metadata_block_cache_size"170) "2048"171)
> "rocksdb.min_blob_size"172) "4096"173) "rocksdb.partition_filters"174)
> "yes"175) "rocksdb.rate_limiter_auto_tuned"176) "yes"177)
> "rocksdb.read_options.async_io"178) "yes"179) "rocksdb.row_cache_size"180)
> "0"181) "rocksdb.share_metadata_and_subkey_block_cache"182) "yes"183)
> "rocksdb.stats_dump_period_sec"184) "0"185)
> "rocksdb.subkey_block_cache_size"186) "2048"187)
> "rocksdb.target_file_size_base"188) "128"189) "rocksdb.wal_compression"190)
> "no"191) "rocksdb.wal_size_limit_mb"192) "16384"193)
> "rocksdb.wal_ttl_seconds"194) "10800"195) "rocksdb.write_buffer_size"196)
> "32"197) "rocksdb.write_options.disable_wal"198) "no"199)
> "rocksdb.write_options.low_pri"200) "no"201)
> "rocksdb.write_options.memtable_insert_hint_per_batch"202) "no"203)
> "rocksdb.write_options.no_slowdown"204) "no"205)
> "rocksdb.write_options.sync"206) "no"207)
> "rocksdb.write_options.write_batch_max_bytes"208) "0"209)
> "skip-block-cache-deallocation-on-close"210) "no"211)
> "slave-empty-db-before-fullsync"212) "no"213) "slave-priority"214)
> "100"215) "slave-read-only"216) "yes"217) "slave-serve-stale-data"218)
> "yes"219) "slaveof"220) ""221) "slowlog-log-slower-than"222) "200000"223)
> "slowlog-max-len"224) "128"225) "socket-fd"226) "-1"227) "supervised"228)
> "no"229) "tcp-backlog"230) "511"231) "timeout"232) "0"233)
> "tls-auth-clients"234) ""235) "tls-ca-cert-dir"236) ""237)
> "tls-ca-cert-file"238) ""239) "tls-cert-file"240) ""241) "tls-ciphers"242)
> ""243) "tls-ciphersuites"244) ""245) "tls-key-file"246) ""247)
> "tls-key-file-pass"248) ""249) "tls-port"250) "0"251)
> "tls-prefer-server-ciphers"252) "no"253) "tls-protocols"254) ""255)
> "tls-replication"256) "no"257) "tls-session-cache-size"258) "20480"259)
> "tls-session-cache-timeout"260) "300"261) "tls-session-caching"262)
> "yes"263) "txn-context-enabled"264) "no"265) "unixsocket"266) ""267)
> "unixsocketperm"268) "777"269) "use-rsid-psync"270) "no"271) "workers"272)
> "8"
> > >> Quan
> > >> Vào ngày thg 4 21, 2025 1:52 SA, từ Benoit Tellier <
> btellier@linagora.com>Careful
> > >>
> > >> Formating sucks
> > >>
> > >> Maybe resent it as a plain/text version to avoid this pitfall?
> > >>
> > >> --
> > >>
> > >> Best regards,
> > >>
> > >> Benoit TELLIER
> > >>
> > >> General manager of Linagora VIETNAM.
> > >> Product owner for Team-Mail product.
> > >> Chairman of the Apache James project.
> > >>
> > >> Mail: btellier@linagora.com
> > >> Tel: (0033) 6 77 26 04 58 (WhatsApp, Signal)
> > >>
> > >>
> > >> On Apr 20, 2025 3:43 PM, from Hong Quan Tran <hq...@linagora.com>Hi
> Hulk,
> > >> > Thanks for your information, could you please provide the output by
> running the command in redis-cli `config get *`?
> > >> Please see the output below.
> > >>
> > >> kvrocks@561e141f9b72:/$ redis-cli -p 6379 CONFIG GET '*' 1)
> "auto-resize-block-and-sst" 2) "yes" 3) "backup-dir" 4)
> "/var/lib/kvrocks/backup" 5) "bgsave-cron" 6) "" 7) "bind" 8) "0.0.0.0" 9)
> "cluster-enabled" 10) "no" 11) "compact-cron" 12) "" 13)
> "compaction-checker-cron" 14) "" 15) "compaction-checker-range" 16) "" 17)
> "daemonize" 18) "no" 19) "db-name" 20) "change.me.db" 21)
> "dbsize-scan-cron" 22) "" 23) "dir" 24) "/var/lib/kvrocks" 25)
> "force-compact-file-age" 26) "172800" 27)
> "force-compact-file-min-deleted-percentage" 28) "10" 29)
> "fullsync-recv-file-delay" 30) "0" 31) "histogram-bucket-boundaries" 32) ""
> 33) "json-max-nesting-depth" 34) "1024" 35) "json-storage-format" 36)
> "json" 37) "log-dir" 38) "/var/lib/kvrocks" 39) "log-level" 40) "info" 41)
> "log-retention-days" 42) "-1" 43) "master-use-repl-port" 44) "no" 45)
> "masterauth" 46) "" 47) "max-backup-keep-hours" 48) "0" 49)
> "max-backup-to-keep" 50) "1" 51) "max-bitmap-to-string-mb" 52) "16" 53)
> "max-db-size" 54) "0" 55) "max-io-mb" 56) "0" 57) "max-replication-mb" 58)
> "0" 59) "maxclients" 60) "10240" 61) "migrate-batch-rate-limit-mb" 62) "16"
> 63) "migrate-batch-size-kb" 64) "16" 65) "migrate-pipeline-size" 66) "16"
> 67) "migrate-sequence-gap" 68) "10000" 69) "migrate-speed" 70) "4096" 71)
> "migrate-type" 72) "redis-command" 73) "persist-cluster-nodes-enabled" 74)
> "yes" 75) "pidfile" 76) "/var/run/kvrocks/kvrocks.pid" 77) "port" 78)
> "6379" 79) "profiling-sample-commands" 80) "" 81) "profiling-sample-ratio"
> 82) "0" 83) "profiling-sample-record-max-len" 84) "256" 85)
> "profiling-sample-record-threshold-ms" 86) "100" 87) "proto-max-bulk-len"
> 88) "536870912" 89) "purge-backup-on-fullsync" 90) "no" 91)
> "redis-cursor-compatible" 92) "yes" 93) "repl-namespace-enabled" 94) "no"
> 95) "replica-announce-ip" 96) "" 97) "replica-announce-port" 98) "0" 99)
> "replicaof"100) ""101) "replication-connect-timeout-ms"102) "3100"103)
> "replication-recv-timeout-ms"104) "3200"105) "requirepass"106) ""107)
> "resp3-enabled"108) "no"109) "rocksdb.avoid_unnecessary_blocking_io"110)
> "yes"111) "rocksdb.blob_file_size"112) "268435456"113)
> "rocksdb.blob_garbage_collection_age_cutoff"114) "25"115)
> "rocksdb.block_cache_size"116) "256"117) "rocksdb.block_cache_type"118)
> "lru"119) "rocksdb.block_size"120) "16384"121)
> "rocksdb.cache_index_and_filter_blocks"122) "yes"123)
> "rocksdb.compaction_readahead_size"124) "2097152"125)
> "rocksdb.compression"126) "no"127) "rocksdb.compression_level"128)
> "32767"129) "rocksdb.compression_start_level"130) "2"131)
> "rocksdb.delayed_write_rate"132) "0"133)
> "rocksdb.disable_auto_compactions"134) "no"135)
> "rocksdb.dump_malloc_stats"136) "yes"137) "rocksdb.enable_blob_files"138)
> "no"139) "rocksdb.enable_blob_garbage_collection"140) "yes"141)
> "rocksdb.enable_pipelined_write"142) "no"143)
> "rocksdb.level0_file_num_compaction_trigger"144) "4"145)
> "rocksdb.level0_slowdown_writes_trigger"146) "20"147)
> "rocksdb.level0_stop_writes_trigger"148) "40"149)
> "rocksdb.level_compaction_dynamic_level_bytes"150) "no"151)
> "rocksdb.max_background_compactions"152) "2"153)
> "rocksdb.max_background_flushes"154) "2"155)
> "rocksdb.max_background_jobs"156) "4"157)
> "rocksdb.max_bytes_for_level_base"158) "268435456"159)
> "rocksdb.max_bytes_for_level_multiplier"160) "10"161)
> "rocksdb.max_open_files"162) "8096"163) "rocksdb.max_subcompactions"164)
> "2"165) "rocksdb.max_total_wal_size"166) "256"167)
> "rocksdb.max_write_buffer_number"168) "2"169)
> "rocksdb.metadata_block_cache_size"170) "2048"171)
> "rocksdb.min_blob_size"172) "4096"173) "rocksdb.partition_filters"174)
> "yes"175) "rocksdb.rate_limiter_auto_tuned"176) "yes"177)
> "rocksdb.read_options.async_io"178) "yes"179) "rocksdb.row_cache_size"180)
> "0"181) "rocksdb.share_metadata_and_subkey_block_cache"182) "yes"183)
> "rocksdb.stats_dump_period_sec"184) "0"185)
> "rocksdb.subkey_block_cache_size"186) "2048"187)
> "rocksdb.target_file_size_base"188) "128"189) "rocksdb.wal_compression"190)
> "no"191) "rocksdb.wal_size_limit_mb"192) "16384"193)
> "rocksdb.wal_ttl_seconds"194) "10800"195) "rocksdb.write_buffer_size"196)
> "32"197) "rocksdb.write_options.disable_wal"198) "no"199)
> "rocksdb.write_options.low_pri"200) "no"201)
> "rocksdb.write_options.memtable_insert_hint_per_batch"202) "no"203)
> "rocksdb.write_options.no_slowdown"204) "no"205)
> "rocksdb.write_options.sync"206) "no"207)
> "rocksdb.write_options.write_batch_max_bytes"208) "0"209)
> "skip-block-cache-deallocation-on-close"210) "no"211)
> "slave-empty-db-before-fullsync"212) "no"213) "slave-priority"214)
> "100"215) "slave-read-only"216) "yes"217) "slave-serve-stale-data"218)
> "yes"219) "slaveof"220) ""221) "slowlog-log-slower-than"222) "200000"223)
> "slowlog-max-len"224) "128"225) "socket-fd"226) "-1"227) "supervised"228)
> "no"229) "tcp-backlog"230) "511"231) "timeout"232) "0"233)
> "tls-auth-clients"234) ""235) "tls-ca-cert-dir"236) ""237)
> "tls-ca-cert-file"238) ""239) "tls-cert-file"240) ""241) "tls-ciphers"242)
> ""243) "tls-ciphersuites"244) ""245) "tls-key-file"246) ""247)
> "tls-key-file-pass"248) ""249) "tls-port"250) "0"251)
> "tls-prefer-server-ciphers"252) "no"253) "tls-protocols"254) ""255)
> "tls-replication"256) "no"257) "tls-session-cache-size"258) "20480"259)
> "tls-session-cache-timeout"260) "300"261) "tls-session-caching"262)
> "yes"263) "txn-context-enabled"264) "no"265) "unixsocket"266) ""267)
> "unixsocketperm"268) "777"269) "use-rsid-psync"270) "no"271) "workers"272)
> "8"
> > >> Vào ngày thg 4 18, 2025 5:49 CH, từ Hulk <hu...@gmail.com>Hi
> Quan,
> > >>
> > >> Thanks for your information, could you please provide the output by
> running
> > >> the command in redis-cli `config get *`?
> > >>
> > >> On Fri, 18 Apr 2025 at 18:24, Hong Quan TRAN <hq...@linagora.com>
> wrote:
> > >>
> > >> > Hi the Kvrocks community,
> > >> > We would like to update you our progress on the integration topic.
> > >> > We have started to migrate a Redis deployment to Kvrocks for a
> week. The
> > >> > Kvrocks is running stable I would say. However, we have concern
> after
> > >> > observing the Kvrocks's memory comsuption as it keeps slowly
> consume more
> > >> > memory without releasing them back to the OS. At first the memory
> usage was
> > >> > 500MB, then it is up to 700MB, 1GB, 1.3GB...
> > >> > Please note that we already tuning the memory usage using the
> > >> > recommendations at github.com/apache/kvrocks/discussions/2767.
> > >> > ```kvrocks.conflog-level
> > >> > <github.com/apache/kvrocks/discussions/2767.kvrocks.conflog-level>
> > >> > infoport 6379rocksdb.max_total_wal_size
> 256rocksdb.max_write_buffer_number
> > >> > 2rocksdb.write_buffer_size 32rocksdb.block_cache_size 256```
> > >> > We expect the memory consumption would be low and stable with this
> applied
> > >> > configuration. However, it keeps slowly increasing, which makes us
> worry.
> > >> > Our memory stats so far:
> > >> > ```docker statsCONTAINER ID NAME CPU % MEM USAGE /
> > >> > LIMIT MEM % NET I/O BLOCK I/O PIDS561e141f9b72
> > >> > kvrocks 0.14% 1.292GiB / 14.3GiB 9.03% 4.77GB /
> > >> > 1.51GB 270kB / 8.08GB 19```
> > >> > ```kvrocks@561e141f9b72:/$ redis-cli -p 6379 info memory#
> > >> >
> Memoryused_memory_rss:983244800used_memory_rss_human:937.70Mused_memory_lua:59392used_memory_lua_human:58.00Kused_memory_startup:63860736```
> > >> > FYI we are using Kvrocks version 2.11.1.
> > >> > Could you help provide insights and advice on the Kvrocks memory
> > >> > management topic? How to limit the Kvrocks memory usage effectively?
> > >> >
> > >> > Best regards,
> > >> > Quan
> > >> >
> > >> >
> > >>
> > >> --
> > >> Best Regards,
> > >> - *Hulk Lin*
> > >
> > >
> > >
> > > --
> > > Best Regards,
> > > - Hulk Lin
> >
> >
> >
> > --
> > Best Regards,
> > - Hulk Lin
> >
>
>
> --
> Best Regards,
> - Hulk Lin
>
--
Best Regards,
- *Hulk Lin*
Re: Apache James meets KVRocks
Posted by hulk <hu...@gmail.com>.
Hi Quan,
I find a gap in the memory usage in Docker stats, which is about double
that of Kvrocks's RSS.
You might have a check as well using the command `info memory` via
redis-cli.
Not sure if it's caused by the different algorithm when counting the memory
usage.
*docker stats a68a50cf4a82*
a68a50cf4a82 tender_jackson 143.39% 635MiB / 14.63GiB 4.24%
22.8GB / 4.7GB 844kB / 21GB 18
*hulk@instance-20241021-041032:~$ redis-cli -p 6666 info memory*
# Memory
used_memory_rss:247369728
used_memory_rss_human:235.91M
used_memory_lua:319488
used_memory_lua_human:312.00K
used_memory_startup:33468416
On Sat, 3 May 2025 at 19:02, hulk <hu...@gmail.com> wrote:
> Hi, Quan
>
> Thanks for your detailed information. I can reproduce it on my side. I
> ran the benchmark a few times, and its memory is around the same(about
> 1.3GiB),
> but I haven't dived deep into the Jemalloc stats yet.
>
> On Fri, 2 May 2025 at 17:31, Hong Quan TRAN <hq...@linagora.com> wrote:
> >
> > Hi Hulk,
> >
> > Sorry for the late reply (we were on holiday).
> >
> > > I have no clue about why it's up to 1.3 GiB, will try to reproduce
> > this on my side and analyze the memory in Jemalloc.
> >
> > We use Kvrocks as Rspamd's bayes training database. I would attach the
> Kvrocks used command stats for more details.
> >
> > BTW on my side, I tried to run the redis-benchmark tool against Kvrocks
> (still the same configuration). After a dozen times running
> `redis-benchmark -n 2000000 -c 50` command, Kvrocks memory is upto around
> 1.1GB and does not seems to be released automatically after a while.
> >
> > I hope this would help you ease reproducing the issue. I would be happy
> to do more diagnostic and share the output to you if needed.
> >
> > Regards,
> >
> > Quan
> >
> >
> > On Apr 27, 2025 4:27 PM, from Hulk <hu...@gmail.com>
> >
> > Hi Benoit, Quan
> >
> > Sorry for the late reply. I took a look at your configuration, and it
> > all looks good.
> >
> > I have no clue about why it's up to 1.3 GiB, will try to reproduce
> > this on my side and analyze the memory in Jemalloc.
> >
> > Will update you once we have any progress.
> >
> > On Mon, 21 Apr 2025 at 11:00, hulk <hu...@gmail.com> wrote:
> > >
> > > Thanks all for taking care of this, will have a look.
> > >
> > > On Mon, 21 Apr 2025 at 10:56, Hong Quan TRAN <hq...@linagora.com>
> wrote:
> > >>
> > >> Hi,
> > >>
> > >> Somehow the format is still bad (although I tested sending it to my
> personal mail).
> > >>
> > >> I would send the Kvrocks config output via an attachment this time.
> > >>
> > >> Quan
> > >>
> > >> Vào ngày thg 4 21, 2025 9:39 SA, từ Hong Quan Tran <
> hqtran@linagora.com>
> > >>
> > >> Hi everyone,
> > >> Sorry for the bad formatting output.
> > >> I resend the config output. I hope it is fine this time.
> > >>
> > >> kvrocks@561e141f9b72:/$ redis-cli -p 6379 CONFIG GET '*' 1)
> "auto-resize-block-and-sst" 2) "yes" 3) "backup-dir" 4)
> "/var/lib/kvrocks/backup" 5) "bgsave-cron" 6) "" 7) "bind" 8) "0.0.0.0" 9)
> "cluster-enabled" 10) "no" 11) "compact-cron" 12) "" 13)
> "compaction-checker-cron" 14) "" 15) "compaction-checker-range" 16) "" 17)
> "daemonize" 18) "no" 19) "db-name" 20) "change.me.db" 21)
> "dbsize-scan-cron" 22) "" 23) "dir" 24) "/var/lib/kvrocks" 25)
> "force-compact-file-age" 26) "172800" 27)
> "force-compact-file-min-deleted-percentage" 28) "10" 29)
> "fullsync-recv-file-delay" 30) "0" 31) "histogram-bucket-boundaries" 32) ""
> 33) "json-max-nesting-depth" 34) "1024" 35) "json-storage-format" 36)
> "json" 37) "log-dir" 38) "/var/lib/kvrocks" 39) "log-level" 40) "info" 41)
> "log-retention-days" 42) "-1" 43) "master-use-repl-port" 44) "no" 45)
> "masterauth" 46) "" 47) "max-backup-keep-hours" 48) "0" 49)
> "max-backup-to-keep" 50) "1" 51) "max-bitmap-to-string-mb" 52) "16" 53)
> "max-db-size" 54) "0" 55) "max-io-mb" 56) "0" 57) "max-replication-mb" 58)
> "0" 59) "maxclients" 60) "10240" 61) "migrate-batch-rate-limit-mb" 62) "16"
> 63) "migrate-batch-size-kb" 64) "16" 65) "migrate-pipeline-size" 66) "16"
> 67) "migrate-sequence-gap" 68) "10000" 69) "migrate-speed" 70) "4096" 71)
> "migrate-type" 72) "redis-command" 73) "persist-cluster-nodes-enabled" 74)
> "yes" 75) "pidfile" 76) "/var/run/kvrocks/kvrocks.pid" 77) "port" 78)
> "6379" 79) "profiling-sample-commands" 80) "" 81) "profiling-sample-ratio"
> 82) "0" 83) "profiling-sample-record-max-len" 84) "256" 85)
> "profiling-sample-record-threshold-ms" 86) "100" 87) "proto-max-bulk-len"
> 88) "536870912" 89) "purge-backup-on-fullsync" 90) "no" 91)
> "redis-cursor-compatible" 92) "yes" 93) "repl-namespace-enabled" 94) "no"
> 95) "replica-announce-ip" 96) "" 97) "replica-announce-port" 98) "0" 99)
> "replicaof"100) ""101) "replication-connect-timeout-ms"102) "3100"103)
> "replication-recv-timeout-ms"104) "3200"105) "requirepass"106) ""107)
> "resp3-enabled"108) "no"109) "rocksdb.avoid_unnecessary_blocking_io"110)
> "yes"111) "rocksdb.blob_file_size"112) "268435456"113)
> "rocksdb.blob_garbage_collection_age_cutoff"114) "25"115)
> "rocksdb.block_cache_size"116) "256"117) "rocksdb.block_cache_type"118)
> "lru"119) "rocksdb.block_size"120) "16384"121)
> "rocksdb.cache_index_and_filter_blocks"122) "yes"123)
> "rocksdb.compaction_readahead_size"124) "2097152"125)
> "rocksdb.compression"126) "no"127) "rocksdb.compression_level"128)
> "32767"129) "rocksdb.compression_start_level"130) "2"131)
> "rocksdb.delayed_write_rate"132) "0"133)
> "rocksdb.disable_auto_compactions"134) "no"135)
> "rocksdb.dump_malloc_stats"136) "yes"137) "rocksdb.enable_blob_files"138)
> "no"139) "rocksdb.enable_blob_garbage_collection"140) "yes"141)
> "rocksdb.enable_pipelined_write"142) "no"143)
> "rocksdb.level0_file_num_compaction_trigger"144) "4"145)
> "rocksdb.level0_slowdown_writes_trigger"146) "20"147)
> "rocksdb.level0_stop_writes_trigger"148) "40"149)
> "rocksdb.level_compaction_dynamic_level_bytes"150) "no"151)
> "rocksdb.max_background_compactions"152) "2"153)
> "rocksdb.max_background_flushes"154) "2"155)
> "rocksdb.max_background_jobs"156) "4"157)
> "rocksdb.max_bytes_for_level_base"158) "268435456"159)
> "rocksdb.max_bytes_for_level_multiplier"160) "10"161)
> "rocksdb.max_open_files"162) "8096"163) "rocksdb.max_subcompactions"164)
> "2"165) "rocksdb.max_total_wal_size"166) "256"167)
> "rocksdb.max_write_buffer_number"168) "2"169)
> "rocksdb.metadata_block_cache_size"170) "2048"171)
> "rocksdb.min_blob_size"172) "4096"173) "rocksdb.partition_filters"174)
> "yes"175) "rocksdb.rate_limiter_auto_tuned"176) "yes"177)
> "rocksdb.read_options.async_io"178) "yes"179) "rocksdb.row_cache_size"180)
> "0"181) "rocksdb.share_metadata_and_subkey_block_cache"182) "yes"183)
> "rocksdb.stats_dump_period_sec"184) "0"185)
> "rocksdb.subkey_block_cache_size"186) "2048"187)
> "rocksdb.target_file_size_base"188) "128"189) "rocksdb.wal_compression"190)
> "no"191) "rocksdb.wal_size_limit_mb"192) "16384"193)
> "rocksdb.wal_ttl_seconds"194) "10800"195) "rocksdb.write_buffer_size"196)
> "32"197) "rocksdb.write_options.disable_wal"198) "no"199)
> "rocksdb.write_options.low_pri"200) "no"201)
> "rocksdb.write_options.memtable_insert_hint_per_batch"202) "no"203)
> "rocksdb.write_options.no_slowdown"204) "no"205)
> "rocksdb.write_options.sync"206) "no"207)
> "rocksdb.write_options.write_batch_max_bytes"208) "0"209)
> "skip-block-cache-deallocation-on-close"210) "no"211)
> "slave-empty-db-before-fullsync"212) "no"213) "slave-priority"214)
> "100"215) "slave-read-only"216) "yes"217) "slave-serve-stale-data"218)
> "yes"219) "slaveof"220) ""221) "slowlog-log-slower-than"222) "200000"223)
> "slowlog-max-len"224) "128"225) "socket-fd"226) "-1"227) "supervised"228)
> "no"229) "tcp-backlog"230) "511"231) "timeout"232) "0"233)
> "tls-auth-clients"234) ""235) "tls-ca-cert-dir"236) ""237)
> "tls-ca-cert-file"238) ""239) "tls-cert-file"240) ""241) "tls-ciphers"242)
> ""243) "tls-ciphersuites"244) ""245) "tls-key-file"246) ""247)
> "tls-key-file-pass"248) ""249) "tls-port"250) "0"251)
> "tls-prefer-server-ciphers"252) "no"253) "tls-protocols"254) ""255)
> "tls-replication"256) "no"257) "tls-session-cache-size"258) "20480"259)
> "tls-session-cache-timeout"260) "300"261) "tls-session-caching"262)
> "yes"263) "txn-context-enabled"264) "no"265) "unixsocket"266) ""267)
> "unixsocketperm"268) "777"269) "use-rsid-psync"270) "no"271) "workers"272)
> "8"
> > >> Quan
> > >> Vào ngày thg 4 21, 2025 1:52 SA, từ Benoit Tellier <
> btellier@linagora.com>Careful
> > >>
> > >> Formating sucks
> > >>
> > >> Maybe resent it as a plain/text version to avoid this pitfall?
> > >>
> > >> --
> > >>
> > >> Best regards,
> > >>
> > >> Benoit TELLIER
> > >>
> > >> General manager of Linagora VIETNAM.
> > >> Product owner for Team-Mail product.
> > >> Chairman of the Apache James project.
> > >>
> > >> Mail: btellier@linagora.com
> > >> Tel: (0033) 6 77 26 04 58 (WhatsApp, Signal)
> > >>
> > >>
> > >> On Apr 20, 2025 3:43 PM, from Hong Quan Tran <hq...@linagora.com>Hi
> Hulk,
> > >> > Thanks for your information, could you please provide the output by
> running the command in redis-cli `config get *`?
> > >> Please see the output below.
> > >>
> > >> kvrocks@561e141f9b72:/$ redis-cli -p 6379 CONFIG GET '*' 1)
> "auto-resize-block-and-sst" 2) "yes" 3) "backup-dir" 4)
> "/var/lib/kvrocks/backup" 5) "bgsave-cron" 6) "" 7) "bind" 8) "0.0.0.0" 9)
> "cluster-enabled" 10) "no" 11) "compact-cron" 12) "" 13)
> "compaction-checker-cron" 14) "" 15) "compaction-checker-range" 16) "" 17)
> "daemonize" 18) "no" 19) "db-name" 20) "change.me.db" 21)
> "dbsize-scan-cron" 22) "" 23) "dir" 24) "/var/lib/kvrocks" 25)
> "force-compact-file-age" 26) "172800" 27)
> "force-compact-file-min-deleted-percentage" 28) "10" 29)
> "fullsync-recv-file-delay" 30) "0" 31) "histogram-bucket-boundaries" 32) ""
> 33) "json-max-nesting-depth" 34) "1024" 35) "json-storage-format" 36)
> "json" 37) "log-dir" 38) "/var/lib/kvrocks" 39) "log-level" 40) "info" 41)
> "log-retention-days" 42) "-1" 43) "master-use-repl-port" 44) "no" 45)
> "masterauth" 46) "" 47) "max-backup-keep-hours" 48) "0" 49)
> "max-backup-to-keep" 50) "1" 51) "max-bitmap-to-string-mb" 52) "16" 53)
> "max-db-size" 54) "0" 55) "max-io-mb" 56) "0" 57) "max-replication-mb" 58)
> "0" 59) "maxclients" 60) "10240" 61) "migrate-batch-rate-limit-mb" 62) "16"
> 63) "migrate-batch-size-kb" 64) "16" 65) "migrate-pipeline-size" 66) "16"
> 67) "migrate-sequence-gap" 68) "10000" 69) "migrate-speed" 70) "4096" 71)
> "migrate-type" 72) "redis-command" 73) "persist-cluster-nodes-enabled" 74)
> "yes" 75) "pidfile" 76) "/var/run/kvrocks/kvrocks.pid" 77) "port" 78)
> "6379" 79) "profiling-sample-commands" 80) "" 81) "profiling-sample-ratio"
> 82) "0" 83) "profiling-sample-record-max-len" 84) "256" 85)
> "profiling-sample-record-threshold-ms" 86) "100" 87) "proto-max-bulk-len"
> 88) "536870912" 89) "purge-backup-on-fullsync" 90) "no" 91)
> "redis-cursor-compatible" 92) "yes" 93) "repl-namespace-enabled" 94) "no"
> 95) "replica-announce-ip" 96) "" 97) "replica-announce-port" 98) "0" 99)
> "replicaof"100) ""101) "replication-connect-timeout-ms"102) "3100"103)
> "replication-recv-timeout-ms"104) "3200"105) "requirepass"106) ""107)
> "resp3-enabled"108) "no"109) "rocksdb.avoid_unnecessary_blocking_io"110)
> "yes"111) "rocksdb.blob_file_size"112) "268435456"113)
> "rocksdb.blob_garbage_collection_age_cutoff"114) "25"115)
> "rocksdb.block_cache_size"116) "256"117) "rocksdb.block_cache_type"118)
> "lru"119) "rocksdb.block_size"120) "16384"121)
> "rocksdb.cache_index_and_filter_blocks"122) "yes"123)
> "rocksdb.compaction_readahead_size"124) "2097152"125)
> "rocksdb.compression"126) "no"127) "rocksdb.compression_level"128)
> "32767"129) "rocksdb.compression_start_level"130) "2"131)
> "rocksdb.delayed_write_rate"132) "0"133)
> "rocksdb.disable_auto_compactions"134) "no"135)
> "rocksdb.dump_malloc_stats"136) "yes"137) "rocksdb.enable_blob_files"138)
> "no"139) "rocksdb.enable_blob_garbage_collection"140) "yes"141)
> "rocksdb.enable_pipelined_write"142) "no"143)
> "rocksdb.level0_file_num_compaction_trigger"144) "4"145)
> "rocksdb.level0_slowdown_writes_trigger"146) "20"147)
> "rocksdb.level0_stop_writes_trigger"148) "40"149)
> "rocksdb.level_compaction_dynamic_level_bytes"150) "no"151)
> "rocksdb.max_background_compactions"152) "2"153)
> "rocksdb.max_background_flushes"154) "2"155)
> "rocksdb.max_background_jobs"156) "4"157)
> "rocksdb.max_bytes_for_level_base"158) "268435456"159)
> "rocksdb.max_bytes_for_level_multiplier"160) "10"161)
> "rocksdb.max_open_files"162) "8096"163) "rocksdb.max_subcompactions"164)
> "2"165) "rocksdb.max_total_wal_size"166) "256"167)
> "rocksdb.max_write_buffer_number"168) "2"169)
> "rocksdb.metadata_block_cache_size"170) "2048"171)
> "rocksdb.min_blob_size"172) "4096"173) "rocksdb.partition_filters"174)
> "yes"175) "rocksdb.rate_limiter_auto_tuned"176) "yes"177)
> "rocksdb.read_options.async_io"178) "yes"179) "rocksdb.row_cache_size"180)
> "0"181) "rocksdb.share_metadata_and_subkey_block_cache"182) "yes"183)
> "rocksdb.stats_dump_period_sec"184) "0"185)
> "rocksdb.subkey_block_cache_size"186) "2048"187)
> "rocksdb.target_file_size_base"188) "128"189) "rocksdb.wal_compression"190)
> "no"191) "rocksdb.wal_size_limit_mb"192) "16384"193)
> "rocksdb.wal_ttl_seconds"194) "10800"195) "rocksdb.write_buffer_size"196)
> "32"197) "rocksdb.write_options.disable_wal"198) "no"199)
> "rocksdb.write_options.low_pri"200) "no"201)
> "rocksdb.write_options.memtable_insert_hint_per_batch"202) "no"203)
> "rocksdb.write_options.no_slowdown"204) "no"205)
> "rocksdb.write_options.sync"206) "no"207)
> "rocksdb.write_options.write_batch_max_bytes"208) "0"209)
> "skip-block-cache-deallocation-on-close"210) "no"211)
> "slave-empty-db-before-fullsync"212) "no"213) "slave-priority"214)
> "100"215) "slave-read-only"216) "yes"217) "slave-serve-stale-data"218)
> "yes"219) "slaveof"220) ""221) "slowlog-log-slower-than"222) "200000"223)
> "slowlog-max-len"224) "128"225) "socket-fd"226) "-1"227) "supervised"228)
> "no"229) "tcp-backlog"230) "511"231) "timeout"232) "0"233)
> "tls-auth-clients"234) ""235) "tls-ca-cert-dir"236) ""237)
> "tls-ca-cert-file"238) ""239) "tls-cert-file"240) ""241) "tls-ciphers"242)
> ""243) "tls-ciphersuites"244) ""245) "tls-key-file"246) ""247)
> "tls-key-file-pass"248) ""249) "tls-port"250) "0"251)
> "tls-prefer-server-ciphers"252) "no"253) "tls-protocols"254) ""255)
> "tls-replication"256) "no"257) "tls-session-cache-size"258) "20480"259)
> "tls-session-cache-timeout"260) "300"261) "tls-session-caching"262)
> "yes"263) "txn-context-enabled"264) "no"265) "unixsocket"266) ""267)
> "unixsocketperm"268) "777"269) "use-rsid-psync"270) "no"271) "workers"272)
> "8"
> > >> Vào ngày thg 4 18, 2025 5:49 CH, từ Hulk <hu...@gmail.com>Hi
> Quan,
> > >>
> > >> Thanks for your information, could you please provide the output by
> running
> > >> the command in redis-cli `config get *`?
> > >>
> > >> On Fri, 18 Apr 2025 at 18:24, Hong Quan TRAN <hq...@linagora.com>
> wrote:
> > >>
> > >> > Hi the Kvrocks community,
> > >> > We would like to update you our progress on the integration topic.
> > >> > We have started to migrate a Redis deployment to Kvrocks for a
> week. The
> > >> > Kvrocks is running stable I would say. However, we have concern
> after
> > >> > observing the Kvrocks's memory comsuption as it keeps slowly
> consume more
> > >> > memory without releasing them back to the OS. At first the memory
> usage was
> > >> > 500MB, then it is up to 700MB, 1GB, 1.3GB...
> > >> > Please note that we already tuning the memory usage using the
> > >> > recommendations at github.com/apache/kvrocks/discussions/2767.
> > >> > ```kvrocks.conflog-level
> > >> > <github.com/apache/kvrocks/discussions/2767.kvrocks.conflog-level>
> > >> > infoport 6379rocksdb.max_total_wal_size
> 256rocksdb.max_write_buffer_number
> > >> > 2rocksdb.write_buffer_size 32rocksdb.block_cache_size 256```
> > >> > We expect the memory consumption would be low and stable with this
> applied
> > >> > configuration. However, it keeps slowly increasing, which makes us
> worry.
> > >> > Our memory stats so far:
> > >> > ```docker statsCONTAINER ID NAME CPU % MEM USAGE /
> > >> > LIMIT MEM % NET I/O BLOCK I/O PIDS561e141f9b72
> > >> > kvrocks 0.14% 1.292GiB / 14.3GiB 9.03% 4.77GB /
> > >> > 1.51GB 270kB / 8.08GB 19```
> > >> > ```kvrocks@561e141f9b72:/$ redis-cli -p 6379 info memory#
> > >> >
> Memoryused_memory_rss:983244800used_memory_rss_human:937.70Mused_memory_lua:59392used_memory_lua_human:58.00Kused_memory_startup:63860736```
> > >> > FYI we are using Kvrocks version 2.11.1.
> > >> > Could you help provide insights and advice on the Kvrocks memory
> > >> > management topic? How to limit the Kvrocks memory usage effectively?
> > >> >
> > >> > Best regards,
> > >> > Quan
> > >> >
> > >> >
> > >>
> > >> --
> > >> Best Regards,
> > >> - *Hulk Lin*
> > >
> > >
> > >
> > > --
> > > Best Regards,
> > > - Hulk Lin
> >
> >
> >
> > --
> > Best Regards,
> > - Hulk Lin
> >
>
>
> --
> Best Regards,
> - Hulk Lin
>
--
Best Regards,
- *Hulk Lin*
Re: Apache James meets KVRocks
Posted by hulk <hu...@gmail.com>.
Hi, Quan
Thanks for your detailed information. I can reproduce it on my side. I
ran the benchmark a few times, and its memory is around the same(about
1.3GiB),
but I haven't dived deep into the Jemalloc stats yet.
On Fri, 2 May 2025 at 17:31, Hong Quan TRAN <hq...@linagora.com> wrote:
>
> Hi Hulk,
>
> Sorry for the late reply (we were on holiday).
>
> > I have no clue about why it's up to 1.3 GiB, will try to reproduce
> this on my side and analyze the memory in Jemalloc.
>
> We use Kvrocks as Rspamd's bayes training database. I would attach the Kvrocks used command stats for more details.
>
> BTW on my side, I tried to run the redis-benchmark tool against Kvrocks (still the same configuration). After a dozen times running `redis-benchmark -n 2000000 -c 50` command, Kvrocks memory is upto around 1.1GB and does not seems to be released automatically after a while.
>
> I hope this would help you ease reproducing the issue. I would be happy to do more diagnostic and share the output to you if needed.
>
> Regards,
>
> Quan
>
>
> On Apr 27, 2025 4:27 PM, from Hulk <hu...@gmail.com>
>
> Hi Benoit, Quan
>
> Sorry for the late reply. I took a look at your configuration, and it
> all looks good.
>
> I have no clue about why it's up to 1.3 GiB, will try to reproduce
> this on my side and analyze the memory in Jemalloc.
>
> Will update you once we have any progress.
>
> On Mon, 21 Apr 2025 at 11:00, hulk <hu...@gmail.com> wrote:
> >
> > Thanks all for taking care of this, will have a look.
> >
> > On Mon, 21 Apr 2025 at 10:56, Hong Quan TRAN <hq...@linagora.com> wrote:
> >>
> >> Hi,
> >>
> >> Somehow the format is still bad (although I tested sending it to my personal mail).
> >>
> >> I would send the Kvrocks config output via an attachment this time.
> >>
> >> Quan
> >>
> >> Vào ngày thg 4 21, 2025 9:39 SA, từ Hong Quan Tran <hq...@linagora.com>
> >>
> >> Hi everyone,
> >> Sorry for the bad formatting output.
> >> I resend the config output. I hope it is fine this time.
> >>
> >> kvrocks@561e141f9b72:/$ redis-cli -p 6379 CONFIG GET '*' 1) "auto-resize-block-and-sst" 2) "yes" 3) "backup-dir" 4) "/var/lib/kvrocks/backup" 5) "bgsave-cron" 6) "" 7) "bind" 8) "0.0.0.0" 9) "cluster-enabled" 10) "no" 11) "compact-cron" 12) "" 13) "compaction-checker-cron" 14) "" 15) "compaction-checker-range" 16) "" 17) "daemonize" 18) "no" 19) "db-name" 20) "change.me.db" 21) "dbsize-scan-cron" 22) "" 23) "dir" 24) "/var/lib/kvrocks" 25) "force-compact-file-age" 26) "172800" 27) "force-compact-file-min-deleted-percentage" 28) "10" 29) "fullsync-recv-file-delay" 30) "0" 31) "histogram-bucket-boundaries" 32) "" 33) "json-max-nesting-depth" 34) "1024" 35) "json-storage-format" 36) "json" 37) "log-dir" 38) "/var/lib/kvrocks" 39) "log-level" 40) "info" 41) "log-retention-days" 42) "-1" 43) "master-use-repl-port" 44) "no" 45) "masterauth" 46) "" 47) "max-backup-keep-hours" 48) "0" 49) "max-backup-to-keep" 50) "1" 51) "max-bitmap-to-string-mb" 52) "16" 53) "max-db-size" 54) "0" 55) "max-io-mb" 56) "0" 57) "max-replication-mb" 58) "0" 59) "maxclients" 60) "10240" 61) "migrate-batch-rate-limit-mb" 62) "16" 63) "migrate-batch-size-kb" 64) "16" 65) "migrate-pipeline-size" 66) "16" 67) "migrate-sequence-gap" 68) "10000" 69) "migrate-speed" 70) "4096" 71) "migrate-type" 72) "redis-command" 73) "persist-cluster-nodes-enabled" 74) "yes" 75) "pidfile" 76) "/var/run/kvrocks/kvrocks.pid" 77) "port" 78) "6379" 79) "profiling-sample-commands" 80) "" 81) "profiling-sample-ratio" 82) "0" 83) "profiling-sample-record-max-len" 84) "256" 85) "profiling-sample-record-threshold-ms" 86) "100" 87) "proto-max-bulk-len" 88) "536870912" 89) "purge-backup-on-fullsync" 90) "no" 91) "redis-cursor-compatible" 92) "yes" 93) "repl-namespace-enabled" 94) "no" 95) "replica-announce-ip" 96) "" 97) "replica-announce-port" 98) "0" 99) "replicaof"100) ""101) "replication-connect-timeout-ms"102) "3100"103) "replication-recv-timeout-ms"104) "3200"105) "requirepass"106) ""107) "resp3-enabled"108) "no"109) "rocksdb.avoid_unnecessary_blocking_io"110) "yes"111) "rocksdb.blob_file_size"112) "268435456"113) "rocksdb.blob_garbage_collection_age_cutoff"114) "25"115) "rocksdb.block_cache_size"116) "256"117) "rocksdb.block_cache_type"118) "lru"119) "rocksdb.block_size"120) "16384"121) "rocksdb.cache_index_and_filter_blocks"122) "yes"123) "rocksdb.compaction_readahead_size"124) "2097152"125) "rocksdb.compression"126) "no"127) "rocksdb.compression_level"128) "32767"129) "rocksdb.compression_start_level"130) "2"131) "rocksdb.delayed_write_rate"132) "0"133) "rocksdb.disable_auto_compactions"134) "no"135) "rocksdb.dump_malloc_stats"136) "yes"137) "rocksdb.enable_blob_files"138) "no"139) "rocksdb.enable_blob_garbage_collection"140) "yes"141) "rocksdb.enable_pipelined_write"142) "no"143) "rocksdb.level0_file_num_compaction_trigger"144) "4"145) "rocksdb.level0_slowdown_writes_trigger"146) "20"147) "rocksdb.level0_stop_writes_trigger"148) "40"149) "rocksdb.level_compaction_dynamic_level_bytes"150) "no"151) "rocksdb.max_background_compactions"152) "2"153) "rocksdb.max_background_flushes"154) "2"155) "rocksdb.max_background_jobs"156) "4"157) "rocksdb.max_bytes_for_level_base"158) "268435456"159) "rocksdb.max_bytes_for_level_multiplier"160) "10"161) "rocksdb.max_open_files"162) "8096"163) "rocksdb.max_subcompactions"164) "2"165) "rocksdb.max_total_wal_size"166) "256"167) "rocksdb.max_write_buffer_number"168) "2"169) "rocksdb.metadata_block_cache_size"170) "2048"171) "rocksdb.min_blob_size"172) "4096"173) "rocksdb.partition_filters"174) "yes"175) "rocksdb.rate_limiter_auto_tuned"176) "yes"177) "rocksdb.read_options.async_io"178) "yes"179) "rocksdb.row_cache_size"180) "0"181) "rocksdb.share_metadata_and_subkey_block_cache"182) "yes"183) "rocksdb.stats_dump_period_sec"184) "0"185) "rocksdb.subkey_block_cache_size"186) "2048"187) "rocksdb.target_file_size_base"188) "128"189) "rocksdb.wal_compression"190) "no"191) "rocksdb.wal_size_limit_mb"192) "16384"193) "rocksdb.wal_ttl_seconds"194) "10800"195) "rocksdb.write_buffer_size"196) "32"197) "rocksdb.write_options.disable_wal"198) "no"199) "rocksdb.write_options.low_pri"200) "no"201) "rocksdb.write_options.memtable_insert_hint_per_batch"202) "no"203) "rocksdb.write_options.no_slowdown"204) "no"205) "rocksdb.write_options.sync"206) "no"207) "rocksdb.write_options.write_batch_max_bytes"208) "0"209) "skip-block-cache-deallocation-on-close"210) "no"211) "slave-empty-db-before-fullsync"212) "no"213) "slave-priority"214) "100"215) "slave-read-only"216) "yes"217) "slave-serve-stale-data"218) "yes"219) "slaveof"220) ""221) "slowlog-log-slower-than"222) "200000"223) "slowlog-max-len"224) "128"225) "socket-fd"226) "-1"227) "supervised"228) "no"229) "tcp-backlog"230) "511"231) "timeout"232) "0"233) "tls-auth-clients"234) ""235) "tls-ca-cert-dir"236) ""237) "tls-ca-cert-file"238) ""239) "tls-cert-file"240) ""241) "tls-ciphers"242) ""243) "tls-ciphersuites"244) ""245) "tls-key-file"246) ""247) "tls-key-file-pass"248) ""249) "tls-port"250) "0"251) "tls-prefer-server-ciphers"252) "no"253) "tls-protocols"254) ""255) "tls-replication"256) "no"257) "tls-session-cache-size"258) "20480"259) "tls-session-cache-timeout"260) "300"261) "tls-session-caching"262) "yes"263) "txn-context-enabled"264) "no"265) "unixsocket"266) ""267) "unixsocketperm"268) "777"269) "use-rsid-psync"270) "no"271) "workers"272) "8"
> >> Quan
> >> Vào ngày thg 4 21, 2025 1:52 SA, từ Benoit Tellier <bt...@linagora.com>Careful
> >>
> >> Formating sucks
> >>
> >> Maybe resent it as a plain/text version to avoid this pitfall?
> >>
> >> --
> >>
> >> Best regards,
> >>
> >> Benoit TELLIER
> >>
> >> General manager of Linagora VIETNAM.
> >> Product owner for Team-Mail product.
> >> Chairman of the Apache James project.
> >>
> >> Mail: btellier@linagora.com
> >> Tel: (0033) 6 77 26 04 58 (WhatsApp, Signal)
> >>
> >>
> >> On Apr 20, 2025 3:43 PM, from Hong Quan Tran <hq...@linagora.com>Hi Hulk,
> >> > Thanks for your information, could you please provide the output by running the command in redis-cli `config get *`?
> >> Please see the output below.
> >>
> >> kvrocks@561e141f9b72:/$ redis-cli -p 6379 CONFIG GET '*' 1) "auto-resize-block-and-sst" 2) "yes" 3) "backup-dir" 4) "/var/lib/kvrocks/backup" 5) "bgsave-cron" 6) "" 7) "bind" 8) "0.0.0.0" 9) "cluster-enabled" 10) "no" 11) "compact-cron" 12) "" 13) "compaction-checker-cron" 14) "" 15) "compaction-checker-range" 16) "" 17) "daemonize" 18) "no" 19) "db-name" 20) "change.me.db" 21) "dbsize-scan-cron" 22) "" 23) "dir" 24) "/var/lib/kvrocks" 25) "force-compact-file-age" 26) "172800" 27) "force-compact-file-min-deleted-percentage" 28) "10" 29) "fullsync-recv-file-delay" 30) "0" 31) "histogram-bucket-boundaries" 32) "" 33) "json-max-nesting-depth" 34) "1024" 35) "json-storage-format" 36) "json" 37) "log-dir" 38) "/var/lib/kvrocks" 39) "log-level" 40) "info" 41) "log-retention-days" 42) "-1" 43) "master-use-repl-port" 44) "no" 45) "masterauth" 46) "" 47) "max-backup-keep-hours" 48) "0" 49) "max-backup-to-keep" 50) "1" 51) "max-bitmap-to-string-mb" 52) "16" 53) "max-db-size" 54) "0" 55) "max-io-mb" 56) "0" 57) "max-replication-mb" 58) "0" 59) "maxclients" 60) "10240" 61) "migrate-batch-rate-limit-mb" 62) "16" 63) "migrate-batch-size-kb" 64) "16" 65) "migrate-pipeline-size" 66) "16" 67) "migrate-sequence-gap" 68) "10000" 69) "migrate-speed" 70) "4096" 71) "migrate-type" 72) "redis-command" 73) "persist-cluster-nodes-enabled" 74) "yes" 75) "pidfile" 76) "/var/run/kvrocks/kvrocks.pid" 77) "port" 78) "6379" 79) "profiling-sample-commands" 80) "" 81) "profiling-sample-ratio" 82) "0" 83) "profiling-sample-record-max-len" 84) "256" 85) "profiling-sample-record-threshold-ms" 86) "100" 87) "proto-max-bulk-len" 88) "536870912" 89) "purge-backup-on-fullsync" 90) "no" 91) "redis-cursor-compatible" 92) "yes" 93) "repl-namespace-enabled" 94) "no" 95) "replica-announce-ip" 96) "" 97) "replica-announce-port" 98) "0" 99) "replicaof"100) ""101) "replication-connect-timeout-ms"102) "3100"103) "replication-recv-timeout-ms"104) "3200"105) "requirepass"106) ""107) "resp3-enabled"108) "no"109) "rocksdb.avoid_unnecessary_blocking_io"110) "yes"111) "rocksdb.blob_file_size"112) "268435456"113) "rocksdb.blob_garbage_collection_age_cutoff"114) "25"115) "rocksdb.block_cache_size"116) "256"117) "rocksdb.block_cache_type"118) "lru"119) "rocksdb.block_size"120) "16384"121) "rocksdb.cache_index_and_filter_blocks"122) "yes"123) "rocksdb.compaction_readahead_size"124) "2097152"125) "rocksdb.compression"126) "no"127) "rocksdb.compression_level"128) "32767"129) "rocksdb.compression_start_level"130) "2"131) "rocksdb.delayed_write_rate"132) "0"133) "rocksdb.disable_auto_compactions"134) "no"135) "rocksdb.dump_malloc_stats"136) "yes"137) "rocksdb.enable_blob_files"138) "no"139) "rocksdb.enable_blob_garbage_collection"140) "yes"141) "rocksdb.enable_pipelined_write"142) "no"143) "rocksdb.level0_file_num_compaction_trigger"144) "4"145) "rocksdb.level0_slowdown_writes_trigger"146) "20"147) "rocksdb.level0_stop_writes_trigger"148) "40"149) "rocksdb.level_compaction_dynamic_level_bytes"150) "no"151) "rocksdb.max_background_compactions"152) "2"153) "rocksdb.max_background_flushes"154) "2"155) "rocksdb.max_background_jobs"156) "4"157) "rocksdb.max_bytes_for_level_base"158) "268435456"159) "rocksdb.max_bytes_for_level_multiplier"160) "10"161) "rocksdb.max_open_files"162) "8096"163) "rocksdb.max_subcompactions"164) "2"165) "rocksdb.max_total_wal_size"166) "256"167) "rocksdb.max_write_buffer_number"168) "2"169) "rocksdb.metadata_block_cache_size"170) "2048"171) "rocksdb.min_blob_size"172) "4096"173) "rocksdb.partition_filters"174) "yes"175) "rocksdb.rate_limiter_auto_tuned"176) "yes"177) "rocksdb.read_options.async_io"178) "yes"179) "rocksdb.row_cache_size"180) "0"181) "rocksdb.share_metadata_and_subkey_block_cache"182) "yes"183) "rocksdb.stats_dump_period_sec"184) "0"185) "rocksdb.subkey_block_cache_size"186) "2048"187) "rocksdb.target_file_size_base"188) "128"189) "rocksdb.wal_compression"190) "no"191) "rocksdb.wal_size_limit_mb"192) "16384"193) "rocksdb.wal_ttl_seconds"194) "10800"195) "rocksdb.write_buffer_size"196) "32"197) "rocksdb.write_options.disable_wal"198) "no"199) "rocksdb.write_options.low_pri"200) "no"201) "rocksdb.write_options.memtable_insert_hint_per_batch"202) "no"203) "rocksdb.write_options.no_slowdown"204) "no"205) "rocksdb.write_options.sync"206) "no"207) "rocksdb.write_options.write_batch_max_bytes"208) "0"209) "skip-block-cache-deallocation-on-close"210) "no"211) "slave-empty-db-before-fullsync"212) "no"213) "slave-priority"214) "100"215) "slave-read-only"216) "yes"217) "slave-serve-stale-data"218) "yes"219) "slaveof"220) ""221) "slowlog-log-slower-than"222) "200000"223) "slowlog-max-len"224) "128"225) "socket-fd"226) "-1"227) "supervised"228) "no"229) "tcp-backlog"230) "511"231) "timeout"232) "0"233) "tls-auth-clients"234) ""235) "tls-ca-cert-dir"236) ""237) "tls-ca-cert-file"238) ""239) "tls-cert-file"240) ""241) "tls-ciphers"242) ""243) "tls-ciphersuites"244) ""245) "tls-key-file"246) ""247) "tls-key-file-pass"248) ""249) "tls-port"250) "0"251) "tls-prefer-server-ciphers"252) "no"253) "tls-protocols"254) ""255) "tls-replication"256) "no"257) "tls-session-cache-size"258) "20480"259) "tls-session-cache-timeout"260) "300"261) "tls-session-caching"262) "yes"263) "txn-context-enabled"264) "no"265) "unixsocket"266) ""267) "unixsocketperm"268) "777"269) "use-rsid-psync"270) "no"271) "workers"272) "8"
> >> Vào ngày thg 4 18, 2025 5:49 CH, từ Hulk <hu...@gmail.com>Hi Quan,
> >>
> >> Thanks for your information, could you please provide the output by running
> >> the command in redis-cli `config get *`?
> >>
> >> On Fri, 18 Apr 2025 at 18:24, Hong Quan TRAN <hq...@linagora.com> wrote:
> >>
> >> > Hi the Kvrocks community,
> >> > We would like to update you our progress on the integration topic.
> >> > We have started to migrate a Redis deployment to Kvrocks for a week. The
> >> > Kvrocks is running stable I would say. However, we have concern after
> >> > observing the Kvrocks's memory comsuption as it keeps slowly consume more
> >> > memory without releasing them back to the OS. At first the memory usage was
> >> > 500MB, then it is up to 700MB, 1GB, 1.3GB...
> >> > Please note that we already tuning the memory usage using the
> >> > recommendations at github.com/apache/kvrocks/discussions/2767.
> >> > ```kvrocks.conflog-level
> >> > <github.com/apache/kvrocks/discussions/2767.kvrocks.conflog-level>
> >> > infoport 6379rocksdb.max_total_wal_size 256rocksdb.max_write_buffer_number
> >> > 2rocksdb.write_buffer_size 32rocksdb.block_cache_size 256```
> >> > We expect the memory consumption would be low and stable with this applied
> >> > configuration. However, it keeps slowly increasing, which makes us worry.
> >> > Our memory stats so far:
> >> > ```docker statsCONTAINER ID NAME CPU % MEM USAGE /
> >> > LIMIT MEM % NET I/O BLOCK I/O PIDS561e141f9b72
> >> > kvrocks 0.14% 1.292GiB / 14.3GiB 9.03% 4.77GB /
> >> > 1.51GB 270kB / 8.08GB 19```
> >> > ```kvrocks@561e141f9b72:/$ redis-cli -p 6379 info memory#
> >> > Memoryused_memory_rss:983244800used_memory_rss_human:937.70Mused_memory_lua:59392used_memory_lua_human:58.00Kused_memory_startup:63860736```
> >> > FYI we are using Kvrocks version 2.11.1.
> >> > Could you help provide insights and advice on the Kvrocks memory
> >> > management topic? How to limit the Kvrocks memory usage effectively?
> >> >
> >> > Best regards,
> >> > Quan
> >> >
> >> >
> >>
> >> --
> >> Best Regards,
> >> - *Hulk Lin*
> >
> >
> >
> > --
> > Best Regards,
> > - Hulk Lin
>
>
>
> --
> Best Regards,
> - Hulk Lin
>
--
Best Regards,
- Hulk Lin
Re: Apache James meets KVRocks
Posted by Hong Quan TRAN <hq...@linagora.com>.
Hi Hulk,
Sorry for the late reply (we were on holiday).
> I have no clue about why it's up to 1.3 GiB, will try to reproducethis on my side and analyze the memory in Jemalloc.
We use Kvrocks as Rspamd's bayes training database. I would attach the Kvrocks used command stats for more details.
BTW on my side, I tried to run the redis-benchmark tool against Kvrocks (still the same configuration). After a dozen times running `redis-benchmark -n 2000000 -c 50` command, Kvrocks memory is upto around 1.1GB and does not seems to be released automatically after a while.
I hope this would help you ease reproducing the issue. I would be happy to do more diagnostic and share the output to you if needed.
Regards,
Quan
On Apr 27, 2025 4:27 PM, from Hulk <hu...@gmail.com>Hi Benoit, Quan
Sorry for the late reply. I took a look at your configuration, and it
all looks good.
I have no clue about why it's up to 1.3 GiB, will try to reproduce
this on my side and analyze the memory in Jemalloc.
Will update you once we have any progress.
On Mon, 21 Apr 2025 at 11:00, hulk <hu...@gmail.com> wrote:
>
> Thanks all for taking care of this, will have a look.
>
> On Mon, 21 Apr 2025 at 10:56, Hong Quan TRAN <hq...@linagora.com> wrote:
>>
>> Hi,
>>
>> Somehow the format is still bad (although I tested sending it to my personal mail).
>>
>> I would send the Kvrocks config output via an attachment this time.
>>
>> Quan
>>
>> Vào ngày thg 4 21, 2025 9:39 SA, từ Hong Quan Tran <hq...@linagora.com>
>>
>> Hi everyone,
>> Sorry for the bad formatting output.
>> I resend the config output. I hope it is fine this time.
>>
>> kvrocks@561e141f9b72:/$ redis-cli -p 6379 CONFIG GET '*' 1) "auto-resize-block-and-sst" 2) "yes" 3) "backup-dir" 4) "/var/lib/kvrocks/backup" 5) "bgsave-cron" 6) "" 7) "bind" 8) "0.0.0.0" 9) "cluster-enabled" 10) "no" 11) "compact-cron" 12) "" 13) "compaction-checker-cron" 14) "" 15) "compaction-checker-range" 16) "" 17) "daemonize" 18) "no" 19) "db-name" 20) "change.me.db" 21) "dbsize-scan-cron" 22) "" 23) "dir" 24) "/var/lib/kvrocks" 25) "force-compact-file-age" 26) "172800" 27) "force-compact-file-min-deleted-percentage" 28) "10" 29) "fullsync-recv-file-delay" 30) "0" 31) "histogram-bucket-boundaries" 32) "" 33) "json-max-nesting-depth" 34) "1024" 35) "json-storage-format" 36) "json" 37) "log-dir" 38) "/var/lib/kvrocks" 39) "log-level" 40) "info" 41) "log-retention-days" 42) "-1" 43) "master-use-repl-port" 44) "no" 45) "masterauth" 46) "" 47) "max-backup-keep-hours" 48) "0" 49) "max-backup-to-keep" 50) "1" 51) "max-bitmap-to-string-mb" 52) "16" 53) "max-db-size" 54) "0" 55) "max-io-mb" 56) "0" 57) "max-replication-mb" 58) "0" 59) "maxclients" 60) "10240" 61) "migrate-batch-rate-limit-mb" 62) "16" 63) "migrate-batch-size-kb" 64) "16" 65) "migrate-pipeline-size" 66) "16" 67) "migrate-sequence-gap" 68) "10000" 69) "migrate-speed" 70) "4096" 71) "migrate-type" 72) "redis-command" 73) "persist-cluster-nodes-enabled" 74) "yes" 75) "pidfile" 76) "/var/run/kvrocks/kvrocks.pid" 77) "port" 78) "6379" 79) "profiling-sample-commands" 80) "" 81) "profiling-sample-ratio" 82) "0" 83) "profiling-sample-record-max-len" 84) "256" 85) "profiling-sample-record-threshold-ms" 86) "100" 87) "proto-max-bulk-len" 88) "536870912" 89) "purge-backup-on-fullsync" 90) "no" 91) "redis-cursor-compatible" 92) "yes" 93) "repl-namespace-enabled" 94) "no" 95) "replica-announce-ip" 96) "" 97) "replica-announce-port" 98) "0" 99) "replicaof"100) ""101) "replication-connect-timeout-ms"102) "3100"103) "replication-recv-timeout-ms"104) "3200"105) "requirepass"106) ""107) "resp3-enabled"108) "no"109) "rocksdb.avoid_unnecessary_blocking_io"110) "yes"111) "rocksdb.blob_file_size"112) "268435456"113) "rocksdb.blob_garbage_collection_age_cutoff"114) "25"115) "rocksdb.block_cache_size"116) "256"117) "rocksdb.block_cache_type"118) "lru"119) "rocksdb.block_size"120) "16384"121) "rocksdb.cache_index_and_filter_blocks"122) "yes"123) "rocksdb.compaction_readahead_size"124) "2097152"125) "rocksdb.compression"126) "no"127) "rocksdb.compression_level"128) "32767"129) "rocksdb.compression_start_level"130) "2"131) "rocksdb.delayed_write_rate"132) "0"133) "rocksdb.disable_auto_compactions"134) "no"135) "rocksdb.dump_malloc_stats"136) "yes"137) "rocksdb.enable_blob_files"138) "no"139) "rocksdb.enable_blob_garbage_collection"140) "yes"141) "rocksdb.enable_pipelined_write"142) "no"143) "rocksdb.level0_file_num_compaction_trigger"144) "4"145) "rocksdb.level0_slowdown_writes_trigger"146) "20"147) "rocksdb.level0_stop_writes_trigger"148) "40"149) "rocksdb.level_compaction_dynamic_level_bytes"150) "no"151) "rocksdb.max_background_compactions"152) "2"153) "rocksdb.max_background_flushes"154) "2"155) "rocksdb.max_background_jobs"156) "4"157) "rocksdb.max_bytes_for_level_base"158) "268435456"159) "rocksdb.max_bytes_for_level_multiplier"160) "10"161) "rocksdb.max_open_files"162) "8096"163) "rocksdb.max_subcompactions"164) "2"165) "rocksdb.max_total_wal_size"166) "256"167) "rocksdb.max_write_buffer_number"168) "2"169) "rocksdb.metadata_block_cache_size"170) "2048"171) "rocksdb.min_blob_size"172) "4096"173) "rocksdb.partition_filters"174) "yes"175) "rocksdb.rate_limiter_auto_tuned"176) "yes"177) "rocksdb.read_options.async_io"178) "yes"179) "rocksdb.row_cache_size"180) "0"181) "rocksdb.share_metadata_and_subkey_block_cache"182) "yes"183) "rocksdb.stats_dump_period_sec"184) "0"185) "rocksdb.subkey_block_cache_size"186) "2048"187) "rocksdb.target_file_size_base"188) "128"189) "rocksdb.wal_compression"190) "no"191) "rocksdb.wal_size_limit_mb"192) "16384"193) "rocksdb.wal_ttl_seconds"194) "10800"195) "rocksdb.write_buffer_size"196) "32"197) "rocksdb.write_options.disable_wal"198) "no"199) "rocksdb.write_options.low_pri"200) "no"201) "rocksdb.write_options.memtable_insert_hint_per_batch"202) "no"203) "rocksdb.write_options.no_slowdown"204) "no"205) "rocksdb.write_options.sync"206) "no"207) "rocksdb.write_options.write_batch_max_bytes"208) "0"209) "skip-block-cache-deallocation-on-close"210) "no"211) "slave-empty-db-before-fullsync"212) "no"213) "slave-priority"214) "100"215) "slave-read-only"216) "yes"217) "slave-serve-stale-data"218) "yes"219) "slaveof"220) ""221) "slowlog-log-slower-than"222) "200000"223) "slowlog-max-len"224) "128"225) "socket-fd"226) "-1"227) "supervised"228) "no"229) "tcp-backlog"230) "511"231) "timeout"232) "0"233) "tls-auth-clients"234) ""235) "tls-ca-cert-dir"236) ""237) "tls-ca-cert-file"238) ""239) "tls-cert-file"240) ""241) "tls-ciphers"242) ""243) "tls-ciphersuites"244) ""245) "tls-key-file"246) ""247) "tls-key-file-pass"248) ""249) "tls-port"250) "0"251) "tls-prefer-server-ciphers"252) "no"253) "tls-protocols"254) ""255) "tls-replication"256) "no"257) "tls-session-cache-size"258) "20480"259) "tls-session-cache-timeout"260) "300"261) "tls-session-caching"262) "yes"263) "txn-context-enabled"264) "no"265) "unixsocket"266) ""267) "unixsocketperm"268) "777"269) "use-rsid-psync"270) "no"271) "workers"272) "8"
>> Quan
>> Vào ngày thg 4 21, 2025 1:52 SA, từ Benoit Tellier <bt...@linagora.com>Careful
>>
>> Formating sucks
>>
>> Maybe resent it as a plain/text version to avoid this pitfall?
>>
>> --
>>
>> Best regards,
>>
>> Benoit TELLIER
>>
>> General manager of Linagora VIETNAM.
>> Product owner for Team-Mail product.
>> Chairman of the Apache James project.
>>
>> Mail: btellier@linagora.com
>> Tel: (0033) 6 77 26 04 58 (WhatsApp, Signal)
>>
>>
>> On Apr 20, 2025 3:43 PM, from Hong Quan Tran <hq...@linagora.com>Hi Hulk,
>> > Thanks for your information, could you please provide the output by running the command in redis-cli `config get *`?
>> Please see the output below.
>>
>> kvrocks@561e141f9b72:/$ redis-cli -p 6379 CONFIG GET '*' 1) "auto-resize-block-and-sst" 2) "yes" 3) "backup-dir" 4) "/var/lib/kvrocks/backup" 5) "bgsave-cron" 6) "" 7) "bind" 8) "0.0.0.0" 9) "cluster-enabled" 10) "no" 11) "compact-cron" 12) "" 13) "compaction-checker-cron" 14) "" 15) "compaction-checker-range" 16) "" 17) "daemonize" 18) "no" 19) "db-name" 20) "change.me.db" 21) "dbsize-scan-cron" 22) "" 23) "dir" 24) "/var/lib/kvrocks" 25) "force-compact-file-age" 26) "172800" 27) "force-compact-file-min-deleted-percentage" 28) "10" 29) "fullsync-recv-file-delay" 30) "0" 31) "histogram-bucket-boundaries" 32) "" 33) "json-max-nesting-depth" 34) "1024" 35) "json-storage-format" 36) "json" 37) "log-dir" 38) "/var/lib/kvrocks" 39) "log-level" 40) "info" 41) "log-retention-days" 42) "-1" 43) "master-use-repl-port" 44) "no" 45) "masterauth" 46) "" 47) "max-backup-keep-hours" 48) "0" 49) "max-backup-to-keep" 50) "1" 51) "max-bitmap-to-string-mb" 52) "16" 53) "max-db-size" 54) "0" 55) "max-io-mb" 56) "0" 57) "max-replication-mb" 58) "0" 59) "maxclients" 60) "10240" 61) "migrate-batch-rate-limit-mb" 62) "16" 63) "migrate-batch-size-kb" 64) "16" 65) "migrate-pipeline-size" 66) "16" 67) "migrate-sequence-gap" 68) "10000" 69) "migrate-speed" 70) "4096" 71) "migrate-type" 72) "redis-command" 73) "persist-cluster-nodes-enabled" 74) "yes" 75) "pidfile" 76) "/var/run/kvrocks/kvrocks.pid" 77) "port" 78) "6379" 79) "profiling-sample-commands" 80) "" 81) "profiling-sample-ratio" 82) "0" 83) "profiling-sample-record-max-len" 84) "256" 85) "profiling-sample-record-threshold-ms" 86) "100" 87) "proto-max-bulk-len" 88) "536870912" 89) "purge-backup-on-fullsync" 90) "no" 91) "redis-cursor-compatible" 92) "yes" 93) "repl-namespace-enabled" 94) "no" 95) "replica-announce-ip" 96) "" 97) "replica-announce-port" 98) "0" 99) "replicaof"100) ""101) "replication-connect-timeout-ms"102) "3100"103) "replication-recv-timeout-ms"104) "3200"105) "requirepass"106) ""107) "resp3-enabled"108) "no"109) "rocksdb.avoid_unnecessary_blocking_io"110) "yes"111) "rocksdb.blob_file_size"112) "268435456"113) "rocksdb.blob_garbage_collection_age_cutoff"114) "25"115) "rocksdb.block_cache_size"116) "256"117) "rocksdb.block_cache_type"118) "lru"119) "rocksdb.block_size"120) "16384"121) "rocksdb.cache_index_and_filter_blocks"122) "yes"123) "rocksdb.compaction_readahead_size"124) "2097152"125) "rocksdb.compression"126) "no"127) "rocksdb.compression_level"128) "32767"129) "rocksdb.compression_start_level"130) "2"131) "rocksdb.delayed_write_rate"132) "0"133) "rocksdb.disable_auto_compactions"134) "no"135) "rocksdb.dump_malloc_stats"136) "yes"137) "rocksdb.enable_blob_files"138) "no"139) "rocksdb.enable_blob_garbage_collection"140) "yes"141) "rocksdb.enable_pipelined_write"142) "no"143) "rocksdb.level0_file_num_compaction_trigger"144) "4"145) "rocksdb.level0_slowdown_writes_trigger"146) "20"147) "rocksdb.level0_stop_writes_trigger"148) "40"149) "rocksdb.level_compaction_dynamic_level_bytes"150) "no"151) "rocksdb.max_background_compactions"152) "2"153) "rocksdb.max_background_flushes"154) "2"155) "rocksdb.max_background_jobs"156) "4"157) "rocksdb.max_bytes_for_level_base"158) "268435456"159) "rocksdb.max_bytes_for_level_multiplier"160) "10"161) "rocksdb.max_open_files"162) "8096"163) "rocksdb.max_subcompactions"164) "2"165) "rocksdb.max_total_wal_size"166) "256"167) "rocksdb.max_write_buffer_number"168) "2"169) "rocksdb.metadata_block_cache_size"170) "2048"171) "rocksdb.min_blob_size"172) "4096"173) "rocksdb.partition_filters"174) "yes"175) "rocksdb.rate_limiter_auto_tuned"176) "yes"177) "rocksdb.read_options.async_io"178) "yes"179) "rocksdb.row_cache_size"180) "0"181) "rocksdb.share_metadata_and_subkey_block_cache"182) "yes"183) "rocksdb.stats_dump_period_sec"184) "0"185) "rocksdb.subkey_block_cache_size"186) "2048"187) "rocksdb.target_file_size_base"188) "128"189) "rocksdb.wal_compression"190) "no"191) "rocksdb.wal_size_limit_mb"192) "16384"193) "rocksdb.wal_ttl_seconds"194) "10800"195) "rocksdb.write_buffer_size"196) "32"197) "rocksdb.write_options.disable_wal"198) "no"199) "rocksdb.write_options.low_pri"200) "no"201) "rocksdb.write_options.memtable_insert_hint_per_batch"202) "no"203) "rocksdb.write_options.no_slowdown"204) "no"205) "rocksdb.write_options.sync"206) "no"207) "rocksdb.write_options.write_batch_max_bytes"208) "0"209) "skip-block-cache-deallocation-on-close"210) "no"211) "slave-empty-db-before-fullsync"212) "no"213) "slave-priority"214) "100"215) "slave-read-only"216) "yes"217) "slave-serve-stale-data"218) "yes"219) "slaveof"220) ""221) "slowlog-log-slower-than"222) "200000"223) "slowlog-max-len"224) "128"225) "socket-fd"226) "-1"227) "supervised"228) "no"229) "tcp-backlog"230) "511"231) "timeout"232) "0"233) "tls-auth-clients"234) ""235) "tls-ca-cert-dir"236) ""237) "tls-ca-cert-file"238) ""239) "tls-cert-file"240) ""241) "tls-ciphers"242) ""243) "tls-ciphersuites"244) ""245) "tls-key-file"246) ""247) "tls-key-file-pass"248) ""249) "tls-port"250) "0"251) "tls-prefer-server-ciphers"252) "no"253) "tls-protocols"254) ""255) "tls-replication"256) "no"257) "tls-session-cache-size"258) "20480"259) "tls-session-cache-timeout"260) "300"261) "tls-session-caching"262) "yes"263) "txn-context-enabled"264) "no"265) "unixsocket"266) ""267) "unixsocketperm"268) "777"269) "use-rsid-psync"270) "no"271) "workers"272) "8"
>> Vào ngày thg 4 18, 2025 5:49 CH, từ Hulk <hu...@gmail.com>Hi Quan,
>>
>> Thanks for your information, could you please provide the output by running
>> the command in redis-cli `config get *`?
>>
>> On Fri, 18 Apr 2025 at 18:24, Hong Quan TRAN <hq...@linagora.com> wrote:
>>
>> > Hi the Kvrocks community,
>> > We would like to update you our progress on the integration topic.
>> > We have started to migrate a Redis deployment to Kvrocks for a week. The
>> > Kvrocks is running stable I would say. However, we have concern after
>> > observing the Kvrocks's memory comsuption as it keeps slowly consume more
>> > memory without releasing them back to the OS. At first the memory usage was
>> > 500MB, then it is up to 700MB, 1GB, 1.3GB...
>> > Please note that we already tuning the memory usage using the
>> > recommendations at github.com/apache/kvrocks/discussions/2767.
>> > ```kvrocks.conflog-level
>> > <github.com/apache/kvrocks/discussions/2767.kvrocks.conflog-level>
>> > infoport 6379rocksdb.max_total_wal_size 256rocksdb.max_write_buffer_number
>> > 2rocksdb.write_buffer_size 32rocksdb.block_cache_size 256```
>> > We expect the memory consumption would be low and stable with this applied
>> > configuration. However, it keeps slowly increasing, which makes us worry.
>> > Our memory stats so far:
>> > ```docker statsCONTAINER ID NAME CPU % MEM USAGE /
>> > LIMIT MEM % NET I/O BLOCK I/O PIDS561e141f9b72
>> > kvrocks 0.14% 1.292GiB / 14.3GiB 9.03% 4.77GB /
>> > 1.51GB 270kB / 8.08GB 19```
>> > ```kvrocks@561e141f9b72:/$ redis-cli -p 6379 info memory#
>> > Memoryused_memory_rss:983244800used_memory_rss_human:937.70Mused_memory_lua:59392used_memory_lua_human:58.00Kused_memory_startup:63860736```
>> > FYI we are using Kvrocks version 2.11.1.
>> > Could you help provide insights and advice on the Kvrocks memory
>> > management topic? How to limit the Kvrocks memory usage effectively?
>> >
>> > Best regards,
>> > Quan
>> >
>> >
>>
>> --
>> Best Regards,
>> - *Hulk Lin*
>
>
>
> --
> Best Regards,
> - Hulk Lin
--
Best Regards,
- Hulk Lin
Re: Apache James meets KVRocks
Posted by hulk <hu...@gmail.com>.
Hi Benoit, Quan
Sorry for the late reply. I took a look at your configuration, and it
all looks good.
I have no clue about why it's up to 1.3 GiB, will try to reproduce
this on my side and analyze the memory in Jemalloc.
Will update you once we have any progress.
On Mon, 21 Apr 2025 at 11:00, hulk <hu...@gmail.com> wrote:
>
> Thanks all for taking care of this, will have a look.
>
> On Mon, 21 Apr 2025 at 10:56, Hong Quan TRAN <hq...@linagora.com> wrote:
>>
>> Hi,
>>
>> Somehow the format is still bad (although I tested sending it to my personal mail).
>>
>> I would send the Kvrocks config output via an attachment this time.
>>
>> Quan
>>
>> Vào ngày thg 4 21, 2025 9:39 SA, từ Hong Quan Tran <hq...@linagora.com>
>>
>> Hi everyone,
>> Sorry for the bad formatting output.
>> I resend the config output. I hope it is fine this time.
>>
>> kvrocks@561e141f9b72:/$ redis-cli -p 6379 CONFIG GET '*' 1) "auto-resize-block-and-sst" 2) "yes" 3) "backup-dir" 4) "/var/lib/kvrocks/backup" 5) "bgsave-cron" 6) "" 7) "bind" 8) "0.0.0.0" 9) "cluster-enabled" 10) "no" 11) "compact-cron" 12) "" 13) "compaction-checker-cron" 14) "" 15) "compaction-checker-range" 16) "" 17) "daemonize" 18) "no" 19) "db-name" 20) "change.me.db" 21) "dbsize-scan-cron" 22) "" 23) "dir" 24) "/var/lib/kvrocks" 25) "force-compact-file-age" 26) "172800" 27) "force-compact-file-min-deleted-percentage" 28) "10" 29) "fullsync-recv-file-delay" 30) "0" 31) "histogram-bucket-boundaries" 32) "" 33) "json-max-nesting-depth" 34) "1024" 35) "json-storage-format" 36) "json" 37) "log-dir" 38) "/var/lib/kvrocks" 39) "log-level" 40) "info" 41) "log-retention-days" 42) "-1" 43) "master-use-repl-port" 44) "no" 45) "masterauth" 46) "" 47) "max-backup-keep-hours" 48) "0" 49) "max-backup-to-keep" 50) "1" 51) "max-bitmap-to-string-mb" 52) "16" 53) "max-db-size" 54) "0" 55) "max-io-mb" 56) "0" 57) "max-replication-mb" 58) "0" 59) "maxclients" 60) "10240" 61) "migrate-batch-rate-limit-mb" 62) "16" 63) "migrate-batch-size-kb" 64) "16" 65) "migrate-pipeline-size" 66) "16" 67) "migrate-sequence-gap" 68) "10000" 69) "migrate-speed" 70) "4096" 71) "migrate-type" 72) "redis-command" 73) "persist-cluster-nodes-enabled" 74) "yes" 75) "pidfile" 76) "/var/run/kvrocks/kvrocks.pid" 77) "port" 78) "6379" 79) "profiling-sample-commands" 80) "" 81) "profiling-sample-ratio" 82) "0" 83) "profiling-sample-record-max-len" 84) "256" 85) "profiling-sample-record-threshold-ms" 86) "100" 87) "proto-max-bulk-len" 88) "536870912" 89) "purge-backup-on-fullsync" 90) "no" 91) "redis-cursor-compatible" 92) "yes" 93) "repl-namespace-enabled" 94) "no" 95) "replica-announce-ip" 96) "" 97) "replica-announce-port" 98) "0" 99) "replicaof"100) ""101) "replication-connect-timeout-ms"102) "3100"103) "replication-recv-timeout-ms"104) "3200"105) "requirepass"106) ""107) "resp3-enabled"108) "no"109) "rocksdb.avoid_unnecessary_blocking_io"110) "yes"111) "rocksdb.blob_file_size"112) "268435456"113) "rocksdb.blob_garbage_collection_age_cutoff"114) "25"115) "rocksdb.block_cache_size"116) "256"117) "rocksdb.block_cache_type"118) "lru"119) "rocksdb.block_size"120) "16384"121) "rocksdb.cache_index_and_filter_blocks"122) "yes"123) "rocksdb.compaction_readahead_size"124) "2097152"125) "rocksdb.compression"126) "no"127) "rocksdb.compression_level"128) "32767"129) "rocksdb.compression_start_level"130) "2"131) "rocksdb.delayed_write_rate"132) "0"133) "rocksdb.disable_auto_compactions"134) "no"135) "rocksdb.dump_malloc_stats"136) "yes"137) "rocksdb.enable_blob_files"138) "no"139) "rocksdb.enable_blob_garbage_collection"140) "yes"141) "rocksdb.enable_pipelined_write"142) "no"143) "rocksdb.level0_file_num_compaction_trigger"144) "4"145) "rocksdb.level0_slowdown_writes_trigger"146) "20"147) "rocksdb.level0_stop_writes_trigger"148) "40"149) "rocksdb.level_compaction_dynamic_level_bytes"150) "no"151) "rocksdb.max_background_compactions"152) "2"153) "rocksdb.max_background_flushes"154) "2"155) "rocksdb.max_background_jobs"156) "4"157) "rocksdb.max_bytes_for_level_base"158) "268435456"159) "rocksdb.max_bytes_for_level_multiplier"160) "10"161) "rocksdb.max_open_files"162) "8096"163) "rocksdb.max_subcompactions"164) "2"165) "rocksdb.max_total_wal_size"166) "256"167) "rocksdb.max_write_buffer_number"168) "2"169) "rocksdb.metadata_block_cache_size"170) "2048"171) "rocksdb.min_blob_size"172) "4096"173) "rocksdb.partition_filters"174) "yes"175) "rocksdb.rate_limiter_auto_tuned"176) "yes"177) "rocksdb.read_options.async_io"178) "yes"179) "rocksdb.row_cache_size"180) "0"181) "rocksdb.share_metadata_and_subkey_block_cache"182) "yes"183) "rocksdb.stats_dump_period_sec"184) "0"185) "rocksdb.subkey_block_cache_size"186) "2048"187) "rocksdb.target_file_size_base"188) "128"189) "rocksdb.wal_compression"190) "no"191) "rocksdb.wal_size_limit_mb"192) "16384"193) "rocksdb.wal_ttl_seconds"194) "10800"195) "rocksdb.write_buffer_size"196) "32"197) "rocksdb.write_options.disable_wal"198) "no"199) "rocksdb.write_options.low_pri"200) "no"201) "rocksdb.write_options.memtable_insert_hint_per_batch"202) "no"203) "rocksdb.write_options.no_slowdown"204) "no"205) "rocksdb.write_options.sync"206) "no"207) "rocksdb.write_options.write_batch_max_bytes"208) "0"209) "skip-block-cache-deallocation-on-close"210) "no"211) "slave-empty-db-before-fullsync"212) "no"213) "slave-priority"214) "100"215) "slave-read-only"216) "yes"217) "slave-serve-stale-data"218) "yes"219) "slaveof"220) ""221) "slowlog-log-slower-than"222) "200000"223) "slowlog-max-len"224) "128"225) "socket-fd"226) "-1"227) "supervised"228) "no"229) "tcp-backlog"230) "511"231) "timeout"232) "0"233) "tls-auth-clients"234) ""235) "tls-ca-cert-dir"236) ""237) "tls-ca-cert-file"238) ""239) "tls-cert-file"240) ""241) "tls-ciphers"242) ""243) "tls-ciphersuites"244) ""245) "tls-key-file"246) ""247) "tls-key-file-pass"248) ""249) "tls-port"250) "0"251) "tls-prefer-server-ciphers"252) "no"253) "tls-protocols"254) ""255) "tls-replication"256) "no"257) "tls-session-cache-size"258) "20480"259) "tls-session-cache-timeout"260) "300"261) "tls-session-caching"262) "yes"263) "txn-context-enabled"264) "no"265) "unixsocket"266) ""267) "unixsocketperm"268) "777"269) "use-rsid-psync"270) "no"271) "workers"272) "8"
>> Quan
>> Vào ngày thg 4 21, 2025 1:52 SA, từ Benoit Tellier <bt...@linagora.com>Careful
>>
>> Formating sucks
>>
>> Maybe resent it as a plain/text version to avoid this pitfall?
>>
>> --
>>
>> Best regards,
>>
>> Benoit TELLIER
>>
>> General manager of Linagora VIETNAM.
>> Product owner for Team-Mail product.
>> Chairman of the Apache James project.
>>
>> Mail: btellier@linagora.com
>> Tel: (0033) 6 77 26 04 58 (WhatsApp, Signal)
>>
>>
>> On Apr 20, 2025 3:43 PM, from Hong Quan Tran <hq...@linagora.com>Hi Hulk,
>> > Thanks for your information, could you please provide the output by running the command in redis-cli `config get *`?
>> Please see the output below.
>>
>> kvrocks@561e141f9b72:/$ redis-cli -p 6379 CONFIG GET '*' 1) "auto-resize-block-and-sst" 2) "yes" 3) "backup-dir" 4) "/var/lib/kvrocks/backup" 5) "bgsave-cron" 6) "" 7) "bind" 8) "0.0.0.0" 9) "cluster-enabled" 10) "no" 11) "compact-cron" 12) "" 13) "compaction-checker-cron" 14) "" 15) "compaction-checker-range" 16) "" 17) "daemonize" 18) "no" 19) "db-name" 20) "change.me.db" 21) "dbsize-scan-cron" 22) "" 23) "dir" 24) "/var/lib/kvrocks" 25) "force-compact-file-age" 26) "172800" 27) "force-compact-file-min-deleted-percentage" 28) "10" 29) "fullsync-recv-file-delay" 30) "0" 31) "histogram-bucket-boundaries" 32) "" 33) "json-max-nesting-depth" 34) "1024" 35) "json-storage-format" 36) "json" 37) "log-dir" 38) "/var/lib/kvrocks" 39) "log-level" 40) "info" 41) "log-retention-days" 42) "-1" 43) "master-use-repl-port" 44) "no" 45) "masterauth" 46) "" 47) "max-backup-keep-hours" 48) "0" 49) "max-backup-to-keep" 50) "1" 51) "max-bitmap-to-string-mb" 52) "16" 53) "max-db-size" 54) "0" 55) "max-io-mb" 56) "0" 57) "max-replication-mb" 58) "0" 59) "maxclients" 60) "10240" 61) "migrate-batch-rate-limit-mb" 62) "16" 63) "migrate-batch-size-kb" 64) "16" 65) "migrate-pipeline-size" 66) "16" 67) "migrate-sequence-gap" 68) "10000" 69) "migrate-speed" 70) "4096" 71) "migrate-type" 72) "redis-command" 73) "persist-cluster-nodes-enabled" 74) "yes" 75) "pidfile" 76) "/var/run/kvrocks/kvrocks.pid" 77) "port" 78) "6379" 79) "profiling-sample-commands" 80) "" 81) "profiling-sample-ratio" 82) "0" 83) "profiling-sample-record-max-len" 84) "256" 85) "profiling-sample-record-threshold-ms" 86) "100" 87) "proto-max-bulk-len" 88) "536870912" 89) "purge-backup-on-fullsync" 90) "no" 91) "redis-cursor-compatible" 92) "yes" 93) "repl-namespace-enabled" 94) "no" 95) "replica-announce-ip" 96) "" 97) "replica-announce-port" 98) "0" 99) "replicaof"100) ""101) "replication-connect-timeout-ms"102) "3100"103) "replication-recv-timeout-ms"104) "3200"105) "requirepass"106) ""107) "resp3-enabled"108) "no"109) "rocksdb.avoid_unnecessary_blocking_io"110) "yes"111) "rocksdb.blob_file_size"112) "268435456"113) "rocksdb.blob_garbage_collection_age_cutoff"114) "25"115) "rocksdb.block_cache_size"116) "256"117) "rocksdb.block_cache_type"118) "lru"119) "rocksdb.block_size"120) "16384"121) "rocksdb.cache_index_and_filter_blocks"122) "yes"123) "rocksdb.compaction_readahead_size"124) "2097152"125) "rocksdb.compression"126) "no"127) "rocksdb.compression_level"128) "32767"129) "rocksdb.compression_start_level"130) "2"131) "rocksdb.delayed_write_rate"132) "0"133) "rocksdb.disable_auto_compactions"134) "no"135) "rocksdb.dump_malloc_stats"136) "yes"137) "rocksdb.enable_blob_files"138) "no"139) "rocksdb.enable_blob_garbage_collection"140) "yes"141) "rocksdb.enable_pipelined_write"142) "no"143) "rocksdb.level0_file_num_compaction_trigger"144) "4"145) "rocksdb.level0_slowdown_writes_trigger"146) "20"147) "rocksdb.level0_stop_writes_trigger"148) "40"149) "rocksdb.level_compaction_dynamic_level_bytes"150) "no"151) "rocksdb.max_background_compactions"152) "2"153) "rocksdb.max_background_flushes"154) "2"155) "rocksdb.max_background_jobs"156) "4"157) "rocksdb.max_bytes_for_level_base"158) "268435456"159) "rocksdb.max_bytes_for_level_multiplier"160) "10"161) "rocksdb.max_open_files"162) "8096"163) "rocksdb.max_subcompactions"164) "2"165) "rocksdb.max_total_wal_size"166) "256"167) "rocksdb.max_write_buffer_number"168) "2"169) "rocksdb.metadata_block_cache_size"170) "2048"171) "rocksdb.min_blob_size"172) "4096"173) "rocksdb.partition_filters"174) "yes"175) "rocksdb.rate_limiter_auto_tuned"176) "yes"177) "rocksdb.read_options.async_io"178) "yes"179) "rocksdb.row_cache_size"180) "0"181) "rocksdb.share_metadata_and_subkey_block_cache"182) "yes"183) "rocksdb.stats_dump_period_sec"184) "0"185) "rocksdb.subkey_block_cache_size"186) "2048"187) "rocksdb.target_file_size_base"188) "128"189) "rocksdb.wal_compression"190) "no"191) "rocksdb.wal_size_limit_mb"192) "16384"193) "rocksdb.wal_ttl_seconds"194) "10800"195) "rocksdb.write_buffer_size"196) "32"197) "rocksdb.write_options.disable_wal"198) "no"199) "rocksdb.write_options.low_pri"200) "no"201) "rocksdb.write_options.memtable_insert_hint_per_batch"202) "no"203) "rocksdb.write_options.no_slowdown"204) "no"205) "rocksdb.write_options.sync"206) "no"207) "rocksdb.write_options.write_batch_max_bytes"208) "0"209) "skip-block-cache-deallocation-on-close"210) "no"211) "slave-empty-db-before-fullsync"212) "no"213) "slave-priority"214) "100"215) "slave-read-only"216) "yes"217) "slave-serve-stale-data"218) "yes"219) "slaveof"220) ""221) "slowlog-log-slower-than"222) "200000"223) "slowlog-max-len"224) "128"225) "socket-fd"226) "-1"227) "supervised"228) "no"229) "tcp-backlog"230) "511"231) "timeout"232) "0"233) "tls-auth-clients"234) ""235) "tls-ca-cert-dir"236) ""237) "tls-ca-cert-file"238) ""239) "tls-cert-file"240) ""241) "tls-ciphers"242) ""243) "tls-ciphersuites"244) ""245) "tls-key-file"246) ""247) "tls-key-file-pass"248) ""249) "tls-port"250) "0"251) "tls-prefer-server-ciphers"252) "no"253) "tls-protocols"254) ""255) "tls-replication"256) "no"257) "tls-session-cache-size"258) "20480"259) "tls-session-cache-timeout"260) "300"261) "tls-session-caching"262) "yes"263) "txn-context-enabled"264) "no"265) "unixsocket"266) ""267) "unixsocketperm"268) "777"269) "use-rsid-psync"270) "no"271) "workers"272) "8"
>> Vào ngày thg 4 18, 2025 5:49 CH, từ Hulk <hu...@gmail.com>Hi Quan,
>>
>> Thanks for your information, could you please provide the output by running
>> the command in redis-cli `config get *`?
>>
>> On Fri, 18 Apr 2025 at 18:24, Hong Quan TRAN <hq...@linagora.com> wrote:
>>
>> > Hi the Kvrocks community,
>> > We would like to update you our progress on the integration topic.
>> > We have started to migrate a Redis deployment to Kvrocks for a week. The
>> > Kvrocks is running stable I would say. However, we have concern after
>> > observing the Kvrocks's memory comsuption as it keeps slowly consume more
>> > memory without releasing them back to the OS. At first the memory usage was
>> > 500MB, then it is up to 700MB, 1GB, 1.3GB...
>> > Please note that we already tuning the memory usage using the
>> > recommendations at github.com/apache/kvrocks/discussions/2767.
>> > ```kvrocks.conflog-level
>> > <github.com/apache/kvrocks/discussions/2767.kvrocks.conflog-level>
>> > infoport 6379rocksdb.max_total_wal_size 256rocksdb.max_write_buffer_number
>> > 2rocksdb.write_buffer_size 32rocksdb.block_cache_size 256```
>> > We expect the memory consumption would be low and stable with this applied
>> > configuration. However, it keeps slowly increasing, which makes us worry.
>> > Our memory stats so far:
>> > ```docker statsCONTAINER ID NAME CPU % MEM USAGE /
>> > LIMIT MEM % NET I/O BLOCK I/O PIDS561e141f9b72
>> > kvrocks 0.14% 1.292GiB / 14.3GiB 9.03% 4.77GB /
>> > 1.51GB 270kB / 8.08GB 19```
>> > ```kvrocks@561e141f9b72:/$ redis-cli -p 6379 info memory#
>> > Memoryused_memory_rss:983244800used_memory_rss_human:937.70Mused_memory_lua:59392used_memory_lua_human:58.00Kused_memory_startup:63860736```
>> > FYI we are using Kvrocks version 2.11.1.
>> > Could you help provide insights and advice on the Kvrocks memory
>> > management topic? How to limit the Kvrocks memory usage effectively?
>> >
>> > Best regards,
>> > Quan
>> >
>> >
>>
>> --
>> Best Regards,
>> - *Hulk Lin*
>
>
>
> --
> Best Regards,
> - Hulk Lin
--
Best Regards,
- Hulk Lin
Re: Apache James meets KVRocks
Posted by hulk <hu...@gmail.com>.
Thanks all for taking care of this, will have a look.
On Mon, 21 Apr 2025 at 10:56, Hong Quan TRAN <hq...@linagora.com> wrote:
> Hi,
>
> Somehow the format is still bad (although I tested sending it to my
> personal mail).
>
> I would send the Kvrocks config output via an attachment this time.
>
> Quan
>
> Vào ngày thg 4 21, 2025 9:39 SA, từ Hong Quan Tran <hq...@linagora.com>
>
> Hi everyone,
> Sorry for the bad formatting output.
> I resend the config output. I hope it is fine this time.
>
> kvrocks@561e141f9b72:/$ redis-cli -p 6379 CONFIG GET '*' 1) "auto-resize-block-and-sst" 2) "yes" 3) "backup-dir" 4) "/var/lib/kvrocks/backup" 5) "bgsave-cron" 6) "" 7) "bind" 8) "0.0.0.0" 9) "cluster-enabled" 10) "no" 11) "compact-cron" 12) "" 13) "compaction-checker-cron" 14) "" 15) "compaction-checker-range" 16) "" 17) "daemonize" 18) "no" 19) "db-name" 20) "change.me.db" <https://change.me.db> 21) "dbsize-scan-cron" 22) "" 23) "dir" 24) "/var/lib/kvrocks" 25) "force-compact-file-age" 26) "172800" 27) "force-compact-file-min-deleted-percentage" 28) "10" 29) "fullsync-recv-file-delay" 30) "0" 31) "histogram-bucket-boundaries" 32) "" 33) "json-max-nesting-depth" 34) "1024" 35) "json-storage-format" 36) "json" 37) "log-dir" 38) "/var/lib/kvrocks" 39) "log-level" 40) "info" 41) "log-retention-days" 42) "-1" 43) "master-use-repl-port" 44) "no" 45) "masterauth" 46) "" 47) "max-backup-keep-hours" 48) "0" 49) "max-backup-to-keep" 50) "1" 51) "max-bitmap-to-string-mb" 52) "16" 53) "max-db-size" 54) "0" 55) "max-io-mb" 56) "0" 57) "max-replication-mb" 58) "0" 59) "maxclients" 60) "10240" 61) "migrate-batch-rate-limit-mb" 62) "16" 63) "migrate-batch-size-kb" 64) "16" 65) "migrate-pipeline-size" 66) "16" 67) "migrate-sequence-gap" 68) "10000" 69) "migrate-speed" 70) "4096" 71) "migrate-type" 72) "redis-command" 73) "persist-cluster-nodes-enabled" 74) "yes" 75) "pidfile" 76) "/var/run/kvrocks/kvrocks.pid" <https://kvrocks.pid> 77) "port" 78) "6379" 79) "profiling-sample-commands" 80) "" 81) "profiling-sample-ratio" 82) "0" 83) "profiling-sample-record-max-len" 84) "256" 85) "profiling-sample-record-threshold-ms" 86) "100" 87) "proto-max-bulk-len" 88) "536870912" 89) "purge-backup-on-fullsync" 90) "no" 91) "redis-cursor-compatible" 92) "yes" 93) "repl-namespace-enabled" 94) "no" 95) "replica-announce-ip" 96) "" 97) "replica-announce-port" 98) "0" 99) "replicaof"100) ""101) "replication-connect-timeout-ms"102) "3100"103) "replication-recv-timeout-ms"104) "3200"105) "requirepass"106) ""107) "resp3-enabled"108) "no"109) "rocksdb.avoid_unnecessary_blocking_io"110) "yes"111) "rocksdb.blob_file_size"112) "268435456"113) "rocksdb.blob_garbage_collection_age_cutoff"114) "25"115) "rocksdb.block_cache_size"116) "256"117) "rocksdb.block_cache_type"118) "lru"119) "rocksdb.block_size"120) "16384"121) "rocksdb.cache_index_and_filter_blocks"122) "yes"123) "rocksdb.compaction_readahead_size"124) "2097152"125) "rocksdb.compression"126) "no"127) "rocksdb.compression_level"128) "32767"129) "rocksdb.compression_start_level"130) "2"131) "rocksdb.delayed_write_rate"132) "0"133) "rocksdb.disable_auto_compactions"134) "no"135) "rocksdb.dump_malloc_stats"136) "yes"137) "rocksdb.enable_blob_files"138) "no"139) "rocksdb.enable_blob_garbage_collection"140) "yes"141) "rocksdb.enable_pipelined_write"142) "no"143) "rocksdb.level0_file_num_compaction_trigger"144) "4"145) "rocksdb.level0_slowdown_writes_trigger"146) "20"147) "rocksdb.level0_stop_writes_trigger"148) "40"149) "rocksdb.level_compaction_dynamic_level_bytes"150) "no"151) "rocksdb.max_background_compactions"152) "2"153) "rocksdb.max_background_flushes"154) "2"155) "rocksdb.max_background_jobs"156) "4"157) "rocksdb.max_bytes_for_level_base"158) "268435456"159) "rocksdb.max_bytes_for_level_multiplier"160) "10"161) "rocksdb.max_open_files"162) "8096"163) "rocksdb.max_subcompactions"164) "2"165) "rocksdb.max_total_wal_size"166) "256"167) "rocksdb.max_write_buffer_number"168) "2"169) "rocksdb.metadata_block_cache_size"170) "2048"171) "rocksdb.min_blob_size"172) "4096"173) "rocksdb.partition_filters"174) "yes"175) "rocksdb.rate_limiter_auto_tuned"176) "yes"177) "rocksdb.read_options.async_io"178) "yes"179) "rocksdb.row_cache_size"180) "0"181) "rocksdb.share_metadata_and_subkey_block_cache"182) "yes"183) "rocksdb.stats_dump_period_sec"184) "0"185) "rocksdb.subkey_block_cache_size"186) "2048"187) "rocksdb.target_file_size_base"188) "128"189) "rocksdb.wal_compression"190) "no"191) "rocksdb.wal_size_limit_mb"192) "16384"193) "rocksdb.wal_ttl_seconds"194) "10800"195) "rocksdb.write_buffer_size"196) "32"197) "rocksdb.write_options.disable_wal"198) "no"199) "rocksdb.write_options.low_pri"200) "no"201) "rocksdb.write_options.memtable_insert_hint_per_batch"202) "no"203) "rocksdb.write_options.no_slowdown"204) "no"205) "rocksdb.write_options.sync"206 <https://rocksdb.write_options.sync>) "no"207) "rocksdb.write_options.write_batch_max_bytes"208) "0"209) "skip-block-cache-deallocation-on-close"210) "no"211) "slave-empty-db-before-fullsync"212) "no"213) "slave-priority"214) "100"215) "slave-read-only"216) "yes"217) "slave-serve-stale-data"218) "yes"219) "slaveof"220) ""221) "slowlog-log-slower-than"222) "200000"223) "slowlog-max-len"224) "128"225) "socket-fd"226) "-1"227) "supervised"228) "no"229) "tcp-backlog"230) "511"231) "timeout"232) "0"233) "tls-auth-clients"234) ""235) "tls-ca-cert-dir"236) ""237) "tls-ca-cert-file"238) ""239) "tls-cert-file"240) ""241) "tls-ciphers"242) ""243) "tls-ciphersuites"244) ""245) "tls-key-file"246) ""247) "tls-key-file-pass"248) ""249) "tls-port"250) "0"251) "tls-prefer-server-ciphers"252) "no"253) "tls-protocols"254) ""255) "tls-replication"256) "no"257) "tls-session-cache-size"258) "20480"259) "tls-session-cache-timeout"260) "300"261) "tls-session-caching"262) "yes"263) "txn-context-enabled"264) "no"265) "unixsocket"266) ""267) "unixsocketperm"268) "777"269) "use-rsid-psync"270) "no"271) "workers"272) "8"
> Quan
> Vào ngày thg 4 21, 2025 1:52 SA, từ Benoit Tellier <bt...@linagora.com>Careful
>
> Formating sucks
>
> Maybe resent it as a plain/text version to avoid this pitfall?
>
> --
>
> Best regards,
>
> Benoit TELLIER
>
> General manager of Linagora VIETNAM.
> Product owner for Team-Mail product.
> Chairman of the Apache James project.
>
> Mail: btellier@linagora.com
> Tel: (0033) 6 77 26 04 58 (WhatsApp, Signal)
>
>
> On Apr 20, 2025 3:43 PM, from Hong Quan Tran <hq...@linagora.com>Hi Hulk,
> > Thanks for your information, could you please provide the output by running the command in redis-cli `config get *`?
> Please see the output below.
>
> kvrocks@561e141f9b72:/$ redis-cli -p 6379 CONFIG GET '*' 1) "auto-resize-block-and-sst" 2) "yes" 3) "backup-dir" 4) "/var/lib/kvrocks/backup" 5) "bgsave-cron" 6) "" 7) "bind" 8) "0.0.0.0" 9) "cluster-enabled" 10) "no" 11) "compact-cron" 12) "" 13) "compaction-checker-cron" 14) "" 15) "compaction-checker-range" 16) "" 17) "daemonize" 18) "no" 19) "db-name" 20) "change.me.db" <https://change.me.db> 21) "dbsize-scan-cron" 22) "" 23) "dir" 24) "/var/lib/kvrocks" 25) "force-compact-file-age" 26) "172800" 27) "force-compact-file-min-deleted-percentage" 28) "10" 29) "fullsync-recv-file-delay" 30) "0" 31) "histogram-bucket-boundaries" 32) "" 33) "json-max-nesting-depth" 34) "1024" 35) "json-storage-format" 36) "json" 37) "log-dir" 38) "/var/lib/kvrocks" 39) "log-level" 40) "info" 41) "log-retention-days" 42) "-1" 43) "master-use-repl-port" 44) "no" 45) "masterauth" 46) "" 47) "max-backup-keep-hours" 48) "0" 49) "max-backup-to-keep" 50) "1" 51) "max-bitmap-to-string-mb" 52) "16" 53) "max-db-size" 54) "0" 55) "max-io-mb" 56) "0" 57) "max-replication-mb" 58) "0" 59) "maxclients" 60) "10240" 61) "migrate-batch-rate-limit-mb" 62) "16" 63) "migrate-batch-size-kb" 64) "16" 65) "migrate-pipeline-size" 66) "16" 67) "migrate-sequence-gap" 68) "10000" 69) "migrate-speed" 70) "4096" 71) "migrate-type" 72) "redis-command" 73) "persist-cluster-nodes-enabled" 74) "yes" 75) "pidfile" 76) "/var/run/kvrocks/kvrocks.pid" <https://kvrocks.pid> 77) "port" 78) "6379" 79) "profiling-sample-commands" 80) "" 81) "profiling-sample-ratio" 82) "0" 83) "profiling-sample-record-max-len" 84) "256" 85) "profiling-sample-record-threshold-ms" 86) "100" 87) "proto-max-bulk-len" 88) "536870912" 89) "purge-backup-on-fullsync" 90) "no" 91) "redis-cursor-compatible" 92) "yes" 93) "repl-namespace-enabled" 94) "no" 95) "replica-announce-ip" 96) "" 97) "replica-announce-port" 98) "0" 99) "replicaof"100) ""101) "replication-connect-timeout-ms"102) "3100"103) "replication-recv-timeout-ms"104) "3200"105) "requirepass"106) ""107) "resp3-enabled"108) "no"109) "rocksdb.avoid_unnecessary_blocking_io"110) "yes"111) "rocksdb.blob_file_size"112) "268435456"113) "rocksdb.blob_garbage_collection_age_cutoff"114) "25"115) "rocksdb.block_cache_size"116) "256"117) "rocksdb.block_cache_type"118) "lru"119) "rocksdb.block_size"120) "16384"121) "rocksdb.cache_index_and_filter_blocks"122) "yes"123) "rocksdb.compaction_readahead_size"124) "2097152"125) "rocksdb.compression"126) "no"127) "rocksdb.compression_level"128) "32767"129) "rocksdb.compression_start_level"130) "2"131) "rocksdb.delayed_write_rate"132) "0"133) "rocksdb.disable_auto_compactions"134) "no"135) "rocksdb.dump_malloc_stats"136) "yes"137) "rocksdb.enable_blob_files"138) "no"139) "rocksdb.enable_blob_garbage_collection"140) "yes"141) "rocksdb.enable_pipelined_write"142) "no"143) "rocksdb.level0_file_num_compaction_trigger"144) "4"145) "rocksdb.level0_slowdown_writes_trigger"146) "20"147) "rocksdb.level0_stop_writes_trigger"148) "40"149) "rocksdb.level_compaction_dynamic_level_bytes"150) "no"151) "rocksdb.max_background_compactions"152) "2"153) "rocksdb.max_background_flushes"154) "2"155) "rocksdb.max_background_jobs"156) "4"157) "rocksdb.max_bytes_for_level_base"158) "268435456"159) "rocksdb.max_bytes_for_level_multiplier"160) "10"161) "rocksdb.max_open_files"162) "8096"163) "rocksdb.max_subcompactions"164) "2"165) "rocksdb.max_total_wal_size"166) "256"167) "rocksdb.max_write_buffer_number"168) "2"169) "rocksdb.metadata_block_cache_size"170) "2048"171) "rocksdb.min_blob_size"172) "4096"173) "rocksdb.partition_filters"174) "yes"175) "rocksdb.rate_limiter_auto_tuned"176) "yes"177) "rocksdb.read_options.async_io"178) "yes"179) "rocksdb.row_cache_size"180) "0"181) "rocksdb.share_metadata_and_subkey_block_cache"182) "yes"183) "rocksdb.stats_dump_period_sec"184) "0"185) "rocksdb.subkey_block_cache_size"186) "2048"187) "rocksdb.target_file_size_base"188) "128"189) "rocksdb.wal_compression"190) "no"191) "rocksdb.wal_size_limit_mb"192) "16384"193) "rocksdb.wal_ttl_seconds"194) "10800"195) "rocksdb.write_buffer_size"196) "32"197) "rocksdb.write_options.disable_wal"198) "no"199) "rocksdb.write_options.low_pri"200) "no"201) "rocksdb.write_options.memtable_insert_hint_per_batch"202) "no"203) "rocksdb.write_options.no_slowdown"204) "no"205) "rocksdb.write_options.sync"206 <https://rocksdb.write_options.sync>) "no"207) "rocksdb.write_options.write_batch_max_bytes"208) "0"209) "skip-block-cache-deallocation-on-close"210) "no"211) "slave-empty-db-before-fullsync"212) "no"213) "slave-priority"214) "100"215) "slave-read-only"216) "yes"217) "slave-serve-stale-data"218) "yes"219) "slaveof"220) ""221) "slowlog-log-slower-than"222) "200000"223) "slowlog-max-len"224) "128"225) "socket-fd"226) "-1"227) "supervised"228) "no"229) "tcp-backlog"230) "511"231) "timeout"232) "0"233) "tls-auth-clients"234) ""235) "tls-ca-cert-dir"236) ""237) "tls-ca-cert-file"238) ""239) "tls-cert-file"240) ""241) "tls-ciphers"242) ""243) "tls-ciphersuites"244) ""245) "tls-key-file"246) ""247) "tls-key-file-pass"248) ""249) "tls-port"250) "0"251) "tls-prefer-server-ciphers"252) "no"253) "tls-protocols"254) ""255) "tls-replication"256) "no"257) "tls-session-cache-size"258) "20480"259) "tls-session-cache-timeout"260) "300"261) "tls-session-caching"262) "yes"263) "txn-context-enabled"264) "no"265) "unixsocket"266) ""267) "unixsocketperm"268) "777"269) "use-rsid-psync"270) "no"271) "workers"272) "8"
> Vào ngày thg 4 18, 2025 5:49 CH, từ Hulk <hu...@gmail.com>Hi Quan,
>
> Thanks for your information, could you please provide the output by running
> the command in redis-cli `config get *`?
>
> On Fri, 18 Apr 2025 at 18:24, Hong Quan TRAN <hq...@linagora.com> wrote:
>
> > Hi the Kvrocks community,
> > We would like to update you our progress on the integration topic.
> > We have started to migrate a Redis deployment to Kvrocks for a week. The
> > Kvrocks is running stable I would say. However, we have concern after
> > observing the Kvrocks's memory comsuption as it keeps slowly consume more
> > memory without releasing them back to the OS. At first the memory usage was
> > 500MB, then it is up to 700MB, 1GB, 1.3GB...
> > Please note that we already tuning the memory usage using the
> > recommendations at github.com/apache/kvrocks/discussions/2767.
> > ```kvrocks.conflog-level
> > <github.com/apache/kvrocks/discussions/2767.kvrocks.conflog-level>
> > infoport 6379rocksdb.max_total_wal_size 256rocksdb.max_write_buffer_number
> > 2rocksdb.write_buffer_size 32rocksdb.block_cache_size 256```
> > We expect the memory consumption would be low and stable with this applied
> > configuration. However, it keeps slowly increasing, which makes us worry.
> > Our memory stats so far:
> > ```docker statsCONTAINER ID NAME CPU % MEM USAGE /
> > LIMIT MEM % NET I/O BLOCK I/O PIDS561e141f9b72
> > kvrocks 0.14% 1.292GiB / 14.3GiB 9.03% 4.77GB /
> > 1.51GB 270kB / 8.08GB 19```
> > ```kvrocks@561e141f9b72:/$ redis-cli -p 6379 info memory#
> > Memoryused_memory_rss:983244800used_memory_rss_human:937.70Mused_memory_lua:59392used_memory_lua_human:58.00Kused_memory_startup:63860736```
> > FYI we are using Kvrocks version 2.11.1.
> > Could you help provide insights and advice on the Kvrocks memory
> > management topic? How to limit the Kvrocks memory usage effectively?
> >
> > Best regards,
> > Quan
> >
> >
>
> --
> Best Regards,
> - *Hulk Lin*
>
>
--
Best Regards,
- *Hulk Lin*
Re: Apache James meets KVRocks
Posted by Hong Quan TRAN <hq...@linagora.com>.
Hi,
Somehow the format is still bad (although I tested sending it to my personal mail).
I would send the Kvrocks config output via an attachment this time.
Quan
Vào ngày thg 4 21, 2025 9:39 SA, từ Hong Quan Tran <hq...@linagora.com>Hi everyone,
Sorry for the bad formatting output.
I resend the config output. I hope it is fine this time.
kvrocks@561e141f9b72:/$ redis-cli -p 6379 CONFIG GET '*' 1) "auto-resize-block-and-sst" 2) "yes" 3) "backup-dir" 4) "/var/lib/kvrocks/backup" 5) "bgsave-cron" 6) "" 7) "bind" 8) "0.0.0.0" 9) "cluster-enabled" 10) "no" 11) "compact-cron" 12) "" 13) "compaction-checker-cron" 14) "" 15) "compaction-checker-range" 16) "" 17) "daemonize" 18) "no" 19) "db-name" 20) "change.me.db" 21) "dbsize-scan-cron" 22) "" 23) "dir" 24) "/var/lib/kvrocks" 25) "force-compact-file-age" 26) "172800" 27) "force-compact-file-min-deleted-percentage" 28) "10" 29) "fullsync-recv-file-delay" 30) "0" 31) "histogram-bucket-boundaries" 32) "" 33) "json-max-nesting-depth" 34) "1024" 35) "json-storage-format" 36) "json" 37) "log-dir" 38) "/var/lib/kvrocks" 39) "log-level" 40) "info" 41) "log-retention-days" 42) "-1" 43) "master-use-repl-port" 44) "no" 45) "masterauth" 46) "" 47) "max-backup-keep-hours" 48) "0" 49) "max-backup-to-keep" 50) "1" 51) "max-bitmap-to-string-mb" 52) "16" 53) "max-db-size" 54) "0" 55) "max-io-mb" 56) "0" 57) "max-replication-mb" 58) "0" 59) "maxclients" 60) "10240" 61) "migrate-batch-rate-limit-mb" 62) "16" 63) "migrate-batch-size-kb" 64) "16" 65) "migrate-pipeline-size" 66) "16" 67) "migrate-sequence-gap" 68) "10000" 69) "migrate-speed" 70) "4096" 71) "migrate-type" 72) "redis-command" 73) "persist-cluster-nodes-enabled" 74) "yes" 75) "pidfile" 76) "/var/run/kvrocks/kvrocks.pid" 77) "port" 78) "6379" 79) "profiling-sample-commands" 80) "" 81) "profiling-sample-ratio" 82) "0" 83) "profiling-sample-record-max-len" 84) "256" 85) "profiling-sample-record-threshold-ms" 86) "100" 87) "proto-max-bulk-len" 88) "536870912" 89) "purge-backup-on-fullsync" 90) "no" 91) "redis-cursor-compatible" 92) "yes" 93) "repl-namespace-enabled" 94) "no" 95) "replica-announce-ip" 96) "" 97) "replica-announce-port" 98) "0" 99) "replicaof"100) ""101) "replication-connect-timeout-ms"102) "3100"103) "replication-recv-timeout-ms"104) "3200"105) "requirepass"106) ""107) "resp3-enabled"108) "no"109) "rocksdb.avoid_unnecessary_blocking_io"110) "yes"111) "rocksdb.blob_file_size"112) "268435456"113) "rocksdb.blob_garbage_collection_age_cutoff"114) "25"115) "rocksdb.block_cache_size"116) "256"117) "rocksdb.block_cache_type"118) "lru"119) "rocksdb.block_size"120) "16384"121) "rocksdb.cache_index_and_filter_blocks"122) "yes"123) "rocksdb.compaction_readahead_size"124) "2097152"125) "rocksdb.compression"126) "no"127) "rocksdb.compression_level"128) "32767"129) "rocksdb.compression_start_level"130) "2"131) "rocksdb.delayed_write_rate"132) "0"133) "rocksdb.disable_auto_compactions"134) "no"135) "rocksdb.dump_malloc_stats"136) "yes"137) "rocksdb.enable_blob_files"138) "no"139) "rocksdb.enable_blob_garbage_collection"140) "yes"141) "rocksdb.enable_pipelined_write"142) "no"143) "rocksdb.level0_file_num_compaction_trigger"144) "4"145) "rocksdb.level0_slowdown_writes_trigger"146) "20"147) "rocksdb.level0_stop_writes_trigger"148) "40"149) "rocksdb.level_compaction_dynamic_level_bytes"150) "no"151) "rocksdb.max_background_compactions"152) "2"153) "rocksdb.max_background_flushes"154) "2"155) "rocksdb.max_background_jobs"156) "4"157) "rocksdb.max_bytes_for_level_base"158) "268435456"159) "rocksdb.max_bytes_for_level_multiplier"160) "10"161) "rocksdb.max_open_files"162) "8096"163) "rocksdb.max_subcompactions"164) "2"165) "rocksdb.max_total_wal_size"166) "256"167) "rocksdb.max_write_buffer_number"168) "2"169) "rocksdb.metadata_block_cache_size"170) "2048"171) "rocksdb.min_blob_size"172) "4096"173) "rocksdb.partition_filters"174) "yes"175) "rocksdb.rate_limiter_auto_tuned"176) "yes"177) "rocksdb.read_options.async_io"178) "yes"179) "rocksdb.row_cache_size"180) "0"181) "rocksdb.share_metadata_and_subkey_block_cache"182) "yes"183) "rocksdb.stats_dump_period_sec"184) "0"185) "rocksdb.subkey_block_cache_size"186) "2048"187) "rocksdb.target_file_size_base"188) "128"189) "rocksdb.wal_compression"190) "no"191) "rocksdb.wal_size_limit_mb"192) "16384"193) "rocksdb.wal_ttl_seconds"194) "10800"195) "rocksdb.write_buffer_size"196) "32"197) "rocksdb.write_options.disable_wal"198) "no"199) "rocksdb.write_options.low_pri"200) "no"201) "rocksdb.write_options.memtable_insert_hint_per_batch"202) "no"203) "rocksdb.write_options.no_slowdown"204) "no"205) "rocksdb.write_options.sync"206) "no"207) "rocksdb.write_options.write_batch_max_bytes"208) "0"209) "skip-block-cache-deallocation-on-close"210) "no"211) "slave-empty-db-before-fullsync"212) "no"213) "slave-priority"214) "100"215) "slave-read-only"216) "yes"217) "slave-serve-stale-data"218) "yes"219) "slaveof"220) ""221) "slowlog-log-slower-than"222) "200000"223) "slowlog-max-len"224) "128"225) "socket-fd"226) "-1"227) "supervised"228) "no"229) "tcp-backlog"230) "511"231) "timeout"232) "0"233) "tls-auth-clients"234) ""235) "tls-ca-cert-dir"236) ""237) "tls-ca-cert-file"238) ""239) "tls-cert-file"240) ""241) "tls-ciphers"242) ""243) "tls-ciphersuites"244) ""245) "tls-key-file"246) ""247) "tls-key-file-pass"248) ""249) "tls-port"250) "0"251) "tls-prefer-server-ciphers"252) "no"253) "tls-protocols"254) ""255) "tls-replication"256) "no"257) "tls-session-cache-size"258) "20480"259) "tls-session-cache-timeout"260) "300"261) "tls-session-caching"262) "yes"263) "txn-context-enabled"264) "no"265) "unixsocket"266) ""267) "unixsocketperm"268) "777"269) "use-rsid-psync"270) "no"271) "workers"272) "8"
Quan
Vào ngày thg 4 21, 2025 1:52 SA, từ Benoit Tellier <bt...@linagora.com>Careful
Formating sucks
Maybe resent it as a plain/text version to avoid this pitfall?
--
Best regards,
Benoit TELLIER
General manager of Linagora VIETNAM.
Product owner for Team-Mail product.
Chairman of the Apache James project.
Mail: btellier@linagora.com
Tel: (0033) 6 77 26 04 58 (WhatsApp, Signal)
On Apr 20, 2025 3:43 PM, from Hong Quan Tran <hq...@linagora.com>Hi Hulk,
> Thanks for your information, could you please provide the output by running the command in redis-cli `config get *`?
Please see the output below.
kvrocks@561e141f9b72:/$ redis-cli -p 6379 CONFIG GET '*' 1) "auto-resize-block-and-sst" 2) "yes" 3) "backup-dir" 4) "/var/lib/kvrocks/backup" 5) "bgsave-cron" 6) "" 7) "bind" 8) "0.0.0.0" 9) "cluster-enabled" 10) "no" 11) "compact-cron" 12) "" 13) "compaction-checker-cron" 14) "" 15) "compaction-checker-range" 16) "" 17) "daemonize" 18) "no" 19) "db-name" 20) "change.me.db" 21) "dbsize-scan-cron" 22) "" 23) "dir" 24) "/var/lib/kvrocks" 25) "force-compact-file-age" 26) "172800" 27) "force-compact-file-min-deleted-percentage" 28) "10" 29) "fullsync-recv-file-delay" 30) "0" 31) "histogram-bucket-boundaries" 32) "" 33) "json-max-nesting-depth" 34) "1024" 35) "json-storage-format" 36) "json" 37) "log-dir" 38) "/var/lib/kvrocks" 39) "log-level" 40) "info" 41) "log-retention-days" 42) "-1" 43) "master-use-repl-port" 44) "no" 45) "masterauth" 46) "" 47) "max-backup-keep-hours" 48) "0" 49) "max-backup-to-keep" 50) "1" 51) "max-bitmap-to-string-mb" 52) "16" 53) "max-db-size" 54) "0" 55) "max-io-mb" 56) "0" 57) "max-replication-mb" 58) "0" 59) "maxclients" 60) "10240" 61) "migrate-batch-rate-limit-mb" 62) "16" 63) "migrate-batch-size-kb" 64) "16" 65) "migrate-pipeline-size" 66) "16" 67) "migrate-sequence-gap" 68) "10000" 69) "migrate-speed" 70) "4096" 71) "migrate-type" 72) "redis-command" 73) "persist-cluster-nodes-enabled" 74) "yes" 75) "pidfile" 76) "/var/run/kvrocks/kvrocks.pid" 77) "port" 78) "6379" 79) "profiling-sample-commands" 80) "" 81) "profiling-sample-ratio" 82) "0" 83) "profiling-sample-record-max-len" 84) "256" 85) "profiling-sample-record-threshold-ms" 86) "100" 87) "proto-max-bulk-len" 88) "536870912" 89) "purge-backup-on-fullsync" 90) "no" 91) "redis-cursor-compatible" 92) "yes" 93) "repl-namespace-enabled" 94) "no" 95) "replica-announce-ip" 96) "" 97) "replica-announce-port" 98) "0" 99) "replicaof"100) ""101) "replication-connect-timeout-ms"102) "3100"103) "replication-recv-timeout-ms"104) "3200"105) "requirepass"106) ""107) "resp3-enabled"108) "no"109) "rocksdb.avoid_unnecessary_blocking_io"110) "yes"111) "rocksdb.blob_file_size"112) "268435456"113) "rocksdb.blob_garbage_collection_age_cutoff"114) "25"115) "rocksdb.block_cache_size"116) "256"117) "rocksdb.block_cache_type"118) "lru"119) "rocksdb.block_size"120) "16384"121) "rocksdb.cache_index_and_filter_blocks"122) "yes"123) "rocksdb.compaction_readahead_size"124) "2097152"125) "rocksdb.compression"126) "no"127) "rocksdb.compression_level"128) "32767"129) "rocksdb.compression_start_level"130) "2"131) "rocksdb.delayed_write_rate"132) "0"133) "rocksdb.disable_auto_compactions"134) "no"135) "rocksdb.dump_malloc_stats"136) "yes"137) "rocksdb.enable_blob_files"138) "no"139) "rocksdb.enable_blob_garbage_collection"140) "yes"141) "rocksdb.enable_pipelined_write"142) "no"143) "rocksdb.level0_file_num_compaction_trigger"144) "4"145) "rocksdb.level0_slowdown_writes_trigger"146) "20"147) "rocksdb.level0_stop_writes_trigger"148) "40"149) "rocksdb.level_compaction_dynamic_level_bytes"150) "no"151) "rocksdb.max_background_compactions"152) "2"153) "rocksdb.max_background_flushes"154) "2"155) "rocksdb.max_background_jobs"156) "4"157) "rocksdb.max_bytes_for_level_base"158) "268435456"159) "rocksdb.max_bytes_for_level_multiplier"160) "10"161) "rocksdb.max_open_files"162) "8096"163) "rocksdb.max_subcompactions"164) "2"165) "rocksdb.max_total_wal_size"166) "256"167) "rocksdb.max_write_buffer_number"168) "2"169) "rocksdb.metadata_block_cache_size"170) "2048"171) "rocksdb.min_blob_size"172) "4096"173) "rocksdb.partition_filters"174) "yes"175) "rocksdb.rate_limiter_auto_tuned"176) "yes"177) "rocksdb.read_options.async_io"178) "yes"179) "rocksdb.row_cache_size"180) "0"181) "rocksdb.share_metadata_and_subkey_block_cache"182) "yes"183) "rocksdb.stats_dump_period_sec"184) "0"185) "rocksdb.subkey_block_cache_size"186) "2048"187) "rocksdb.target_file_size_base"188) "128"189) "rocksdb.wal_compression"190) "no"191) "rocksdb.wal_size_limit_mb"192) "16384"193) "rocksdb.wal_ttl_seconds"194) "10800"195) "rocksdb.write_buffer_size"196) "32"197) "rocksdb.write_options.disable_wal"198) "no"199) "rocksdb.write_options.low_pri"200) "no"201) "rocksdb.write_options.memtable_insert_hint_per_batch"202) "no"203) "rocksdb.write_options.no_slowdown"204) "no"205) "rocksdb.write_options.sync"206) "no"207) "rocksdb.write_options.write_batch_max_bytes"208) "0"209) "skip-block-cache-deallocation-on-close"210) "no"211) "slave-empty-db-before-fullsync"212) "no"213) "slave-priority"214) "100"215) "slave-read-only"216) "yes"217) "slave-serve-stale-data"218) "yes"219) "slaveof"220) ""221) "slowlog-log-slower-than"222) "200000"223) "slowlog-max-len"224) "128"225) "socket-fd"226) "-1"227) "supervised"228) "no"229) "tcp-backlog"230) "511"231) "timeout"232) "0"233) "tls-auth-clients"234) ""235) "tls-ca-cert-dir"236) ""237) "tls-ca-cert-file"238) ""239) "tls-cert-file"240) ""241) "tls-ciphers"242) ""243) "tls-ciphersuites"244) ""245) "tls-key-file"246) ""247) "tls-key-file-pass"248) ""249) "tls-port"250) "0"251) "tls-prefer-server-ciphers"252) "no"253) "tls-protocols"254) ""255) "tls-replication"256) "no"257) "tls-session-cache-size"258) "20480"259) "tls-session-cache-timeout"260) "300"261) "tls-session-caching"262) "yes"263) "txn-context-enabled"264) "no"265) "unixsocket"266) ""267) "unixsocketperm"268) "777"269) "use-rsid-psync"270) "no"271) "workers"272) "8"
Vào ngày thg 4 18, 2025 5:49 CH, từ Hulk <hu...@gmail.com>Hi Quan,
Thanks for your information, could you please provide the output by running
the command in redis-cli `config get *`?
On Fri, 18 Apr 2025 at 18:24, Hong Quan TRAN <hq...@linagora.com> wrote:
> Hi the Kvrocks community,
> We would like to update you our progress on the integration topic.
> We have started to migrate a Redis deployment to Kvrocks for a week. The
> Kvrocks is running stable I would say. However, we have concern after
> observing the Kvrocks's memory comsuption as it keeps slowly consume more
> memory without releasing them back to the OS. At first the memory usage was
> 500MB, then it is up to 700MB, 1GB, 1.3GB...
> Please note that we already tuning the memory usage using the
> recommendations at github.com/apache/kvrocks/discussions/2767.
> ```kvrocks.conflog-level
> <github.com/apache/kvrocks/discussions/2767.kvrocks.conflog-level>
> infoport 6379rocksdb.max_total_wal_size 256rocksdb.max_write_buffer_number
> 2rocksdb.write_buffer_size 32rocksdb.block_cache_size 256```
> We expect the memory consumption would be low and stable with this applied
> configuration. However, it keeps slowly increasing, which makes us worry.
> Our memory stats so far:
> ```docker statsCONTAINER ID NAME CPU % MEM USAGE /
> LIMIT MEM % NET I/O BLOCK I/O PIDS561e141f9b72
> kvrocks 0.14% 1.292GiB / 14.3GiB 9.03% 4.77GB /
> 1.51GB 270kB / 8.08GB 19```
> ```kvrocks@561e141f9b72:/$ redis-cli -p 6379 info memory#
> Memoryused_memory_rss:983244800used_memory_rss_human:937.70Mused_memory_lua:59392used_memory_lua_human:58.00Kused_memory_startup:63860736```
> FYI we are using Kvrocks version 2.11.1.
> Could you help provide insights and advice on the Kvrocks memory
> management topic? How to limit the Kvrocks memory usage effectively?
>
> Best regards,
> Quan
>
>
--
Best Regards,
- *Hulk Lin*
Re: Apache James meets KVRocks
Posted by Hong Quan TRAN <hq...@linagora.com>.
Hi everyone,
Sorry for the bad formatting output.
I resend the config output. I hope it is fine this time.
kvrocks@561e141f9b72:/$ redis-cli -p 6379 CONFIG GET '*' 1) "auto-resize-block-and-sst" 2) "yes" 3) "backup-dir" 4) "/var/lib/kvrocks/backup" 5) "bgsave-cron" 6) "" 7) "bind" 8) "0.0.0.0" 9) "cluster-enabled" 10) "no" 11) "compact-cron" 12) "" 13) "compaction-checker-cron" 14) "" 15) "compaction-checker-range" 16) "" 17) "daemonize" 18) "no" 19) "db-name" 20) "change.me.db" 21) "dbsize-scan-cron" 22) "" 23) "dir" 24) "/var/lib/kvrocks" 25) "force-compact-file-age" 26) "172800" 27) "force-compact-file-min-deleted-percentage" 28) "10" 29) "fullsync-recv-file-delay" 30) "0" 31) "histogram-bucket-boundaries" 32) "" 33) "json-max-nesting-depth" 34) "1024" 35) "json-storage-format" 36) "json" 37) "log-dir" 38) "/var/lib/kvrocks" 39) "log-level" 40) "info" 41) "log-retention-days" 42) "-1" 43) "master-use-repl-port" 44) "no" 45) "masterauth" 46) "" 47) "max-backup-keep-hours" 48) "0" 49) "max-backup-to-keep" 50) "1" 51) "max-bitmap-to-string-mb" 52) "16" 53) "max-db-size" 54) "0" 55) "max-io-mb" 56) "0" 57) "max-replication-mb" 58) "0" 59) "maxclients" 60) "10240" 61) "migrate-batch-rate-limit-mb" 62) "16" 63) "migrate-batch-size-kb" 64) "16" 65) "migrate-pipeline-size" 66) "16" 67) "migrate-sequence-gap" 68) "10000" 69) "migrate-speed" 70) "4096" 71) "migrate-type" 72) "redis-command" 73) "persist-cluster-nodes-enabled" 74) "yes" 75) "pidfile" 76) "/var/run/kvrocks/kvrocks.pid" 77) "port" 78) "6379" 79) "profiling-sample-commands" 80) "" 81) "profiling-sample-ratio" 82) "0" 83) "profiling-sample-record-max-len" 84) "256" 85) "profiling-sample-record-threshold-ms" 86) "100" 87) "proto-max-bulk-len" 88) "536870912" 89) "purge-backup-on-fullsync" 90) "no" 91) "redis-cursor-compatible" 92) "yes" 93) "repl-namespace-enabled" 94) "no" 95) "replica-announce-ip" 96) "" 97) "replica-announce-port" 98) "0" 99) "replicaof"100) ""101) "replication-connect-timeout-ms"102) "3100"103) "replication-recv-timeout-ms"104) "3200"105) "requirepass"106) ""107) "resp3-enabled"108) "no"109) "rocksdb.avoid_unnecessary_blocking_io"110) "yes"111) "rocksdb.blob_file_size"112) "268435456"113) "rocksdb.blob_garbage_collection_age_cutoff"114) "25"115) "rocksdb.block_cache_size"116) "256"117) "rocksdb.block_cache_type"118) "lru"119) "rocksdb.block_size"120) "16384"121) "rocksdb.cache_index_and_filter_blocks"122) "yes"123) "rocksdb.compaction_readahead_size"124) "2097152"125) "rocksdb.compression"126) "no"127) "rocksdb.compression_level"128) "32767"129) "rocksdb.compression_start_level"130) "2"131) "rocksdb.delayed_write_rate"132) "0"133) "rocksdb.disable_auto_compactions"134) "no"135) "rocksdb.dump_malloc_stats"136) "yes"137) "rocksdb.enable_blob_files"138) "no"139) "rocksdb.enable_blob_garbage_collection"140) "yes"141) "rocksdb.enable_pipelined_write"142) "no"143) "rocksdb.level0_file_num_compaction_trigger"144) "4"145) "rocksdb.level0_slowdown_writes_trigger"146) "20"147) "rocksdb.level0_stop_writes_trigger"148) "40"149) "rocksdb.level_compaction_dynamic_level_bytes"150) "no"151) "rocksdb.max_background_compactions"152) "2"153) "rocksdb.max_background_flushes"154) "2"155) "rocksdb.max_background_jobs"156) "4"157) "rocksdb.max_bytes_for_level_base"158) "268435456"159) "rocksdb.max_bytes_for_level_multiplier"160) "10"161) "rocksdb.max_open_files"162) "8096"163) "rocksdb.max_subcompactions"164) "2"165) "rocksdb.max_total_wal_size"166) "256"167) "rocksdb.max_write_buffer_number"168) "2"169) "rocksdb.metadata_block_cache_size"170) "2048"171) "rocksdb.min_blob_size"172) "4096"173) "rocksdb.partition_filters"174) "yes"175) "rocksdb.rate_limiter_auto_tuned"176) "yes"177) "rocksdb.read_options.async_io"178) "yes"179) "rocksdb.row_cache_size"180) "0"181) "rocksdb.share_metadata_and_subkey_block_cache"182) "yes"183) "rocksdb.stats_dump_period_sec"184) "0"185) "rocksdb.subkey_block_cache_size"186) "2048"187) "rocksdb.target_file_size_base"188) "128"189) "rocksdb.wal_compression"190) "no"191) "rocksdb.wal_size_limit_mb"192) "16384"193) "rocksdb.wal_ttl_seconds"194) "10800"195) "rocksdb.write_buffer_size"196) "32"197) "rocksdb.write_options.disable_wal"198) "no"199) "rocksdb.write_options.low_pri"200) "no"201) "rocksdb.write_options.memtable_insert_hint_per_batch"202) "no"203) "rocksdb.write_options.no_slowdown"204) "no"205) "rocksdb.write_options.sync"206) "no"207) "rocksdb.write_options.write_batch_max_bytes"208) "0"209) "skip-block-cache-deallocation-on-close"210) "no"211) "slave-empty-db-before-fullsync"212) "no"213) "slave-priority"214) "100"215) "slave-read-only"216) "yes"217) "slave-serve-stale-data"218) "yes"219) "slaveof"220) ""221) "slowlog-log-slower-than"222) "200000"223) "slowlog-max-len"224) "128"225) "socket-fd"226) "-1"227) "supervised"228) "no"229) "tcp-backlog"230) "511"231) "timeout"232) "0"233) "tls-auth-clients"234) ""235) "tls-ca-cert-dir"236) ""237) "tls-ca-cert-file"238) ""239) "tls-cert-file"240) ""241) "tls-ciphers"242) ""243) "tls-ciphersuites"244) ""245) "tls-key-file"246) ""247) "tls-key-file-pass"248) ""249) "tls-port"250) "0"251) "tls-prefer-server-ciphers"252) "no"253) "tls-protocols"254) ""255) "tls-replication"256) "no"257) "tls-session-cache-size"258) "20480"259) "tls-session-cache-timeout"260) "300"261) "tls-session-caching"262) "yes"263) "txn-context-enabled"264) "no"265) "unixsocket"266) ""267) "unixsocketperm"268) "777"269) "use-rsid-psync"270) "no"271) "workers"272) "8"
Quan
Vào ngày thg 4 21, 2025 1:52 SA, từ Benoit Tellier <bt...@linagora.com>Careful
Formating sucks
Maybe resent it as a plain/text version to avoid this pitfall?
--
Best regards,
Benoit TELLIER
General manager of Linagora VIETNAM.
Product owner for Team-Mail product.
Chairman of the Apache James project.
Mail: btellier@linagora.com
Tel: (0033) 6 77 26 04 58 (WhatsApp, Signal)
On Apr 20, 2025 3:43 PM, from Hong Quan Tran <hq...@linagora.com>Hi Hulk,
> Thanks for your information, could you please provide the output by running the command in redis-cli `config get *`?
Please see the output below.
kvrocks@561e141f9b72:/$ redis-cli -p 6379 CONFIG GET '*' 1) "auto-resize-block-and-sst" 2) "yes" 3) "backup-dir" 4) "/var/lib/kvrocks/backup" 5) "bgsave-cron" 6) "" 7) "bind" 8) "0.0.0.0" 9) "cluster-enabled" 10) "no" 11) "compact-cron" 12) "" 13) "compaction-checker-cron" 14) "" 15) "compaction-checker-range" 16) "" 17) "daemonize" 18) "no" 19) "db-name" 20) "change.me.db" 21) "dbsize-scan-cron" 22) "" 23) "dir" 24) "/var/lib/kvrocks" 25) "force-compact-file-age" 26) "172800" 27) "force-compact-file-min-deleted-percentage" 28) "10" 29) "fullsync-recv-file-delay" 30) "0" 31) "histogram-bucket-boundaries" 32) "" 33) "json-max-nesting-depth" 34) "1024" 35) "json-storage-format" 36) "json" 37) "log-dir" 38) "/var/lib/kvrocks" 39) "log-level" 40) "info" 41) "log-retention-days" 42) "-1" 43) "master-use-repl-port" 44) "no" 45) "masterauth" 46) "" 47) "max-backup-keep-hours" 48) "0" 49) "max-backup-to-keep" 50) "1" 51) "max-bitmap-to-string-mb" 52) "16" 53) "max-db-size" 54) "0" 55) "max-io-mb" 56) "0" 57) "max-replication-mb" 58) "0" 59) "maxclients" 60) "10240" 61) "migrate-batch-rate-limit-mb" 62) "16" 63) "migrate-batch-size-kb" 64) "16" 65) "migrate-pipeline-size" 66) "16" 67) "migrate-sequence-gap" 68) "10000" 69) "migrate-speed" 70) "4096" 71) "migrate-type" 72) "redis-command" 73) "persist-cluster-nodes-enabled" 74) "yes" 75) "pidfile" 76) "/var/run/kvrocks/kvrocks.pid" 77) "port" 78) "6379" 79) "profiling-sample-commands" 80) "" 81) "profiling-sample-ratio" 82) "0" 83) "profiling-sample-record-max-len" 84) "256" 85) "profiling-sample-record-threshold-ms" 86) "100" 87) "proto-max-bulk-len" 88) "536870912" 89) "purge-backup-on-fullsync" 90) "no" 91) "redis-cursor-compatible" 92) "yes" 93) "repl-namespace-enabled" 94) "no" 95) "replica-announce-ip" 96) "" 97) "replica-announce-port" 98) "0" 99) "replicaof"100) ""101) "replication-connect-timeout-ms"102) "3100"103) "replication-recv-timeout-ms"104) "3200"105) "requirepass"106) ""107) "resp3-enabled"108) "no"109) "rocksdb.avoid_unnecessary_blocking_io"110) "yes"111) "rocksdb.blob_file_size"112) "268435456"113) "rocksdb.blob_garbage_collection_age_cutoff"114) "25"115) "rocksdb.block_cache_size"116) "256"117) "rocksdb.block_cache_type"118) "lru"119) "rocksdb.block_size"120) "16384"121) "rocksdb.cache_index_and_filter_blocks"122) "yes"123) "rocksdb.compaction_readahead_size"124) "2097152"125) "rocksdb.compression"126) "no"127) "rocksdb.compression_level"128) "32767"129) "rocksdb.compression_start_level"130) "2"131) "rocksdb.delayed_write_rate"132) "0"133) "rocksdb.disable_auto_compactions"134) "no"135) "rocksdb.dump_malloc_stats"136) "yes"137) "rocksdb.enable_blob_files"138) "no"139) "rocksdb.enable_blob_garbage_collection"140) "yes"141) "rocksdb.enable_pipelined_write"142) "no"143) "rocksdb.level0_file_num_compaction_trigger"144) "4"145) "rocksdb.level0_slowdown_writes_trigger"146) "20"147) "rocksdb.level0_stop_writes_trigger"148) "40"149) "rocksdb.level_compaction_dynamic_level_bytes"150) "no"151) "rocksdb.max_background_compactions"152) "2"153) "rocksdb.max_background_flushes"154) "2"155) "rocksdb.max_background_jobs"156) "4"157) "rocksdb.max_bytes_for_level_base"158) "268435456"159) "rocksdb.max_bytes_for_level_multiplier"160) "10"161) "rocksdb.max_open_files"162) "8096"163) "rocksdb.max_subcompactions"164) "2"165) "rocksdb.max_total_wal_size"166) "256"167) "rocksdb.max_write_buffer_number"168) "2"169) "rocksdb.metadata_block_cache_size"170) "2048"171) "rocksdb.min_blob_size"172) "4096"173) "rocksdb.partition_filters"174) "yes"175) "rocksdb.rate_limiter_auto_tuned"176) "yes"177) "rocksdb.read_options.async_io"178) "yes"179) "rocksdb.row_cache_size"180) "0"181) "rocksdb.share_metadata_and_subkey_block_cache"182) "yes"183) "rocksdb.stats_dump_period_sec"184) "0"185) "rocksdb.subkey_block_cache_size"186) "2048"187) "rocksdb.target_file_size_base"188) "128"189) "rocksdb.wal_compression"190) "no"191) "rocksdb.wal_size_limit_mb"192) "16384"193) "rocksdb.wal_ttl_seconds"194) "10800"195) "rocksdb.write_buffer_size"196) "32"197) "rocksdb.write_options.disable_wal"198) "no"199) "rocksdb.write_options.low_pri"200) "no"201) "rocksdb.write_options.memtable_insert_hint_per_batch"202) "no"203) "rocksdb.write_options.no_slowdown"204) "no"205) "rocksdb.write_options.sync"206) "no"207) "rocksdb.write_options.write_batch_max_bytes"208) "0"209) "skip-block-cache-deallocation-on-close"210) "no"211) "slave-empty-db-before-fullsync"212) "no"213) "slave-priority"214) "100"215) "slave-read-only"216) "yes"217) "slave-serve-stale-data"218) "yes"219) "slaveof"220) ""221) "slowlog-log-slower-than"222) "200000"223) "slowlog-max-len"224) "128"225) "socket-fd"226) "-1"227) "supervised"228) "no"229) "tcp-backlog"230) "511"231) "timeout"232) "0"233) "tls-auth-clients"234) ""235) "tls-ca-cert-dir"236) ""237) "tls-ca-cert-file"238) ""239) "tls-cert-file"240) ""241) "tls-ciphers"242) ""243) "tls-ciphersuites"244) ""245) "tls-key-file"246) ""247) "tls-key-file-pass"248) ""249) "tls-port"250) "0"251) "tls-prefer-server-ciphers"252) "no"253) "tls-protocols"254) ""255) "tls-replication"256) "no"257) "tls-session-cache-size"258) "20480"259) "tls-session-cache-timeout"260) "300"261) "tls-session-caching"262) "yes"263) "txn-context-enabled"264) "no"265) "unixsocket"266) ""267) "unixsocketperm"268) "777"269) "use-rsid-psync"270) "no"271) "workers"272) "8"
Vào ngày thg 4 18, 2025 5:49 CH, từ Hulk <hu...@gmail.com>Hi Quan,
Thanks for your information, could you please provide the output by running
the command in redis-cli `config get *`?
On Fri, 18 Apr 2025 at 18:24, Hong Quan TRAN <hq...@linagora.com> wrote:
> Hi the Kvrocks community,
> We would like to update you our progress on the integration topic.
> We have started to migrate a Redis deployment to Kvrocks for a week. The
> Kvrocks is running stable I would say. However, we have concern after
> observing the Kvrocks's memory comsuption as it keeps slowly consume more
> memory without releasing them back to the OS. At first the memory usage was
> 500MB, then it is up to 700MB, 1GB, 1.3GB...
> Please note that we already tuning the memory usage using the
> recommendations at github.com/apache/kvrocks/discussions/2767.
> ```kvrocks.conflog-level
> <github.com/apache/kvrocks/discussions/2767.kvrocks.conflog-level>
> infoport 6379rocksdb.max_total_wal_size 256rocksdb.max_write_buffer_number
> 2rocksdb.write_buffer_size 32rocksdb.block_cache_size 256```
> We expect the memory consumption would be low and stable with this applied
> configuration. However, it keeps slowly increasing, which makes us worry.
> Our memory stats so far:
> ```docker statsCONTAINER ID NAME CPU % MEM USAGE /
> LIMIT MEM % NET I/O BLOCK I/O PIDS561e141f9b72
> kvrocks 0.14% 1.292GiB / 14.3GiB 9.03% 4.77GB /
> 1.51GB 270kB / 8.08GB 19```
> ```kvrocks@561e141f9b72:/$ redis-cli -p 6379 info memory#
> Memoryused_memory_rss:983244800used_memory_rss_human:937.70Mused_memory_lua:59392used_memory_lua_human:58.00Kused_memory_startup:63860736```
> FYI we are using Kvrocks version 2.11.1.
> Could you help provide insights and advice on the Kvrocks memory
> management topic? How to limit the Kvrocks memory usage effectively?
>
> Best regards,
> Quan
>
>
--
Best Regards,
- *Hulk Lin*
Re: Apache James meets KVRocks
Posted by Benoit TELLIER <bt...@linagora.com>.
Careful
Formating sucks
Maybe resent it as a plain/text version to avoid this pitfall?
--
Best regards,
Benoit TELLIER
General manager of Linagora VIETNAM.
Product owner for Team-Mail product.
Chairman of the Apache James project.
Mail: btellier@linagora.com
Tel: (0033) 6 77 26 04 58 (WhatsApp, Signal)
On Apr 20, 2025 3:43 PM, from Hong Quan Tran <hq...@linagora.com>Hi Hulk,
> Thanks for your information, could you please provide the output by running the command in redis-cli `config get *`?
Please see the output below.
kvrocks@561e141f9b72:/$ redis-cli -p 6379 CONFIG GET '*' 1) "auto-resize-block-and-sst" 2) "yes" 3) "backup-dir" 4) "/var/lib/kvrocks/backup" 5) "bgsave-cron" 6) "" 7) "bind" 8) "0.0.0.0" 9) "cluster-enabled" 10) "no" 11) "compact-cron" 12) "" 13) "compaction-checker-cron" 14) "" 15) "compaction-checker-range" 16) "" 17) "daemonize" 18) "no" 19) "db-name" 20) "change.me.db" 21) "dbsize-scan-cron" 22) "" 23) "dir" 24) "/var/lib/kvrocks" 25) "force-compact-file-age" 26) "172800" 27) "force-compact-file-min-deleted-percentage" 28) "10" 29) "fullsync-recv-file-delay" 30) "0" 31) "histogram-bucket-boundaries" 32) "" 33) "json-max-nesting-depth" 34) "1024" 35) "json-storage-format" 36) "json" 37) "log-dir" 38) "/var/lib/kvrocks" 39) "log-level" 40) "info" 41) "log-retention-days" 42) "-1" 43) "master-use-repl-port" 44) "no" 45) "masterauth" 46) "" 47) "max-backup-keep-hours" 48) "0" 49) "max-backup-to-keep" 50) "1" 51) "max-bitmap-to-string-mb" 52) "16" 53) "max-db-size" 54) "0" 55) "max-io-mb" 56) "0" 57) "max-replication-mb" 58) "0" 59) "maxclients" 60) "10240" 61) "migrate-batch-rate-limit-mb" 62) "16" 63) "migrate-batch-size-kb" 64) "16" 65) "migrate-pipeline-size" 66) "16" 67) "migrate-sequence-gap" 68) "10000" 69) "migrate-speed" 70) "4096" 71) "migrate-type" 72) "redis-command" 73) "persist-cluster-nodes-enabled" 74) "yes" 75) "pidfile" 76) "/var/run/kvrocks/kvrocks.pid" 77) "port" 78) "6379" 79) "profiling-sample-commands" 80) "" 81) "profiling-sample-ratio" 82) "0" 83) "profiling-sample-record-max-len" 84) "256" 85) "profiling-sample-record-threshold-ms" 86) "100" 87) "proto-max-bulk-len" 88) "536870912" 89) "purge-backup-on-fullsync" 90) "no" 91) "redis-cursor-compatible" 92) "yes" 93) "repl-namespace-enabled" 94) "no" 95) "replica-announce-ip" 96) "" 97) "replica-announce-port" 98) "0" 99) "replicaof"100) ""101) "replication-connect-timeout-ms"102) "3100"103) "replication-recv-timeout-ms"104) "3200"105) "requirepass"106) ""107) "resp3-enabled"108) "no"109) "rocksdb.avoid_unnecessary_blocking_io"110) "yes"111) "rocksdb.blob_file_size"112) "268435456"113) "rocksdb.blob_garbage_collection_age_cutoff"114) "25"115) "rocksdb.block_cache_size"116) "256"117) "rocksdb.block_cache_type"118) "lru"119) "rocksdb.block_size"120) "16384"121) "rocksdb.cache_index_and_filter_blocks"122) "yes"123) "rocksdb.compaction_readahead_size"124) "2097152"125) "rocksdb.compression"126) "no"127) "rocksdb.compression_level"128) "32767"129) "rocksdb.compression_start_level"130) "2"131) "rocksdb.delayed_write_rate"132) "0"133) "rocksdb.disable_auto_compactions"134) "no"135) "rocksdb.dump_malloc_stats"136) "yes"137) "rocksdb.enable_blob_files"138) "no"139) "rocksdb.enable_blob_garbage_collection"140) "yes"141) "rocksdb.enable_pipelined_write"142) "no"143) "rocksdb.level0_file_num_compaction_trigger"144) "4"145) "rocksdb.level0_slowdown_writes_trigger"146) "20"147) "rocksdb.level0_stop_writes_trigger"148) "40"149) "rocksdb.level_compaction_dynamic_level_bytes"150) "no"151) "rocksdb.max_background_compactions"152) "2"153) "rocksdb.max_background_flushes"154) "2"155) "rocksdb.max_background_jobs"156) "4"157) "rocksdb.max_bytes_for_level_base"158) "268435456"159) "rocksdb.max_bytes_for_level_multiplier"160) "10"161) "rocksdb.max_open_files"162) "8096"163) "rocksdb.max_subcompactions"164) "2"165) "rocksdb.max_total_wal_size"166) "256"167) "rocksdb.max_write_buffer_number"168) "2"169) "rocksdb.metadata_block_cache_size"170) "2048"171) "rocksdb.min_blob_size"172) "4096"173) "rocksdb.partition_filters"174) "yes"175) "rocksdb.rate_limiter_auto_tuned"176) "yes"177) "rocksdb.read_options.async_io"178) "yes"179) "rocksdb.row_cache_size"180) "0"181) "rocksdb.share_metadata_and_subkey_block_cache"182) "yes"183) "rocksdb.stats_dump_period_sec"184) "0"185) "rocksdb.subkey_block_cache_size"186) "2048"187) "rocksdb.target_file_size_base"188) "128"189) "rocksdb.wal_compression"190) "no"191) "rocksdb.wal_size_limit_mb"192) "16384"193) "rocksdb.wal_ttl_seconds"194) "10800"195) "rocksdb.write_buffer_size"196) "32"197) "rocksdb.write_options.disable_wal"198) "no"199) "rocksdb.write_options.low_pri"200) "no"201) "rocksdb.write_options.memtable_insert_hint_per_batch"202) "no"203) "rocksdb.write_options.no_slowdown"204) "no"205) "rocksdb.write_options.sync"206) "no"207) "rocksdb.write_options.write_batch_max_bytes"208) "0"209) "skip-block-cache-deallocation-on-close"210) "no"211) "slave-empty-db-before-fullsync"212) "no"213) "slave-priority"214) "100"215) "slave-read-only"216) "yes"217) "slave-serve-stale-data"218) "yes"219) "slaveof"220) ""221) "slowlog-log-slower-than"222) "200000"223) "slowlog-max-len"224) "128"225) "socket-fd"226) "-1"227) "supervised"228) "no"229) "tcp-backlog"230) "511"231) "timeout"232) "0"233) "tls-auth-clients"234) ""235) "tls-ca-cert-dir"236) ""237) "tls-ca-cert-file"238) ""239) "tls-cert-file"240) ""241) "tls-ciphers"242) ""243) "tls-ciphersuites"244) ""245) "tls-key-file"246) ""247) "tls-key-file-pass"248) ""249) "tls-port"250) "0"251) "tls-prefer-server-ciphers"252) "no"253) "tls-protocols"254) ""255) "tls-replication"256) "no"257) "tls-session-cache-size"258) "20480"259) "tls-session-cache-timeout"260) "300"261) "tls-session-caching"262) "yes"263) "txn-context-enabled"264) "no"265) "unixsocket"266) ""267) "unixsocketperm"268) "777"269) "use-rsid-psync"270) "no"271) "workers"272) "8"
Vào ngày thg 4 18, 2025 5:49 CH, từ Hulk <hu...@gmail.com>Hi Quan,
Thanks for your information, could you please provide the output by running
the command in redis-cli `config get *`?
On Fri, 18 Apr 2025 at 18:24, Hong Quan TRAN <hq...@linagora.com> wrote:
> Hi the Kvrocks community,
> We would like to update you our progress on the integration topic.
> We have started to migrate a Redis deployment to Kvrocks for a week. The
> Kvrocks is running stable I would say. However, we have concern after
> observing the Kvrocks's memory comsuption as it keeps slowly consume more
> memory without releasing them back to the OS. At first the memory usage was
> 500MB, then it is up to 700MB, 1GB, 1.3GB...
> Please note that we already tuning the memory usage using the
> recommendations at github.com/apache/kvrocks/discussions/2767.
> ```kvrocks.conflog-level
> <github.com/apache/kvrocks/discussions/2767.kvrocks.conflog-level>
> infoport 6379rocksdb.max_total_wal_size 256rocksdb.max_write_buffer_number
> 2rocksdb.write_buffer_size 32rocksdb.block_cache_size 256```
> We expect the memory consumption would be low and stable with this applied
> configuration. However, it keeps slowly increasing, which makes us worry.
> Our memory stats so far:
> ```docker statsCONTAINER ID NAME CPU % MEM USAGE /
> LIMIT MEM % NET I/O BLOCK I/O PIDS561e141f9b72
> kvrocks 0.14% 1.292GiB / 14.3GiB 9.03% 4.77GB /
> 1.51GB 270kB / 8.08GB 19```
> ```kvrocks@561e141f9b72:/$ redis-cli -p 6379 info memory#
> Memoryused_memory_rss:983244800used_memory_rss_human:937.70Mused_memory_lua:59392used_memory_lua_human:58.00Kused_memory_startup:63860736```
> FYI we are using Kvrocks version 2.11.1.
> Could you help provide insights and advice on the Kvrocks memory
> management topic? How to limit the Kvrocks memory usage effectively?
>
> Best regards,
> Quan
>
>
--
Best Regards,
- *Hulk Lin*
Re: Apache James meets KVRocks
Posted by Hong Quan TRAN <hq...@linagora.com>.
Hi Hulk,
> Thanks for your information, could you please provide the output by running the command in redis-cli `config get *`?
Please see the output below.
kvrocks@561e141f9b72:/$ redis-cli -p 6379 CONFIG GET '*' 1) "auto-resize-block-and-sst" 2) "yes" 3) "backup-dir" 4) "/var/lib/kvrocks/backup" 5) "bgsave-cron" 6) "" 7) "bind" 8) "0.0.0.0" 9) "cluster-enabled" 10) "no" 11) "compact-cron" 12) "" 13) "compaction-checker-cron" 14) "" 15) "compaction-checker-range" 16) "" 17) "daemonize" 18) "no" 19) "db-name" 20) "change.me.db" 21) "dbsize-scan-cron" 22) "" 23) "dir" 24) "/var/lib/kvrocks" 25) "force-compact-file-age" 26) "172800" 27) "force-compact-file-min-deleted-percentage" 28) "10" 29) "fullsync-recv-file-delay" 30) "0" 31) "histogram-bucket-boundaries" 32) "" 33) "json-max-nesting-depth" 34) "1024" 35) "json-storage-format" 36) "json" 37) "log-dir" 38) "/var/lib/kvrocks" 39) "log-level" 40) "info" 41) "log-retention-days" 42) "-1" 43) "master-use-repl-port" 44) "no" 45) "masterauth" 46) "" 47) "max-backup-keep-hours" 48) "0" 49) "max-backup-to-keep" 50) "1" 51) "max-bitmap-to-string-mb" 52) "16" 53) "max-db-size" 54) "0" 55) "max-io-mb" 56) "0" 57) "max-replication-mb" 58) "0" 59) "maxclients" 60) "10240" 61) "migrate-batch-rate-limit-mb" 62) "16" 63) "migrate-batch-size-kb" 64) "16" 65) "migrate-pipeline-size" 66) "16" 67) "migrate-sequence-gap" 68) "10000" 69) "migrate-speed" 70) "4096" 71) "migrate-type" 72) "redis-command" 73) "persist-cluster-nodes-enabled" 74) "yes" 75) "pidfile" 76) "/var/run/kvrocks/kvrocks.pid" 77) "port" 78) "6379" 79) "profiling-sample-commands" 80) "" 81) "profiling-sample-ratio" 82) "0" 83) "profiling-sample-record-max-len" 84) "256" 85) "profiling-sample-record-threshold-ms" 86) "100" 87) "proto-max-bulk-len" 88) "536870912" 89) "purge-backup-on-fullsync" 90) "no" 91) "redis-cursor-compatible" 92) "yes" 93) "repl-namespace-enabled" 94) "no" 95) "replica-announce-ip" 96) "" 97) "replica-announce-port" 98) "0" 99) "replicaof"100) ""101) "replication-connect-timeout-ms"102) "3100"103) "replication-recv-timeout-ms"104) "3200"105) "requirepass"106) ""107) "resp3-enabled"108) "no"109) "rocksdb.avoid_unnecessary_blocking_io"110) "yes"111) "rocksdb.blob_file_size"112) "268435456"113) "rocksdb.blob_garbage_collection_age_cutoff"114) "25"115) "rocksdb.block_cache_size"116) "256"117) "rocksdb.block_cache_type"118) "lru"119) "rocksdb.block_size"120) "16384"121) "rocksdb.cache_index_and_filter_blocks"122) "yes"123) "rocksdb.compaction_readahead_size"124) "2097152"125) "rocksdb.compression"126) "no"127) "rocksdb.compression_level"128) "32767"129) "rocksdb.compression_start_level"130) "2"131) "rocksdb.delayed_write_rate"132) "0"133) "rocksdb.disable_auto_compactions"134) "no"135) "rocksdb.dump_malloc_stats"136) "yes"137) "rocksdb.enable_blob_files"138) "no"139) "rocksdb.enable_blob_garbage_collection"140) "yes"141) "rocksdb.enable_pipelined_write"142) "no"143) "rocksdb.level0_file_num_compaction_trigger"144) "4"145) "rocksdb.level0_slowdown_writes_trigger"146) "20"147) "rocksdb.level0_stop_writes_trigger"148) "40"149) "rocksdb.level_compaction_dynamic_level_bytes"150) "no"151) "rocksdb.max_background_compactions"152) "2"153) "rocksdb.max_background_flushes"154) "2"155) "rocksdb.max_background_jobs"156) "4"157) "rocksdb.max_bytes_for_level_base"158) "268435456"159) "rocksdb.max_bytes_for_level_multiplier"160) "10"161) "rocksdb.max_open_files"162) "8096"163) "rocksdb.max_subcompactions"164) "2"165) "rocksdb.max_total_wal_size"166) "256"167) "rocksdb.max_write_buffer_number"168) "2"169) "rocksdb.metadata_block_cache_size"170) "2048"171) "rocksdb.min_blob_size"172) "4096"173) "rocksdb.partition_filters"174) "yes"175) "rocksdb.rate_limiter_auto_tuned"176) "yes"177) "rocksdb.read_options.async_io"178) "yes"179) "rocksdb.row_cache_size"180) "0"181) "rocksdb.share_metadata_and_subkey_block_cache"182) "yes"183) "rocksdb.stats_dump_period_sec"184) "0"185) "rocksdb.subkey_block_cache_size"186) "2048"187) "rocksdb.target_file_size_base"188) "128"189) "rocksdb.wal_compression"190) "no"191) "rocksdb.wal_size_limit_mb"192) "16384"193) "rocksdb.wal_ttl_seconds"194) "10800"195) "rocksdb.write_buffer_size"196) "32"197) "rocksdb.write_options.disable_wal"198) "no"199) "rocksdb.write_options.low_pri"200) "no"201) "rocksdb.write_options.memtable_insert_hint_per_batch"202) "no"203) "rocksdb.write_options.no_slowdown"204) "no"205) "rocksdb.write_options.sync"206) "no"207) "rocksdb.write_options.write_batch_max_bytes"208) "0"209) "skip-block-cache-deallocation-on-close"210) "no"211) "slave-empty-db-before-fullsync"212) "no"213) "slave-priority"214) "100"215) "slave-read-only"216) "yes"217) "slave-serve-stale-data"218) "yes"219) "slaveof"220) ""221) "slowlog-log-slower-than"222) "200000"223) "slowlog-max-len"224) "128"225) "socket-fd"226) "-1"227) "supervised"228) "no"229) "tcp-backlog"230) "511"231) "timeout"232) "0"233) "tls-auth-clients"234) ""235) "tls-ca-cert-dir"236) ""237) "tls-ca-cert-file"238) ""239) "tls-cert-file"240) ""241) "tls-ciphers"242) ""243) "tls-ciphersuites"244) ""245) "tls-key-file"246) ""247) "tls-key-file-pass"248) ""249) "tls-port"250) "0"251) "tls-prefer-server-ciphers"252) "no"253) "tls-protocols"254) ""255) "tls-replication"256) "no"257) "tls-session-cache-size"258) "20480"259) "tls-session-cache-timeout"260) "300"261) "tls-session-caching"262) "yes"263) "txn-context-enabled"264) "no"265) "unixsocket"266) ""267) "unixsocketperm"268) "777"269) "use-rsid-psync"270) "no"271) "workers"272) "8"
Vào ngày thg 4 18, 2025 5:49 CH, từ Hulk <hu...@gmail.com>Hi Quan,
Thanks for your information, could you please provide the output by running
the command in redis-cli `config get *`?
On Fri, 18 Apr 2025 at 18:24, Hong Quan TRAN <hq...@linagora.com> wrote:
> Hi the Kvrocks community,
> We would like to update you our progress on the integration topic.
> We have started to migrate a Redis deployment to Kvrocks for a week. The
> Kvrocks is running stable I would say. However, we have concern after
> observing the Kvrocks's memory comsuption as it keeps slowly consume more
> memory without releasing them back to the OS. At first the memory usage was
> 500MB, then it is up to 700MB, 1GB, 1.3GB...
> Please note that we already tuning the memory usage using the
> recommendations at github.com/apache/kvrocks/discussions/2767.
> ```kvrocks.conflog-level
> <github.com/apache/kvrocks/discussions/2767.kvrocks.conflog-level>
> infoport 6379rocksdb.max_total_wal_size 256rocksdb.max_write_buffer_number
> 2rocksdb.write_buffer_size 32rocksdb.block_cache_size 256```
> We expect the memory consumption would be low and stable with this applied
> configuration. However, it keeps slowly increasing, which makes us worry.
> Our memory stats so far:
> ```docker statsCONTAINER ID NAME CPU % MEM USAGE /
> LIMIT MEM % NET I/O BLOCK I/O PIDS561e141f9b72
> kvrocks 0.14% 1.292GiB / 14.3GiB 9.03% 4.77GB /
> 1.51GB 270kB / 8.08GB 19```
> ```kvrocks@561e141f9b72:/$ redis-cli -p 6379 info memory#
> Memoryused_memory_rss:983244800used_memory_rss_human:937.70Mused_memory_lua:59392used_memory_lua_human:58.00Kused_memory_startup:63860736```
> FYI we are using Kvrocks version 2.11.1.
> Could you help provide insights and advice on the Kvrocks memory
> management topic? How to limit the Kvrocks memory usage effectively?
>
> Best regards,
> Quan
>
>
--
Best Regards,
- *Hulk Lin*
Re: Apache James meets KVRocks
Posted by hulk <hu...@gmail.com>.
Hi Quan,
Thanks for your information, could you please provide the output by running
the command in redis-cli `config get *`?
On Fri, 18 Apr 2025 at 18:24, Hong Quan TRAN <hq...@linagora.com> wrote:
> Hi the Kvrocks community,
> We would like to update you our progress on the integration topic.
> We have started to migrate a Redis deployment to Kvrocks for a week. The
> Kvrocks is running stable I would say. However, we have concern after
> observing the Kvrocks's memory comsuption as it keeps slowly consume more
> memory without releasing them back to the OS. At first the memory usage was
> 500MB, then it is up to 700MB, 1GB, 1.3GB...
> Please note that we already tuning the memory usage using the
> recommendations at https://github.com/apache/kvrocks/discussions/2767.
> ```kvrocks.conflog-level
> <https://github.com/apache/kvrocks/discussions/2767.kvrocks.conflog-level>
> infoport 6379rocksdb.max_total_wal_size 256rocksdb.max_write_buffer_number
> 2rocksdb.write_buffer_size 32rocksdb.block_cache_size 256```
> We expect the memory consumption would be low and stable with this applied
> configuration. However, it keeps slowly increasing, which makes us worry.
> Our memory stats so far:
> ```docker statsCONTAINER ID NAME CPU % MEM USAGE /
> LIMIT MEM % NET I/O BLOCK I/O PIDS561e141f9b72
> kvrocks 0.14% 1.292GiB / 14.3GiB 9.03% 4.77GB /
> 1.51GB 270kB / 8.08GB 19```
> ```kvrocks@561e141f9b72:/$ redis-cli -p 6379 info memory#
> Memoryused_memory_rss:983244800used_memory_rss_human:937.70Mused_memory_lua:59392used_memory_lua_human:58.00Kused_memory_startup:63860736```
> FYI we are using Kvrocks version 2.11.1.
> Could you help provide insights and advice on the Kvrocks memory
> management topic? How to limit the Kvrocks memory usage effectively?
>
> Best regards,
> Quan
>
>
--
Best Regards,
- *Hulk Lin*
Re: Apache James meets KVRocks
Posted by Hong Quan TRAN <hq...@linagora.com>.
Hi the Kvrocks community,
We would like to update you our progress on the integration topic.
We have started to migrate a Redis deployment to Kvrocks for a week. The Kvrocks is running stable I would say. However, we have concern after observing the Kvrocks's memory comsuption as it keeps slowly consume more memory without releasing them back to the OS. At first the memory usage was 500MB, then it is up to 700MB, 1GB, 1.3GB...
Please note that we already tuning the memory usage using the recommendations at https://github.com/apache/kvrocks/discussions/2767.
```kvrocks.conflog-level infoport 6379rocksdb.max_total_wal_size 256rocksdb.max_write_buffer_number 2rocksdb.write_buffer_size 32rocksdb.block_cache_size 256```
We expect the memory consumption would be low and stable with this applied configuration. However, it keeps slowly increasing, which makes us worry. Our memory stats so far:
```docker statsCONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS561e141f9b72 kvrocks 0.14% 1.292GiB / 14.3GiB 9.03% 4.77GB / 1.51GB 270kB / 8.08GB 19```
```kvrocks@561e141f9b72:/$ redis-cli -p 6379 info memory# Memoryused_memory_rss:983244800used_memory_rss_human:937.70Mused_memory_lua:59392used_memory_lua_human:58.00Kused_memory_startup:63860736```
FYI we are using Kvrocks version 2.11.1.
Could you help provide insights and advice on the Kvrocks memory management topic? How to limit the Kvrocks memory usage effectively?
Best regards,
Quan
Re: Apache James meets KVRocks
Posted by Hong Quan TRAN <hq...@linagora.com>.
> In the meantime, we would create an issue on the Rspamd side to see if them can adapt the EVALSHA_RO command usage.
Please see https://github.com/rspamd/rspamd/issues/5418.
Quan
On Apr 1, 2025 10:05 AM, from Hong Quan Tran <hq...@linagora.com>Hi Hulk and Edward,
Thank you for your very warm and lightning fast feedback.
In the meantime, we would create an issue on the Rspamd side to see if them can adapt the EVALSHA_RO command usage.
Regards,
Quan
Re: Apache James meets KVRocks
Posted by Hong Quan TRAN <hq...@linagora.com>.
Hi Hulk and Edward,
Thank you for your very warm and lightning fast feedback.
In the meantime, we would create an issue on the Rspamd side to see if them can adapt the EVALSHA_RO command usage.
Regards,
Quan
Re: Apache James meets KVRocks
Posted by hulk <hu...@gmail.com>.
Hi Edward,
> Please let me know if there is anything that I could help.😊
Sure, thankfully for lending a hand.
On Mon, 3 Mar 2025 at 22:00, Twice <tw...@apache.org> wrote:
> Hi Benoit,
>
> I'm thrilled to hear that Apache James can integrate with Apache
> Kvrocks. Kvrocks supports the use cases you mentioned.
>
> If you have any issues or questions during integration, please reach out
> to us.
> Feel free to share your progress with the Kvrocks community anytime -
> we're here to help and support you.
>
> Best wishes,
> Twice
>
> On Mon, Mar 3, 2025 at 5:29 PM hulk <hu...@gmail.com> wrote:
> >
> > Hi Benoit,
> >
> > As a Kvrocks PMC member, I'm very happy to see this happens. Also, great
> > thanks to Benoit for bringing
> > the potential opportunities where the two communities could work
> together.
> >
> > On Mon, 3 Mar 2025 at 16:12, Benoit TELLIER <bt...@linagora.com>
> wrote:
> >
> > > Hello folks,
> > >
> > > I'm involved on the Apache James email server, and as part of Apache
> James
> > > we currently have a Redis integration.
> > >
> > > Redis is used for:
> > >
> > > - 1. Implementing rate limiting in a distributed fashion by using
> leaky
> > > bucket algorithm
> > > - 2. Implementing sid storage as part of our OpenID connect
> backchannel
> > > logout (integrated in Apache APISIX - java plugin runner)
> > > - 3. Use redis streams for backing apache James pub sub (~email
> > > notifications)
> > > - 4. Store bayes database for our RSPAMD setup
> > >
> > > Our setup needs to be using sentinel (some use case like 4. relies on
> > > commands not supplied from Redis cluster)
> > >
> > > Our biggest pain is that, as we use a per user database for bayes the
> > > database grows significantly and storing this DB in memory is not
> > > financially viable.
> > >
> > > In order to solve our concerns with 4. (and as we are not performance
> > > bound for Redis components) we are considering switching our use cases
> to
> > > KVrocks, and potentially makes KVrocks the default key value store /
> cache
> > > for Apache James.
> > >
> > > I (in the name of the Apache James PMC and LINAGORA contributors) would
> > > happily report on how this integrations goes on.
> > >
> > > --
> > >
> > > Best regards,
> > >
> > > Benoit TELLIER
> > >
> > > General manager of Linagora VIETNAM.
> > > Product owner for Team-Mail product.
> > > Chairman of the Apache James project.
> > >
> > > Mail: btellier@linagora.com
> > > Tel: (0033) 6 77 26 04 58 (WhatsApp, Signal)
> > >
> > >
> > >
> >
> > --
> > Best Regards,
> > - *Hulk Lin*
>
--
Best Regards,
- *Hulk Lin*
Re: Apache James meets KVRocks
Posted by Twice <tw...@apache.org>.
Hi Benoit,
I'm thrilled to hear that Apache James can integrate with Apache
Kvrocks. Kvrocks supports the use cases you mentioned.
If you have any issues or questions during integration, please reach out to us.
Feel free to share your progress with the Kvrocks community anytime -
we're here to help and support you.
Best wishes,
Twice
On Mon, Mar 3, 2025 at 5:29 PM hulk <hu...@gmail.com> wrote:
>
> Hi Benoit,
>
> As a Kvrocks PMC member, I'm very happy to see this happens. Also, great
> thanks to Benoit for bringing
> the potential opportunities where the two communities could work together.
>
> On Mon, 3 Mar 2025 at 16:12, Benoit TELLIER <bt...@linagora.com> wrote:
>
> > Hello folks,
> >
> > I'm involved on the Apache James email server, and as part of Apache James
> > we currently have a Redis integration.
> >
> > Redis is used for:
> >
> > - 1. Implementing rate limiting in a distributed fashion by using leaky
> > bucket algorithm
> > - 2. Implementing sid storage as part of our OpenID connect backchannel
> > logout (integrated in Apache APISIX - java plugin runner)
> > - 3. Use redis streams for backing apache James pub sub (~email
> > notifications)
> > - 4. Store bayes database for our RSPAMD setup
> >
> > Our setup needs to be using sentinel (some use case like 4. relies on
> > commands not supplied from Redis cluster)
> >
> > Our biggest pain is that, as we use a per user database for bayes the
> > database grows significantly and storing this DB in memory is not
> > financially viable.
> >
> > In order to solve our concerns with 4. (and as we are not performance
> > bound for Redis components) we are considering switching our use cases to
> > KVrocks, and potentially makes KVrocks the default key value store / cache
> > for Apache James.
> >
> > I (in the name of the Apache James PMC and LINAGORA contributors) would
> > happily report on how this integrations goes on.
> >
> > --
> >
> > Best regards,
> >
> > Benoit TELLIER
> >
> > General manager of Linagora VIETNAM.
> > Product owner for Team-Mail product.
> > Chairman of the Apache James project.
> >
> > Mail: btellier@linagora.com
> > Tel: (0033) 6 77 26 04 58 (WhatsApp, Signal)
> >
> >
> >
>
> --
> Best Regards,
> - *Hulk Lin*
Re: Apache James meets KVRocks
Posted by hulk <hu...@gmail.com>.
Hi Benoit,
As a Kvrocks PMC member, I'm very happy to see this happens. Also, great
thanks to Benoit for bringing
the potential opportunities where the two communities could work together.
On Mon, 3 Mar 2025 at 16:12, Benoit TELLIER <bt...@linagora.com> wrote:
> Hello folks,
>
> I'm involved on the Apache James email server, and as part of Apache James
> we currently have a Redis integration.
>
> Redis is used for:
>
> - 1. Implementing rate limiting in a distributed fashion by using leaky
> bucket algorithm
> - 2. Implementing sid storage as part of our OpenID connect backchannel
> logout (integrated in Apache APISIX - java plugin runner)
> - 3. Use redis streams for backing apache James pub sub (~email
> notifications)
> - 4. Store bayes database for our RSPAMD setup
>
> Our setup needs to be using sentinel (some use case like 4. relies on
> commands not supplied from Redis cluster)
>
> Our biggest pain is that, as we use a per user database for bayes the
> database grows significantly and storing this DB in memory is not
> financially viable.
>
> In order to solve our concerns with 4. (and as we are not performance
> bound for Redis components) we are considering switching our use cases to
> KVrocks, and potentially makes KVrocks the default key value store / cache
> for Apache James.
>
> I (in the name of the Apache James PMC and LINAGORA contributors) would
> happily report on how this integrations goes on.
>
> --
>
> Best regards,
>
> Benoit TELLIER
>
> General manager of Linagora VIETNAM.
> Product owner for Team-Mail product.
> Chairman of the Apache James project.
>
> Mail: btellier@linagora.com
> Tel: (0033) 6 77 26 04 58 (WhatsApp, Signal)
>
>
>
--
Best Regards,
- *Hulk Lin*
Re: Apache James meets KVRocks
Posted by Twice <tw...@gmail.com>.
Hi Quan,
I think a quick workaround is to use EVALSHA_RO in replica mode, which is a
read-only command. We will investigate it soon : )
Best,
Twice
On Mon, Mar 31, 2025, 1:22 PM hulk <hu...@gmail.com> wrote:
> Hi Quan,
>
> > While Redis replicas allow executing read-only Lua scripts, Kvrocks
> replicas appear to be stricter, rejecting the command with the ReadOnly
> error.
>
> Apache Kvrocks did not allow to run EVALSHA in replica node. I will create
> an issue to track this issue and hope it can be supported ASAP.
>
> On Mon, 31 Mar 2025 at 11:58, Hong Quan TRAN <hq...@linagora.com> wrote:
>
> > Hi Apache Kvrocks community,
> >
> >
> > We've been working on integrating Apache Kvrocks with Apache James and
> > Rspamd, documenting our Docker Compose setup in this PR. While setting up
> > Kvrocks in Sentinel mode, we noticed a lack of documentation and would be
> > happy to contribute to improving it.
> >
> >
> > During our Kvrocks Sentinel + Rspamd setup, we observed a difference in
> > how the EVALSHA command is handled compared to Redis. Rspamd uses EVALSHA
> > to execute a read-only Lua script for retrieving BAYES statistics. While
> > Redis replicas allow executing read-only Lua scripts, Kvrocks replicas
> > appear to be stricter, rejecting the command with the ReadOnly error. As
> a
> > workaround, we have configured slave-read-only no to enable Rspamd to
> read
> > BAYES statistics from Kvrocks replicas while relying on Sentinel to
> > correctly detect master and replica endpoints.
> >
> >
> > We would appreciate any feedback on our setup and approach.
> >
> >
> > Best regards,
> >
> >
> > Quan
> >
> >
> > On Mar 3, 2025 3:12 PM, from Benoit Tellier <bt...@linagora.com>Hello
> > folks,
> >
> > I'm involved on the Apache James email server, and as part of Apache
> James
> > we currently have a Redis integration.
> >
> > Redis is used for:
> >
> > - 1. Implementing rate limiting in a distributed fashion by using leaky
> > bucket algorithm
> > - 2. Implementing sid storage as part of our OpenID connect backchannel
> > logout (integrated in Apache APISIX - java plugin runner)
> > - 3. Use redis streams for backing apache James pub sub (~email
> > notifications)
> > - 4. Store bayes database for our RSPAMD setup
> >
> > Our setup needs to be using sentinel (some use case like 4. relies on
> > commands not supplied from Redis cluster)
> >
> > Our biggest pain is that, as we use a per user database for bayes the
> > database grows significantly and storing this DB in memory is not
> > financially viable.
> >
> > In order to solve our concerns with 4. (and as we are not performance
> > bound for Redis components) we are considering switching our use cases to
> > KVrocks, and potentially makes KVrocks the default key value store /
> cache
> > for Apache James.
> >
> > I (in the name of the Apache James PMC and LINAGORA contributors) would
> > happily report on how this integrations goes on.
> >
> > --
> >
> > Best regards,
> >
> > Benoit TELLIER
> >
> > General manager of Linagora VIETNAM .
> > Product owner for Team-Mail product.
> > Chairman of the Apache James project .
> >
> > Mail: btellier@linagora.com
> > Tel: (0033) 6 77 26 04 58 (WhatsApp, Signal)
> >
> >
> >
>
> --
> Best Regards,
> - *Hulk Lin*
>
Re: Apache James meets KVRocks
Posted by hulk <hu...@gmail.com>.
Hi Quan,
> While Redis replicas allow executing read-only Lua scripts, Kvrocks
replicas appear to be stricter, rejecting the command with the ReadOnly
error.
Apache Kvrocks did not allow to run EVALSHA in replica node. I will create
an issue to track this issue and hope it can be supported ASAP.
On Mon, 31 Mar 2025 at 11:58, Hong Quan TRAN <hq...@linagora.com> wrote:
> Hi Apache Kvrocks community,
>
>
> We've been working on integrating Apache Kvrocks with Apache James and
> Rspamd, documenting our Docker Compose setup in this PR. While setting up
> Kvrocks in Sentinel mode, we noticed a lack of documentation and would be
> happy to contribute to improving it.
>
>
> During our Kvrocks Sentinel + Rspamd setup, we observed a difference in
> how the EVALSHA command is handled compared to Redis. Rspamd uses EVALSHA
> to execute a read-only Lua script for retrieving BAYES statistics. While
> Redis replicas allow executing read-only Lua scripts, Kvrocks replicas
> appear to be stricter, rejecting the command with the ReadOnly error. As a
> workaround, we have configured slave-read-only no to enable Rspamd to read
> BAYES statistics from Kvrocks replicas while relying on Sentinel to
> correctly detect master and replica endpoints.
>
>
> We would appreciate any feedback on our setup and approach.
>
>
> Best regards,
>
>
> Quan
>
>
> On Mar 3, 2025 3:12 PM, from Benoit Tellier <bt...@linagora.com>Hello
> folks,
>
> I'm involved on the Apache James email server, and as part of Apache James
> we currently have a Redis integration.
>
> Redis is used for:
>
> - 1. Implementing rate limiting in a distributed fashion by using leaky
> bucket algorithm
> - 2. Implementing sid storage as part of our OpenID connect backchannel
> logout (integrated in Apache APISIX - java plugin runner)
> - 3. Use redis streams for backing apache James pub sub (~email
> notifications)
> - 4. Store bayes database for our RSPAMD setup
>
> Our setup needs to be using sentinel (some use case like 4. relies on
> commands not supplied from Redis cluster)
>
> Our biggest pain is that, as we use a per user database for bayes the
> database grows significantly and storing this DB in memory is not
> financially viable.
>
> In order to solve our concerns with 4. (and as we are not performance
> bound for Redis components) we are considering switching our use cases to
> KVrocks, and potentially makes KVrocks the default key value store / cache
> for Apache James.
>
> I (in the name of the Apache James PMC and LINAGORA contributors) would
> happily report on how this integrations goes on.
>
> --
>
> Best regards,
>
> Benoit TELLIER
>
> General manager of Linagora VIETNAM .
> Product owner for Team-Mail product.
> Chairman of the Apache James project .
>
> Mail: btellier@linagora.com
> Tel: (0033) 6 77 26 04 58 (WhatsApp, Signal)
>
>
>
--
Best Regards,
- *Hulk Lin*
Re: Apache James meets KVRocks
Posted by Hong Quan TRAN <hq...@linagora.com>.
Hi Apache Kvrocks community,
We've been working on integrating Apache Kvrocks with Apache James and Rspamd, documenting our Docker Compose setup in this PR. While setting up Kvrocks in Sentinel mode, we noticed a lack of documentation and would be happy to contribute to improving it.
During our Kvrocks Sentinel + Rspamd setup, we observed a difference in how the EVALSHA command is handled compared to Redis. Rspamd uses EVALSHA to execute a read-only Lua script for retrieving BAYES statistics. While Redis replicas allow executing read-only Lua scripts, Kvrocks replicas appear to be stricter, rejecting the command with the ReadOnly error. As a workaround, we have configured slave-read-only no to enable Rspamd to read BAYES statistics from Kvrocks replicas while relying on Sentinel to correctly detect master and replica endpoints.
We would appreciate any feedback on our setup and approach.
Best regards,
Quan
On Mar 3, 2025 3:12 PM, from Benoit Tellier <bt...@linagora.com>Hello folks,
I'm involved on the Apache James email server, and as part of Apache James we currently have a Redis integration.
Redis is used for:
- 1. Implementing rate limiting in a distributed fashion by using leaky bucket algorithm
- 2. Implementing sid storage as part of our OpenID connect backchannel logout (integrated in Apache APISIX - java plugin runner)
- 3. Use redis streams for backing apache James pub sub (~email notifications)
- 4. Store bayes database for our RSPAMD setup
Our setup needs to be using sentinel (some use case like 4. relies on commands not supplied from Redis cluster)
Our biggest pain is that, as we use a per user database for bayes the database grows significantly and storing this DB in memory is not financially viable.
In order to solve our concerns with 4. (and as we are not performance bound for Redis components) we are considering switching our use cases to KVrocks, and potentially makes KVrocks the default key value store / cache for Apache James.
I (in the name of the Apache James PMC and LINAGORA contributors) would happily report on how this integrations goes on.
--
Best regards,
Benoit TELLIER
General manager of Linagora VIETNAM .
Product owner for Team-Mail product.
Chairman of the Apache James project .
Mail: btellier@linagora.com
Tel: (0033) 6 77 26 04 58 (WhatsApp, Signal)