You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@kvrocks.apache.org by "ltagliamonte-dd (via GitHub)" <gi...@apache.org> on 2024/06/28 15:40:30 UTC
[D] Tune Compaction and Backup [kvrocks]
GitHub user ltagliamonte-dd created a discussion: Tune Compaction and Backup
Hello folks! I'm in a pinch with kvrocks compaction/backup tuning....
I'm working on a data pipeline where I process a bunch of parquet files into kvrocks, every N files I create a checkpoint on S3 by calling compaction (COMPACT) on kvrocks and then backup (BGSAVE), and uploading the backup folder on S3.
Very often during the compaction and backup the memory consumption goes really high and my k8s pod with 40GB of mem get's oomkilled.
Is there a way to keep at bay the mem usage during compaction and backup?
What are the knobs to tune? I don't see any knob for backups.
This is my first time using kvrocks and I'm not really familiar with all the config params.
GitHub link: https://github.com/apache/kvrocks/discussions/2379
----
This is an automatically sent email for issues@kvrocks.apache.org.
To unsubscribe, please send an email to: issues-unsubscribe@kvrocks.apache.org
Re: [D] Tune Compaction and Backup [kvrocks]
Posted by "ltagliamonte-dd (via GitHub)" <gi...@apache.org>.
GitHub user ltagliamonte-dd edited a comment on the discussion: Tune Compaction and Backup
@git-hulk thank for the suggestion, my params are just 3X the defaults, i bumped those up because compaction on the dataset was taking >10minutes
is there any other params I can use to improve the compaction?
the backup also seems to consume a lot of memory, i wonder why kvrocks doesn't just symlink the data folder in the backup folder, like cassandra does for example.
GitHub link: https://github.com/apache/kvrocks/discussions/2379#discussioncomment-9907014
----
This is an automatically sent email for issues@kvrocks.apache.org.
To unsubscribe, please send an email to: issues-unsubscribe@kvrocks.apache.org
Re: [D] Tune Compaction and Backup [kvrocks]
Posted by "git-hulk (via GitHub)" <gi...@apache.org>.
GitHub user git-hulk added a comment to the discussion: Tune Compaction and Backup
> My point about smarter compaction of the SST is that they all get reprocessed all the time isn't it?
Oh, yes. That's right.
GitHub link: https://github.com/apache/kvrocks/discussions/2379#discussioncomment-9910114
----
This is an automatically sent email for issues@kvrocks.apache.org.
To unsubscribe, please send an email to: issues-unsubscribe@kvrocks.apache.org
Re: [D] Tune Compaction and Backup [kvrocks]
Posted by "git-hulk (via GitHub)" <gi...@apache.org>.
GitHub user git-hulk edited a comment on the discussion: Tune Compaction and Backup
> is there any other params I can use to improve the compaction?
Yes, this is a tough issue for rocksdb. It might consume too much memory and CPU while increasing the compaction threads, but decreasing it will affect the speed. But I'm not sure why do you have to do a compaction before creating the backup? not sure if you have many deleted/expired data inside the DB. I think it's a good idea to slow down the compaction speed to avoid consuming too many resources and wait until it's finished.
> the backup also seems to consume a lot of memory, i wonder why kvrocks doesn't just symlink the data folder in the backup folder, like cassandra does for example.
It was a hard symlink as well, so it should not consume much memory. Not sure if this's affected by compaction behavior.
GitHub link: https://github.com/apache/kvrocks/discussions/2379#discussioncomment-9909873
----
This is an automatically sent email for issues@kvrocks.apache.org.
To unsubscribe, please send an email to: issues-unsubscribe@kvrocks.apache.org
Re: [D] Tune Compaction and Backup [kvrocks]
Posted by "git-hulk (via GitHub)" <gi...@apache.org>.
GitHub user git-hulk edited a comment on the discussion: Tune Compaction and Backup
@ltagliamonte-dd Thank you! Maybe you can try to decrease `max_background_jobs` and `max_sub_compactions` to avoid too many ongoing compaction threads. Guess you can just remove those two configurations in your command line.
GitHub link: https://github.com/apache/kvrocks/discussions/2379#discussioncomment-9906930
----
This is an automatically sent email for issues@kvrocks.apache.org.
To unsubscribe, please send an email to: issues-unsubscribe@kvrocks.apache.org
Re: [D] Tune Compaction and Backup [kvrocks]
Posted by "git-hulk (via GitHub)" <gi...@apache.org>.
GitHub user git-hulk edited a comment on the discussion: Tune Compaction and Backup
@ltagliamonte-dd The default value of memory cache is 4GiB and it only be used while having the read requests. So guess it would not help if the memory is consumed by the compaction. Another possible approach is to try to reduce `target_file_size_base` and increase a few compaction threads
GitHub link: https://github.com/apache/kvrocks/discussions/2379#discussioncomment-9927170
----
This is an automatically sent email for issues@kvrocks.apache.org.
To unsubscribe, please send an email to: issues-unsubscribe@kvrocks.apache.org
Re: [D] Tune Compaction and Backup [kvrocks]
Posted by "git-hulk (via GitHub)" <gi...@apache.org>.
GitHub user git-hulk added a comment to the discussion: Tune Compaction and Backup
@ltagliamonte-dd The default value of memory cache is 4GiB and it only be used while having the read requests. So it should be helpless if the memory is consumed by the compaction. Another possible approach is to try to reduce `target_file_size_base` and increase a few compaction threads
GitHub link: https://github.com/apache/kvrocks/discussions/2379#discussioncomment-9927170
----
This is an automatically sent email for issues@kvrocks.apache.org.
To unsubscribe, please send an email to: issues-unsubscribe@kvrocks.apache.org
Re: [D] Tune Compaction and Backup [kvrocks]
Posted by "git-hulk (via GitHub)" <gi...@apache.org>.
GitHub user git-hulk added a comment to the discussion: Tune Compaction and Backup
> is there a compaction strategy that would be "smarter" and touch just the files that got modified?
Kvrocks is now supported by the compaction checker and it will check if SST files need to be compacted periodically. Another point is that SST files are never updated and data are merged by compaction.
```
# The hour range that compaction checker would be active
# e.g. compaction-checker-range 0-7 means compaction checker would be worker between
# 0-7am every day.
compaction-checker-range 0-7
```
GitHub link: https://github.com/apache/kvrocks/discussions/2379#discussioncomment-9909981
----
This is an automatically sent email for issues@kvrocks.apache.org.
To unsubscribe, please send an email to: issues-unsubscribe@kvrocks.apache.org
Re: [D] Tune Compaction and Backup [kvrocks]
Posted by "git-hulk (via GitHub)" <gi...@apache.org>.
GitHub user git-hulk added a comment to the discussion: Tune Compaction and Backup
@ltagliamonte-dd Thank you! Maybe you can try to decrease `max_background_jobs` and `max_sub_compactions` to avoid too many ongoing compaction threads.
GitHub link: https://github.com/apache/kvrocks/discussions/2379#discussioncomment-9906930
----
This is an automatically sent email for issues@kvrocks.apache.org.
To unsubscribe, please send an email to: issues-unsubscribe@kvrocks.apache.org
Re: [D] Tune Compaction and Backup [kvrocks]
Posted by "ltagliamonte-dd (via GitHub)" <gi...@apache.org>.
GitHub user ltagliamonte-dd added a comment to the discussion: Tune Compaction and Backup
Hello @git-hulk well yes and no, mem usage is contained however compaction take up to 60m to complete.
I'd like to find other params i can tune to reduce the memory footprint, like reduce the number in memory cache (i'm just writing in this kvrocks instance) and accelerate the compactions.
GitHub link: https://github.com/apache/kvrocks/discussions/2379#discussioncomment-9927075
----
This is an automatically sent email for issues@kvrocks.apache.org.
To unsubscribe, please send an email to: issues-unsubscribe@kvrocks.apache.org
Re: [D] Tune Compaction and Backup [kvrocks]
Posted by "ltagliamonte (via GitHub)" <gi...@apache.org>.
GitHub user ltagliamonte added a comment to the discussion: Tune Compaction and Backup
Thanks for the answers! Since I'm calling compaction at every run of my software I disabled them passing the flag to the cmd line.
My point about smarter compaction of the SST is that they all get reprocessed all the time isn't it?
GitHub link: https://github.com/apache/kvrocks/discussions/2379#discussioncomment-9910076
----
This is an automatically sent email for issues@kvrocks.apache.org.
To unsubscribe, please send an email to: issues-unsubscribe@kvrocks.apache.org
Re: [D] Tune Compaction and Backup [kvrocks]
Posted by "git-hulk (via GitHub)" <gi...@apache.org>.
GitHub user git-hulk added a comment to the discussion: Tune Compaction and Backup
@ltagliamonte-dd Wanna know if everything is going well after reducing the number of compaction threads.
GitHub link: https://github.com/apache/kvrocks/discussions/2379#discussioncomment-9922480
----
This is an automatically sent email for issues@kvrocks.apache.org.
To unsubscribe, please send an email to: issues-unsubscribe@kvrocks.apache.org
Re: [D] Tune Compaction and Backup [kvrocks]
Posted by "ltagliamonte-dd (via GitHub)" <gi...@apache.org>.
GitHub user ltagliamonte-dd added a comment to the discussion: Tune Compaction and Backup
Hello @git-hulk I'm using the cmd line options, this is my current config:
```
command=/usr/local/bin/kvrocks/kvrocks
--dir %(ENV_KVROCKS_DATA_DIR)s
--port 6379
--rocksdb.disable_auto_compactions yes
--rocksdb.rate_limiter_auto_tuned no
--rocksdb.max_background_jobs 12
--rocksdb.max_sub_compactions 6
--rocksdb.target_file_size_base 1024
```
GitHub link: https://github.com/apache/kvrocks/discussions/2379#discussioncomment-9906922
----
This is an automatically sent email for issues@kvrocks.apache.org.
To unsubscribe, please send an email to: issues-unsubscribe@kvrocks.apache.org
Re: [D] Tune Compaction and Backup [kvrocks]
Posted by "git-hulk (via GitHub)" <gi...@apache.org>.
GitHub user git-hulk added a comment to the discussion: Tune Compaction and Backup
> is there any other params I can use to improve the compaction?
Yes, this is a tough question for rocksdb. It might consume too much memory and CPU while increasing the compaction threads, but decreasing it will affect the speed. But I'm not sure why do you have to do a compaction before creating the backup? not sure if you have many deleted/expired data inside the DB. I think it's a good idea to slow down the compaction speed to avoid consuming too many resources and wait until it's finished.
> the backup also seems to consume a lot of memory, i wonder why kvrocks doesn't just symlink the data folder in the backup folder, like cassandra does for example.
It was a hard symlink as well, so it should not consume much memory. Not sure if this's affected by compaction behavior.
GitHub link: https://github.com/apache/kvrocks/discussions/2379#discussioncomment-9909873
----
This is an automatically sent email for issues@kvrocks.apache.org.
To unsubscribe, please send an email to: issues-unsubscribe@kvrocks.apache.org
Re: [D] Tune Compaction and Backup [kvrocks]
Posted by "ltagliamonte-dd (via GitHub)" <gi...@apache.org>.
GitHub user ltagliamonte-dd added a comment to the discussion: Tune Compaction and Backup
i do just inserts/updates, no deletes.
the problem with the compaction obviously is that they go over all the data, maybe I should just compact every N runs instead of doing it at every run.
is there a compaction strategy that would be "smarter" and touch just the files that got modified?
GitHub link: https://github.com/apache/kvrocks/discussions/2379#discussioncomment-9909918
----
This is an automatically sent email for issues@kvrocks.apache.org.
To unsubscribe, please send an email to: issues-unsubscribe@kvrocks.apache.org
Re: [D] Tune Compaction and Backup [kvrocks]
Posted by "ltagliamonte-dd (via GitHub)" <gi...@apache.org>.
GitHub user ltagliamonte-dd added a comment to the discussion: Tune Compaction and Backup
@git-hulk thank for the suggestion, my params are just 3X the defaults, i bumped those up because compaction on the dataset was taking >10M.
is there any other params I can use to improve the compaction?
the backup also seems to consume a lot of memory, i wonder why kvrocks doesn't just symlink the data folder in the backup folder, like cassandra does for example.
GitHub link: https://github.com/apache/kvrocks/discussions/2379#discussioncomment-9907014
----
This is an automatically sent email for issues@kvrocks.apache.org.
To unsubscribe, please send an email to: issues-unsubscribe@kvrocks.apache.org
Re: [D] Tune Compaction and Backup [kvrocks]
Posted by "git-hulk (via GitHub)" <gi...@apache.org>.
GitHub user git-hulk added a comment to the discussion: Tune Compaction and Backup
@ltagliamonte-dd Could you provide rocksdb.* configurations in kvrocks.conf?
GitHub link: https://github.com/apache/kvrocks/discussions/2379#discussioncomment-9906909
----
This is an automatically sent email for issues@kvrocks.apache.org.
To unsubscribe, please send an email to: issues-unsubscribe@kvrocks.apache.org