You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@kvrocks.apache.org by "VIVALXH (via GitHub)" <gi...@apache.org> on 2024/08/05 07:58:14 UTC
[D] kvrocks压测-set命令服务器资源无法吃满 [kvrocks]
GitHub user VIVALXH created a discussion: kvrocks压测-set命令服务器资源无法吃满
压测目的：我们希望在实际生产一定压力下，确定最优的资源配比，以确保kvrocks能够尽可能利用服务器资源
压测问题：
1. 单看set命令，cpu/内存/磁盘/网络/连接数这些硬件资源基本上都没有打到瓶颈，但是我们无论怎么加压，qps都无法上去。
2. 固态/机械的磁盘都用过，也用过其它压测客户端，资源依然吃不满，我们无法确定是哪个地方堵塞了，请问是否有配置限制或者其它限制？
压测结果：
1. kvrocks_exporter ：https://snapshots.raintank.io/dashboard/snapshot/XAdP9Oh2tjypUg66H5E5yXGKvrlOOgOC
3. node_exporter : https://snapshots.raintank.io/dashboard/snapshot/nmCITcPNMU1h2B57cZpDWJaMQvzMC50p
测试环境：
被打压机器：
- 规格：linux 4c8g
- 运行服务命令：docker run -it -p 6666:6666 apache/kvrocks --bind 0.0.0.0
- kvrocks 镜像版本：sha256:707f93473e7265df39593d38d83081ec1a6a36da31c97b391008858eb45a437f
- 运行监控命令：./kvrocks_exporter --kvrocks.addr kvrocks://192.168.1.72:6666
- kvrocks_exporter 版本：(https://github.com/RocksLabs/kvrocks_exporter/commit/013564909cb969fcd3c44a080d7f320cc6ad5796)
打压机器：
- 规格：linux 12c24g redis-benchmark
- 运行命令：redis-benchmark -t set -r 100000 -n 1000000 -h 192.168.1.72 -p 6666
GitHub link: https://github.com/apache/kvrocks/discussions/2467
----
This is an automatically sent email for issues@kvrocks.apache.org.
To unsubscribe, please send an email to: issues-unsubscribe@kvrocks.apache.org
Re: [D] kvrocks压测-set命令服务器资源无法吃满 [kvrocks]
Posted by "git-hulk (via GitHub)" <gi...@apache.org>.
GitHub user git-hulk edited a comment on the discussion: kvrocks压测-set命令服务器资源无法吃满
@HughPenn Good, could you please provide your benchmark command and the iostat's result?
GitHub link: https://github.com/apache/kvrocks/discussions/2467#discussioncomment-12966588
----
This is an automatically sent email for issues@kvrocks.apache.org.
To unsubscribe, please send an email to: issues-unsubscribe@kvrocks.apache.org
Re: [D] kvrocks压测-set命令服务器资源无法吃满 [kvrocks]
Posted by "VIVALXH (via GitHub)" <gi...@apache.org>.
GitHub user VIVALXH edited a comment on the discussion: kvrocks压测-set命令服务器资源无法吃满
@git-hulk 感谢回答，我们更换了阿里云的机器进行测试：
## 规格
服务器(4c8g ecs.c7.xlarge 挂载ssd云盘)
## 压测命令
redis-benchmark -t set -r 100000 -n 10000000 -h 1.0.105.255 -p 6666 --threads 100
redis-benchmark -t get -r 100000 -n 10000000 -h 1.0.105.255 -p 6666 --threads 100
## 压测结果
### set 命令 iostat
- cpu空闲率依旧有10-20左右
- 虽然%wrqm和w_await较高，但并不持续，可能有可配置的刷盘策略？
- ioutil也并不是很高，感觉磁盘没有到瓶颈？
### set 命令 top htop
- top命令中，kvrocks占用的核数只有2.5(62%)，但是总占用已经达到了85%，理论上这两个指标应该一致吧？
- 从我们之前的测试结果看，限定核数1c的qps是3w qps/s，理想情况4c应该是12w qps/s，但是实际只有8w qps/s， 我们的结论是kvrocks的性能和kvrocks占用的cpu(62%CPU)成正比，但并不与总占用成正比，这数据是否正常呢？
### get 命令 iostat
- cpu基本吃满，应该都是从内存读的，磁盘基本没有压力，能压到cpu上限这个是我们认为正常的。
GitHub link: https://github.com/apache/kvrocks/discussions/2467#discussioncomment-10241914
----
This is an automatically sent email for issues@kvrocks.apache.org.
To unsubscribe, please send an email to: issues-unsubscribe@kvrocks.apache.org
Re: [D] kvrocks压测-set命令服务器资源无法吃满 [kvrocks]
Posted by "VIVALXH (via GitHub)" <gi...@apache.org>.
GitHub user VIVALXH edited a comment on the discussion: kvrocks压测-set命令服务器资源无法吃满
@git-hulk 感谢回答，我们更换了阿里云的机器进行测试：
## 规格
服务器(4c8g ecs.c7.xlarge 挂载ssd云盘)
## 压测命令
redis-benchmark -t set -r 100000 -n 10000000 -h 1.0.105.255 -p 6666 --threads 100
redis-benchmark -t get -r 100000 -n 10000000 -h 1.0.105.255 -p 6666 --threads 100
## 压测结果
### set 命令 iostat
- cpu空闲率依旧有10-20左右
- 虽然%wrqm和w_await较高，但并不持续，可能有可配置的刷盘策略？
- ioutil也并不是很高，感觉磁盘没有到瓶颈？
### set 命令 top htop
- top命令中，kvrocks占用的核数只有2.5(62%)，但是总占用已经达到了85%，理论上这两个指标应该一致吧？
- 从我们之前的测试结果看，限定核数1c的qps是3w qps/s，理想情况4c应该是12w qps/s，但是实际只有8w qps/s，我们的结论是kvrocks的性能和kvrocks占用的cpu(62%CPU)成正比，但并不与总占用成正比，我们提升服务器规格(8c) 后得出的结论也是这样的，这性能转换率是否正常呢？
### get 命令 iostat
- cpu基本吃满，应该都是从内存读的，磁盘基本没有压力，能压到cpu上限这个是我们认为正常的。
GitHub link: https://github.com/apache/kvrocks/discussions/2467#discussioncomment-10241914
----
This is an automatically sent email for issues@kvrocks.apache.org.
To unsubscribe, please send an email to: issues-unsubscribe@kvrocks.apache.org
Re: [D] kvrocks压测-set命令服务器资源无法吃满 [kvrocks]
Posted by "VIVALXH (via GitHub)" <gi...@apache.org>.
GitHub user VIVALXH edited a discussion: kvrocks压测-set命令服务器资源无法吃满
## 压测目的：
我们希望在实际生产一定压力下，确定最优的资源配比，以确保kvrocks能够尽可能利用服务器资源。
## 压测问题：
- 单看set命令，cpu/内存/磁盘/网络/连接数这些硬件资源基本上都没有打到瓶颈，但是我们无论怎么加压，qps都无法上去。
- 固态/机械的磁盘都用过，也用过其它压测客户端，资源依然吃不满，我们无法确定是哪个地方堵塞了，请问是否有配置限制或者其它限制？
## 压测结果：
- kvrocks_exporter ：https://snapshots.raintank.io/dashboard/snapshot/XAdP9Oh2tjypUg66H5E5yXGKvrlOOgOC
- node_exporter : https://snapshots.raintank.io/dashboard/snapshot/nmCITcPNMU1h2B57cZpDWJaMQvzMC50p
## 测试环境：
被打压机器：
- 规格：linux 4c8g
- 运行服务命令：docker run -it -p 6666:6666 apache/kvrocks --bind 0.0.0.0
- kvrocks 镜像版本：sha256:707f93473e7265df39593d38d83081ec1a6a36da31c97b391008858eb45a437f
- 运行监控命令：./kvrocks_exporter --kvrocks.addr kvrocks://192.168.1.72:6666
- kvrocks_exporter 版本：(https://github.com/RocksLabs/kvrocks_exporter/commit/013564909cb969fcd3c44a080d7f320cc6ad5796)
打压机器：
- 规格：linux 12c24g redis-benchmark
- 运行命令：redis-benchmark -t set -r 100000 -n 1000000 -h 192.168.1.72 -p 6666
GitHub link: https://github.com/apache/kvrocks/discussions/2467
----
This is an automatically sent email for issues@kvrocks.apache.org.
To unsubscribe, please send an email to: issues-unsubscribe@kvrocks.apache.org
Re: [D] kvrocks压测-set命令服务器资源无法吃满 [kvrocks]
Posted by "PragmaTwice (via GitHub)" <gi...@apache.org>.
GitHub user PragmaTwice added a comment to the discussion: kvrocks压测-set命令服务器资源无法吃满
- you can try to adjust `workers` to a higher number (e.g. equals to number of cpu cores)
- you can connect kvrocks via a connection pool to utilize the multi-thread processing in kvrocks
GitHub link: https://github.com/apache/kvrocks/discussions/2467#discussioncomment-10241600
----
This is an automatically sent email for issues@kvrocks.apache.org.
To unsubscribe, please send an email to: issues-unsubscribe@kvrocks.apache.org
Re: [D] kvrocks压测-set命令服务器资源无法吃满 [kvrocks]
Posted by "VIVALXH (via GitHub)" <gi...@apache.org>.
GitHub user VIVALXH edited a discussion: kvrocks压测-set命令服务器资源无法吃满
## 压测目的：
我们希望在实际生产一定压力下，确定最优的资源配比，以确保kvrocks能够尽可能利用服务器资源。
## 压测问题：
- 单看set命令，cpu/内存/磁盘/网络/连接数这些硬件相关资源基本上都没有打到瓶颈，但是我们无论怎么加压，qps都无法上去，我们无法确定是哪个地方出现瓶颈了，请问是否有配置限制或者其它相关限制？。
- 固态/机械的磁盘都用过，也用过其它压测客户端，资源依然吃不满。
- 升级过被打压机器的规格(8c16g)，虽然qps可以上去，但资源依旧吃不满。
## 压测结果：
- kvrocks_exporter ：https://snapshots.raintank.io/dashboard/snapshot/XAdP9Oh2tjypUg66H5E5yXGKvrlOOgOC
- node_exporter : https://snapshots.raintank.io/dashboard/snapshot/nmCITcPNMU1h2B57cZpDWJaMQvzMC50p
## 测试环境：
被打压机器：
- 规格：linux 4c8g
- 运行服务命令：docker run -it -p 6666:6666 apache/kvrocks --bind 0.0.0.0
- kvrocks 镜像版本：sha256:707f93473e7265df39593d38d83081ec1a6a36da31c97b391008858eb45a437f
- 运行监控命令：./kvrocks_exporter --kvrocks.addr kvrocks://192.168.1.72:6666
- kvrocks_exporter 版本：(https://github.com/RocksLabs/kvrocks_exporter/commit/013564909cb969fcd3c44a080d7f320cc6ad5796)
打压机器：
- 规格：linux 12c24g redis-benchmark
- 运行命令：redis-benchmark -t set -r 100000 -n 1000000 -h 192.168.1.72 -p 6666
GitHub link: https://github.com/apache/kvrocks/discussions/2467
----
This is an automatically sent email for issues@kvrocks.apache.org.
To unsubscribe, please send an email to: issues-unsubscribe@kvrocks.apache.org
Re: [D] kvrocks压测-set命令服务器资源无法吃满 [kvrocks]
Posted by "VIVALXH (via GitHub)" <gi...@apache.org>.
GitHub user VIVALXH edited a comment on the discussion: kvrocks压测-set命令服务器资源无法吃满
@git-hulk 感谢回答，我们更换了阿里云的机器进行测试：
## 规格
服务器(4c8g ecs.c7.xlarge 挂载ssd云盘)
## 压测命令
redis-benchmark -t set -r 100000 -n 10000000 -h 1.0.105.255 -p 6666 --threads 100
redis-benchmark -t get -r 100000 -n 10000000 -h 1.0.105.255 -p 6666 --threads 100
## 压测结果
### set 命令 iostat
- cpu空闲率依旧有10-20左右
- 虽然%wrqm和w_await较高，但并不持续，可能有可配置的刷盘策略？
- ioutil也并不是很高，感觉磁盘没有到瓶颈？
### sset命令 top
- top命令中，kvrocks占用的核数只有2.5(62%)，但是总占用已经达到了85%，理论上这两个指标应该一致吧？从我们之前的测试结果看，kvrocks的性能和kvrocks的占用的cpu(%CPU)成正比，但并不与总占用成正比，所以很奇怪。
### sget命令 iostat
- cpu基本吃满，应该都是从内存读的，磁盘基本没有压力，能压到cpu上限这个是我们认为正常的。
GitHub link: https://github.com/apache/kvrocks/discussions/2467#discussioncomment-10241914
----
This is an automatically sent email for issues@kvrocks.apache.org.
To unsubscribe, please send an email to: issues-unsubscribe@kvrocks.apache.org
Re: [D] kvrocks压测-set命令服务器资源无法吃满 [kvrocks]
Posted by "VIVALXH (via GitHub)" <gi...@apache.org>.
GitHub user VIVALXH edited a comment on the discussion: kvrocks压测-set命令服务器资源无法吃满
@git-hulk 感谢回答，我们更换了阿里云的机器进行测试：
## 规格
服务器(4c8g ecs.c7.xlarge 挂载ssd云盘)
## 压测命令
redis-benchmark -t set -r 100000 -n 10000000 -h 1.0.105.255 -p 6666 --threads 100
redis-benchmark -t get -r 100000 -n 10000000 -h 1.0.105.255 -p 6666 --threads 100
## 压测结果
### set 命令 iostat
- cpu空闲率依旧有10-20左右
- 虽然%wrqm和w_await较高，但并不持续，可能有可配置的刷盘策略？
- ioutil也并不是很高，感觉磁盘没有到瓶颈？
### set 命令 top
- top命令中，kvrocks占用的核数只有2.5(62%)，但是总占用已经达到了85%，理论上这两个指标应该一致吧？
- 从我们之前的测试结果看，kvrocks的性能和kvrocks的占用的cpu(%CPU)成正比，但并不与总占用成正比，这数据是否正常呢？
### get 命令 iostat
- cpu基本吃满，应该都是从内存读的，磁盘基本没有压力，能压到cpu上限这个是我们认为正常的。
GitHub link: https://github.com/apache/kvrocks/discussions/2467#discussioncomment-10241914
----
This is an automatically sent email for issues@kvrocks.apache.org.
To unsubscribe, please send an email to: issues-unsubscribe@kvrocks.apache.org
Re: [D] kvrocks压测-set命令服务器资源无法吃满 [kvrocks]
Posted by "VIVALXH (via GitHub)" <gi...@apache.org>.
GitHub user VIVALXH edited a discussion: kvrocks压测-set命令服务器资源无法吃满
## 压测目的：
我们希望在实际生产一定压力下，确定最优的资源配比，以确保kvrocks能够尽可能利用服务器资源
## 压测问题：
- 单看set命令，cpu/内存/磁盘/网络/连接数这些硬件资源基本上都没有打到瓶颈，但是我们无论怎么加压，qps都无法上去。
- 固态/机械的磁盘都用过，也用过其它压测客户端，资源依然吃不满，我们无法确定是哪个地方堵塞了，请问是否有配置限制或者其它限制？
## 压测结果：
- kvrocks_exporter ：https://snapshots.raintank.io/dashboard/snapshot/XAdP9Oh2tjypUg66H5E5yXGKvrlOOgOC
- node_exporter : https://snapshots.raintank.io/dashboard/snapshot/nmCITcPNMU1h2B57cZpDWJaMQvzMC50p
## 测试环境：
被打压机器：
- 规格：linux 4c8g
- 运行服务命令：docker run -it -p 6666:6666 apache/kvrocks --bind 0.0.0.0
- kvrocks 镜像版本：sha256:707f93473e7265df39593d38d83081ec1a6a36da31c97b391008858eb45a437f
- 运行监控命令：./kvrocks_exporter --kvrocks.addr kvrocks://192.168.1.72:6666
- kvrocks_exporter 版本：(https://github.com/RocksLabs/kvrocks_exporter/commit/013564909cb969fcd3c44a080d7f320cc6ad5796)
打压机器：
- 规格：linux 12c24g redis-benchmark
- 运行命令：redis-benchmark -t set -r 100000 -n 1000000 -h 192.168.1.72 -p 6666
GitHub link: https://github.com/apache/kvrocks/discussions/2467
----
This is an automatically sent email for issues@kvrocks.apache.org.
To unsubscribe, please send an email to: issues-unsubscribe@kvrocks.apache.org
Re: [D] kvrocks压测-set命令服务器资源无法吃满 [kvrocks]
Posted by "VIVALXH (via GitHub)" <gi...@apache.org>.
GitHub user VIVALXH edited a discussion: kvrocks压测-set命令服务器资源无法吃满
## 压测目的
我们希望在实际生产一定压力下，确定最优的资源配比，以确保kvrocks能够尽可能利用服务器资源。
## 压测问题
- 单看set命令，cpu/内存/磁盘/网络/连接数这些硬件相关资源基本上都没有打到瓶颈，但是我们无论怎么加压，qps都无法上去，我们无法确定是哪个地方出现瓶颈了，请问是否有配置限制或者其它相关限制？。
- 固态/机械的磁盘都用过，也用过其它压测客户端，资源依然吃不满。
- 升级过被打压机器的规格(8c16g)，虽然qps可以上去，但资源依旧吃不满。
## 压测结果
- kvrocks_exporter ：https://snapshots.raintank.io/dashboard/snapshot/XAdP9Oh2tjypUg66H5E5yXGKvrlOOgOC
- node_exporter : https://snapshots.raintank.io/dashboard/snapshot/nmCITcPNMU1h2B57cZpDWJaMQvzMC50p
## 测试环境
被打压机器：
- 规格：linux 4c8g
- 运行服务命令：docker run -it -p 6666:6666 apache/kvrocks --bind 0.0.0.0
- kvrocks 镜像版本：sha256:707f93473e7265df39593d38d83081ec1a6a36da31c97b391008858eb45a437f
- 运行监控命令：./kvrocks_exporter --kvrocks.addr kvrocks://192.168.1.72:6666
- kvrocks_exporter 版本：(https://github.com/RocksLabs/kvrocks_exporter/commit/013564909cb969fcd3c44a080d7f320cc6ad5796)
打压机器：
- 规格：linux 12c24g redis-benchmark
- 运行命令：redis-benchmark -t set -r 100000 -n 1000000 -h 192.168.1.72 -p 6666
GitHub link: https://github.com/apache/kvrocks/discussions/2467
----
This is an automatically sent email for issues@kvrocks.apache.org.
To unsubscribe, please send an email to: issues-unsubscribe@kvrocks.apache.org
Re: [D] kvrocks压测-set命令服务器资源无法吃满 [kvrocks]
Posted by "HughPenn (via GitHub)" <gi...@apache.org>.
GitHub user HughPenn added a comment to the discussion: kvrocks压测-set命令服务器资源无法吃满
<img width="1222" alt="企业微信截图_17458938687564" src="https://github.com/user-attachments/assets/81540d90-e757-4866-84f5-806ecedc5995" />
<img width="1639" alt="企业微信截图_17458938362009" src="https://github.com/user-attachments/assets/478859b2-f16e-4e8f-b7cb-edc3e2d0adca" />
<img width="1639" alt="企业微信截图_17458938284932" src="https://github.com/user-attachments/assets/4d58cd30-8a1b-49a5-8073-bb43c99004c8" />
CMD: v=$(printf -- '-%.0s' {1..32});./redis-benchmark -h 10.190.28.65 -p 6666 -c 500 -n 10000000000 -r 10000000000 HSET hash{key}:__rand_int__ hashfield:__rand_int__ $v
GitHub link: https://github.com/apache/kvrocks/discussions/2467#discussioncomment-12975920
----
This is an automatically sent email for issues@kvrocks.apache.org.
To unsubscribe, please send an email to: issues-unsubscribe@kvrocks.apache.org
Re: [D] kvrocks压测-set命令服务器资源无法吃满 [kvrocks]
Posted by "HughPenn (via GitHub)" <gi...@apache.org>.
GitHub user HughPenn added a comment to the discussion: kvrocks压测-set命令服务器资源无法吃满
ran a benchmark HSET, and although the server still has plenty of hardware resources available, the si (software interrupts) remain low
<img width="768" alt="企业微信截图_1745814764326" src="https://github.com/user-attachments/assets/069d87b0-807f-4596-b70d-0db9937f7f20" />
GitHub link: https://github.com/apache/kvrocks/discussions/2467#discussioncomment-12965166
----
This is an automatically sent email for issues@kvrocks.apache.org.
To unsubscribe, please send an email to: issues-unsubscribe@kvrocks.apache.org
Re: [D] kvrocks压测-set命令服务器资源无法吃满 [kvrocks]
Posted by "VIVALXH (via GitHub)" <gi...@apache.org>.
GitHub user VIVALXH edited a comment on the discussion: kvrocks压测-set命令服务器资源无法吃满
@git-hulk 感谢回答，我们更换了阿里云的机器进行测试：
## 规格
服务器(4c8g ecs.c7.xlarge 挂载ssd云盘)
## 压测命令
redis-benchmark -t set -r 100000 -n 10000000 -h 1.0.105.255 -p 6666 --threads 100
redis-benchmark -t get -r 100000 -n 10000000 -h 1.0.105.255 -p 6666 --threads 100
## 压测结果
### set 命令 iostat
- cpu空闲率依旧有10-20左右
- 虽然%wrqm和w_await较高，但并不持续，可能有可配置的刷盘策略？
- ioutil也并不是很高，感觉磁盘没有到瓶颈？
### set 命令 top
- top命令中，kvrocks占用的核数只有2.5(62%)，但是总占用已经达到了85%，理论上这两个指标应该一致吧？
- 从我们之前的测试结果看，kvrocks的性能和kvrocks占用的cpu(%CPU)成正比，但并不与总占用成正比，这数据是否正常呢？
### get 命令 iostat
- cpu基本吃满，应该都是从内存读的，磁盘基本没有压力，能压到cpu上限这个是我们认为正常的。
GitHub link: https://github.com/apache/kvrocks/discussions/2467#discussioncomment-10241914
----
This is an automatically sent email for issues@kvrocks.apache.org.
To unsubscribe, please send an email to: issues-unsubscribe@kvrocks.apache.org
Re: [D] kvrocks压测-set命令服务器资源无法吃满 [kvrocks]
Posted by "VIVALXH (via GitHub)" <gi...@apache.org>.
GitHub user VIVALXH added a comment to the discussion: kvrocks压测-set命令服务器资源无法吃满
@git-hulk
感谢回答，我们更换了阿里云的机器进行测试：
## 规格
服务器(4c8g ecs.c7.xlarge 挂载ssd云盘)
## 压测命令
redis-benchmark -t set -r 100000 -n 10000000 -h 1.0.105.255 -p 6666 --threads 100
redis-benchmark -t get -r 100000 -n 10000000 -h 1.0.105.255 -p 6666 --threads 100
## 压测结果
### set 命令 iostat
- cpu空闲率依旧有10-20左右
- 虽然%wrqm和w_await较高，但并不持续，可能有可配置的刷盘策略？
- ioutil也并不是很高，感觉磁盘没有到瓶颈？
### sset命令 top
- top命令中，kvrocks占用的核数只有2.5(62%)，但是总占用已经达到了85%，理论上这两个指标应该一致吧？
### sget命令 iostat
- cpu基本吃满，应该都是从内存读的，磁盘基本没有压力，能压到cpu上限这个是我们认为正常的。
GitHub link: https://github.com/apache/kvrocks/discussions/2467#discussioncomment-10241914
----
This is an automatically sent email for issues@kvrocks.apache.org.
To unsubscribe, please send an email to: issues-unsubscribe@kvrocks.apache.org
Re: [D] kvrocks压测-set命令服务器资源无法吃满 [kvrocks]
Posted by "git-hulk (via GitHub)" <gi...@apache.org>.
GitHub user git-hulk added a comment to the discussion: kvrocks压测-set命令服务器资源无法吃满
From this picture, 18% of the CPU is occupied by `si`(soft interrupt by NIC), so I believe it's normal for this case.
GitHub link: https://github.com/apache/kvrocks/discussions/2467#discussioncomment-12960787
----
This is an automatically sent email for issues@kvrocks.apache.org.
To unsubscribe, please send an email to: issues-unsubscribe@kvrocks.apache.org
Re: [D] kvrocks压测-set命令服务器资源无法吃满 [kvrocks]
Posted by "zhixinwen (via GitHub)" <gi...@apache.org>.
GitHub user zhixinwen added a comment to the discussion: kvrocks压测-set命令服务器资源无法吃满
The bottleneck is more likely in RocksDB not KVRocks.
When worker number is high, the concurrent write could be blocked by RocksDB's WriteThread lock.
GitHub link: https://github.com/apache/kvrocks/discussions/2467#discussioncomment-13952634
----
This is an automatically sent email for issues@kvrocks.apache.org.
To unsubscribe, please send an email to: issues-unsubscribe@kvrocks.apache.org
Re: [D] kvrocks压测-set命令服务器资源无法吃满 [kvrocks]
Posted by "VIVALXH (via GitHub)" <gi...@apache.org>.
GitHub user VIVALXH edited a comment on the discussion: kvrocks压测-set命令服务器资源无法吃满
@git-hulk 感谢回答，我们更换了阿里云的机器进行测试：
## 规格
服务器(4c8g ecs.c7.xlarge 挂载ssd云盘)
## 压测命令
redis-benchmark -t set -r 100000 -n 10000000 -h 1.0.105.255 -p 6666 --threads 100
redis-benchmark -t get -r 100000 -n 10000000 -h 1.0.105.255 -p 6666 --threads 100
## 压测结果
### set 命令 iostat
- cpu空闲率依旧有10-20左右
- 虽然%wrqm和w_await较高，但并不持续，可能有可配置的刷盘策略？
- ioutil也并不是很高，感觉磁盘没有到瓶颈？
### set命令 top
- top命令中，kvrocks占用的核数只有2.5(62%)，但是总占用已经达到了85%，理论上这两个指标应该一致吧？
- 从我们之前的测试结果看，kvrocks的性能和kvrocks的占用的cpu(%CPU)成正比，但并不与总占用成正比，这数据是否正常呢？
### get命令 iostat
- cpu基本吃满，应该都是从内存读的，磁盘基本没有压力，能压到cpu上限这个是我们认为正常的。
GitHub link: https://github.com/apache/kvrocks/discussions/2467#discussioncomment-10241914
----
This is an automatically sent email for issues@kvrocks.apache.org.
To unsubscribe, please send an email to: issues-unsubscribe@kvrocks.apache.org
Re: [D] kvrocks压测-set命令服务器资源无法吃满 [kvrocks]
Posted by "Allen315 (via GitHub)" <gi...@apache.org>.
GitHub user Allen315 added a comment to the discussion: kvrocks压测-set命令服务器资源无法吃满
We also encountered this problem. How can we solve it?
GitHub link: https://github.com/apache/kvrocks/discussions/2467#discussioncomment-12960509
----
This is an automatically sent email for issues@kvrocks.apache.org.
To unsubscribe, please send an email to: issues-unsubscribe@kvrocks.apache.org
Re: [D] kvrocks压测-set命令服务器资源无法吃满 [kvrocks]
Posted by "VIVALXH (via GitHub)" <gi...@apache.org>.
GitHub user VIVALXH edited a comment on the discussion: kvrocks压测-set命令服务器资源无法吃满
@git-hulk 感谢回答，我们更换了阿里云的机器进行测试：
## 规格
服务器(4c8g ecs.c7.xlarge 挂载ssd云盘)
## 压测命令
redis-benchmark -t set -r 100000 -n 10000000 -h 1.0.105.255 -p 6666 --threads 100
redis-benchmark -t get -r 100000 -n 10000000 -h 1.0.105.255 -p 6666 --threads 100
## 压测结果
### set 命令 iostat
- cpu空闲率依旧有10-20左右
- 虽然%wrqm和w_await较高，但并不持续，可能有可配置的刷盘策略？
- ioutil也并不是很高，感觉磁盘没有到瓶颈？
### sset命令 top
- top命令中，kvrocks占用的核数只有2.5(62%)，但是总占用已经达到了85%，理论上这两个指标应该一致吧？
- 从我们之前的测试结果看，kvrocks的性能和kvrocks的占用的cpu(%CPU)成正比，但并不与总占用成正比，这数据是否正常呢？
### sget命令 iostat
- cpu基本吃满，应该都是从内存读的，磁盘基本没有压力，能压到cpu上限这个是我们认为正常的。
GitHub link: https://github.com/apache/kvrocks/discussions/2467#discussioncomment-10241914
----
This is an automatically sent email for issues@kvrocks.apache.org.
To unsubscribe, please send an email to: issues-unsubscribe@kvrocks.apache.org
Re: [D] kvrocks压测-set命令服务器资源无法吃满 [kvrocks]
Posted by "VIVALXH (via GitHub)" <gi...@apache.org>.
GitHub user VIVALXH edited a comment on the discussion: kvrocks压测-set命令服务器资源无法吃满
@git-hulk 感谢回答，我们更换了阿里云的机器进行测试：
## 规格
服务器(4c8g ecs.c7.xlarge 挂载ssd云盘)
## 压测命令
redis-benchmark -t set -r 100000 -n 10000000 -h 1.0.105.255 -p 6666 --threads 100
redis-benchmark -t get -r 100000 -n 10000000 -h 1.0.105.255 -p 6666 --threads 100
## 压测结果
### set 命令 iostat
- cpu空闲率依旧有10-20左右
- 虽然%wrqm和w_await较高，但并不持续，可能有可配置的刷盘策略？
- ioutil也并不是很高，感觉磁盘没有到瓶颈？
### set 命令 top htop
- top命令中，kvrocks占用的核数只有2.5(62%)，但是总占用已经达到了85%，理论上这两个指标应该一致吧？
- 从我们之前的测试结果看，限定核数1c的qps是3w qps/s，理想情况4c应该是12w qps/s，但是实际只有8w qps/s，我们的结论是kvrocks的性能和kvrocks占用的cpu(62%CPU)成正比，但并不与总占用成正比，我们提升服务器规格(8c) 后得出的结论也是这样的，这数据是否正常呢？
### get 命令 iostat
- cpu基本吃满，应该都是从内存读的，磁盘基本没有压力，能压到cpu上限这个是我们认为正常的。
GitHub link: https://github.com/apache/kvrocks/discussions/2467#discussioncomment-10241914
----
This is an automatically sent email for issues@kvrocks.apache.org.
To unsubscribe, please send an email to: issues-unsubscribe@kvrocks.apache.org
Re: [D] kvrocks压测-set命令服务器资源无法吃满 [kvrocks]
Posted by "VIVALXH (via GitHub)" <gi...@apache.org>.
GitHub user VIVALXH added a comment to the discussion: kvrocks压测-set命令服务器资源无法吃满
@PragmaTwice Thank you
1. We've tried tweaking the workers configuration, but it didn't work much.
2. We used not only redis-benchmark, but also redigo's connection pooling scheme, and the results were consistent.
redigo: https://github.com/gomodule/redigo
GitHub link: https://github.com/apache/kvrocks/discussions/2467#discussioncomment-10241981
----
This is an automatically sent email for issues@kvrocks.apache.org.
To unsubscribe, please send an email to: issues-unsubscribe@kvrocks.apache.org
Re: [D] kvrocks压测-set命令服务器资源无法吃满 [kvrocks]
Posted by "git-hulk (via GitHub)" <gi...@apache.org>.
GitHub user git-hulk added a comment to the discussion: kvrocks压测-set命令服务器资源无法吃满
First of all, you could increase the `workers` to a larger number(default is 8). And take a look at if all `soft interrupts` are located at the same CPU core(All interrupts will be in CPU-0 if NIC doesn't support multiple queues).
GitHub link: https://github.com/apache/kvrocks/discussions/2467#discussioncomment-12965308
----
This is an automatically sent email for issues@kvrocks.apache.org.
To unsubscribe, please send an email to: issues-unsubscribe@kvrocks.apache.org
Re: [D] kvrocks压测-set命令服务器资源无法吃满 [kvrocks]
Posted by "HughPenn (via GitHub)" <gi...@apache.org>.
GitHub user HughPenn added a comment to the discussion: kvrocks压测-set命令服务器资源无法吃满
<img width="1639" alt="企业微信截图_17458261857462" src="https://github.com/user-attachments/assets/396e41e5-8cf9-427d-84be-16748c4c0926" />
Previously, the workers were set to 16, and the mpstat command shows that soft interrupts (SIRQ) are distributed across different CPU cores, not just on CPU-0.
GitHub link: https://github.com/apache/kvrocks/discussions/2467#discussioncomment-12966558
----
This is an automatically sent email for issues@kvrocks.apache.org.
To unsubscribe, please send an email to: issues-unsubscribe@kvrocks.apache.org
Re: [D] kvrocks压测-set命令服务器资源无法吃满 [kvrocks]
Posted by "git-hulk (via GitHub)" <gi...@apache.org>.
GitHub user git-hulk added a comment to the discussion: kvrocks压测-set命令服务器资源无法吃满
@HughPenn Good, could you please provide your benchmark command?
GitHub link: https://github.com/apache/kvrocks/discussions/2467#discussioncomment-12966588
----
This is an automatically sent email for issues@kvrocks.apache.org.
To unsubscribe, please send an email to: issues-unsubscribe@kvrocks.apache.org
Re: [D] kvrocks压测-set命令服务器资源无法吃满 [kvrocks]
Posted by "VIVALXH (via GitHub)" <gi...@apache.org>.
GitHub user VIVALXH edited a comment on the discussion: kvrocks压测-set命令服务器资源无法吃满
@git-hulk 感谢回答，我们更换了阿里云的机器进行测试：
## 规格
服务器(4c8g ecs.c7.xlarge 挂载ssd云盘)
## 压测命令
redis-benchmark -t set -r 100000 -n 10000000 -h 1.0.105.255 -p 6666 --threads 100
redis-benchmark -t get -r 100000 -n 10000000 -h 1.0.105.255 -p 6666 --threads 100
## 压测结果
### set 命令 iostat
- cpu空闲率依旧有10-20左右
- 虽然%wrqm和w_await较高，但并不持续，可能有可配置的刷盘策略？
- ioutil也并不是很高，感觉磁盘没有到瓶颈？
### set 命令 top htop
- top命令中，kvrocks占用的核数只有2.5(62%)，但是总占用已经达到了85%，理论上这两个指标应该一致吧，怎么感觉有额外的系统调用？
- 从我们之前的测试结果看，限定核数1c的qps是3w qps/s，理想情况4c应该是12w qps/s，但是实际只有8w qps/s，我们的结论是kvrocks的性能和kvrocks占用的cpu(62%CPU)成正比，但并不与总占用成正比，我们提升服务器规格(8c) 后得出的结论也是这样的，这性能转换率是否正常呢？
### get 命令 iostat
- cpu基本吃满，应该都是从内存读的，磁盘基本没有压力，能压到cpu上限这个是我们认为正常的。
GitHub link: https://github.com/apache/kvrocks/discussions/2467#discussioncomment-10241914
----
This is an automatically sent email for issues@kvrocks.apache.org.
To unsubscribe, please send an email to: issues-unsubscribe@kvrocks.apache.org
Re: [D] kvrocks压测-set命令服务器资源无法吃满 [kvrocks]
Posted by "git-hulk (via GitHub)" <gi...@apache.org>.
GitHub user git-hulk added a comment to the discussion: kvrocks压测-set命令服务器资源无法吃满
1. 关注一下 ioutil 的情况
2. redis-benchmark 要使用(Redis 版本 >= 7)，提高 `--threads` 参数来压测
GitHub link: https://github.com/apache/kvrocks/discussions/2467#discussioncomment-10241152
----
This is an automatically sent email for issues@kvrocks.apache.org.
To unsubscribe, please send an email to: issues-unsubscribe@kvrocks.apache.org