You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@apisix.apache.org by GitBox <gi...@apache.org> on 2022/06/13 09:37:32 UTC
[GitHub] [apisix] lzxuan opened a new issue, #7244: help request: subpar performance of apisix in kubernetes
lzxuan opened a new issue, #7244:
URL: https://github.com/apache/apisix/issues/7244
### Description
Following [this Slack thread](https://the-asf.slack.com/archives/CUC5MN17A/p1653900217440819), where I benchmarked APISIX running on a single-node K3s of version `v1.23.6+k3s1` atop `Ubuntu 20.04` VM with `8-core Intel Xeon Processor (Skylake, IBRS)` CPU and `8192MB` memory, the QPS obtained was way below than what was stated in [the official benchmark page](https://apisix.apache.org/docs/apisix/benchmark/#benchmark-test-for-reverse-proxy-enabled-2-plugins). The benchmark result was obtained using `wrk -d 60 --latency` (following [the official benchmark page](https://apisix.apache.org/docs/apisix/benchmark/#benchmark-test-for-reverse-proxy-enabled-2-plugins)):
OpenResty upstream with 1KB response:
| Number of CPU Core | 1 |
|--------------------|:--------:|
| Average Latency | 178.18us |
| Average Req/Sec | 30.70k |
With APISIX (without any plugin enabled):
| Number of CPU Core | 1 | 2 | 4 |
|--------------------|:------:|:--------:|:--------:|
| Average Latency | 0.94ms | 559.90us | 391.50us |
| Average Req/Sec | 5.49k | 9.36k | 14.62k |
With APISIX (with only prometheus plugin enabled):
| Number of CPU Core | 1 | 2 | 4 |
|--------------------|:------:|:--------:|:--------:|
| Average Latency | 1.19ms | 678.52us | 460.13us |
| Average Req/Sec | 4.29k | 7.74k | 12.58k |
However, I was later being pointed out that the wrk command should be `wrk -d 5 -c 16` (following [the official benchmark script](https://github.com/apache/apisix/blob/master/benchmark/run.sh)). I then reran my benchmark on n1-highcpu-8 (8 vCPUs, 7.2 GB memory) on Google Cloud, with APISIX still running in a K3s cluster. The benchmark result of APISIX with `worker_processes` set to 1:
```
$ wrk -d 5 -c 16 http://up.test.com:30080
Running 5s test @ http://up.test.com:30080
2 threads and 16 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 1.72ms 1.05ms 32.26ms 99.08%
Req/Sec 4.79k 497.22 5.71k 70.00%
47637 requests in 5.00s, 8.72MB read
Requests/sec: 9524.89
Transfer/sec: 1.74MB
```
The QPS did increase but it is highly suspected to be solely due to the change of the wrk command to using more connections, as I obtained similar figures when running the second wrk command on the first Ubuntu VM. Nonetheless, the new QPS obtained is still way below the 15k QPS stated in the official benchmark page.
**Steps to Reproduce:**
1. Create `n1-highcpu-8` compute instance and SSH to it.
2. Git clone and helm install (apisix)[https://github.com/lzxuan/apisix-prometheus-helm-chart].
3. Kubectl apply (upstream)[https://gist.github.com/lzxuan/f7dbc9e9a7c30ed273463e1011b77e99].
4. `sudo echo '127.0.0.1 up.test.com' >> /etc/hosts`.
5. Add APISIX route to upstream with host set to `up.test.com`.
6. Install wrk and `wrk -d 5 -c 16 http://up.test.com:30080`.
### Environment
- ASISIX Helm chart version: `0.10.0`
- APISIX version: `2.14.1`
- Operating system: `Linux instance-1 5.13.0-1031-gcp #37~20.04.1-Ubuntu SMP Thu Jun 9 13:25:43 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux`
- OpenResty version: `1.19.9.1`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] github-actions[bot] closed issue #7244: help request: subpar performance of apisix in kubernetes
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #7244: help request: subpar performance of apisix in kubernetes
URL: https://github.com/apache/apisix/issues/7244
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] lzxuan commented on issue #7244: help request: subpar performance of apisix in kubernetes
Posted by GitBox <gi...@apache.org>.
lzxuan commented on issue #7244:
URL: https://github.com/apache/apisix/issues/7244#issuecomment-1154757180
@tzssangglass I see, thanks a bunch!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] github-actions[bot] commented on issue #7244: help request: subpar performance of apisix in kubernetes
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #7244:
URL: https://github.com/apache/apisix/issues/7244#issuecomment-1588976087
This issue has been closed due to lack of activity. If you think that is incorrect, or the issue requires additional review, you can revive the issue at any time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] moonming commented on issue #7244: help request: subpar performance of apisix in kubernetes
Posted by GitBox <gi...@apache.org>.
moonming commented on issue #7244:
URL: https://github.com/apache/apisix/issues/7244#issuecomment-1154570949
First of all, every major version of Apache APISIX is released, and there will be regressions in performance tests, and there will be no such performance degradation. Therefore, this data is caused by the inappropriate method of your performance test.
Then, during the stress test, check whether the CPU is fully loaded, whether there is an error log, whether there is a problem with the network settings(maybe K3s), etc.
At the same time, you can use perf, flame graph, etc. to find performance problems.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] lzxuan commented on issue #7244: help request: subpar performance of apisix in kubernetes
Posted by GitBox <gi...@apache.org>.
lzxuan commented on issue #7244:
URL: https://github.com/apache/apisix/issues/7244#issuecomment-1154609891
@moonming Thanks for your reply! First off, I belief the performance of APISIX is not degraded for every release and the result at the official benchmark page holds true. I would be very grateful if you could detail a little more on the inappropriate method of my performance test and perhaps what can be amended.
To add more details, the involved CPU cores were fully loaded during the test, no error log could be found, and the K3s was installed as-is via https://get.k3s.io so it was using the default network settings (I have updated the Steps to Reproduce, thanks for pointing out!). Anything that I can tune in order to obtain more performance?
FYI, I have tried various ways to generate the flame graph, including stapxx, systemtap, perf and BPF Compiler Collection, however I could not get the symbols resolved, since it was running in K3s. Would deeply appreciate if you could enlighten me!!
Thanks again.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] tzssangglass commented on issue #7244: help request: subpar performance of apisix in kubernetes
Posted by GitBox <gi...@apache.org>.
tzssangglass commented on issue #7244:
URL: https://github.com/apache/apisix/issues/7244#issuecomment-1154631985
First of all, the official test benchmark is based on cloud servers, while your test is based on virtualization (k8s or k3s).
As a rule of thumb, performance testing in virtualized containers requires tuning of virtualization platform or container parameters (which I am not very familiar with).
BTW, we will be discussing performance testing related topics in our community meetings, ref: https://github.com/apache/apisix/discussions/7241
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] fairyqb commented on issue #7244: help request: subpar performance of apisix in kubernetes
Posted by GitBox <gi...@apache.org>.
fairyqb commented on issue #7244:
URL: https://github.com/apache/apisix/issues/7244#issuecomment-1154518157
@spacewander @membphis @moonming @tzssangglass @Firstsawyou
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] github-actions[bot] commented on issue #7244: help request: subpar performance of apisix in kubernetes
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #7244:
URL: https://github.com/apache/apisix/issues/7244#issuecomment-1568156715
This issue has been marked as stale due to 350 days of inactivity. It will be closed in 2 weeks if no further activity occurs. If this issue is still relevant, please simply write any comment. Even if closed, you can still revive the issue at any time or discuss it on the dev@apisix.apache.org list. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org