You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@apisix.apache.org by GitBox <gi...@apache.org> on 2022/04/12 02:51:33 UTC
[GitHub] [apisix] Mar1usSo opened a new issue, #6826: help request: Jmeter benchmark admin-api,but just 30 qps
Mar1usSo opened a new issue, #6826:
URL: https://github.com/apache/apisix/issues/6826
### Description
We use Jmeter to benchmark admin-api(create upstream),But qps just reach 30,can't improve anymore.
We found every metrics in apisix is fine (include cpu,memory.disk IO)，and there is no error on error.log
But CPU usage of ETCD is very high,almost 100%
- VM:
4C8G*3 200G HDD ETCD
4C8G*2 200G HDD APISIX
Here is some metrics:
1. cpu usage of etcd
2. io of disk
3. etcd status
Finally I use etcd-benckmark to benchmark etcd,etcd ops can reach more than 3000 put/second
https://github.com/etcd-io/etcd/tree/v3.3.2/tools/benchmark
So just a wired guess,the way apisix call etcd should be optimized
Here with the Connection state of apisix
```
[root@pekshcsitd44045 ~]# ss -ant | grep 2379
TIME-WAIT 0 0 192.168.1.11:41790 192.168.1.208:2379
ESTAB 0 0 192.168.1.11:41348 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41664 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41824 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41774 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41398 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41680 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41822 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41606 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41630 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41768 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41618 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41544 192.168.1.208:2379
ESTAB 0 0 192.168.1.11:41286 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41548 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41460 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41342 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41510 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41696 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41662 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41592 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41496 192.168.1.208:2379
ESTAB 0 0 192.168.1.11:41322 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41562 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41414 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41530 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41764 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41454 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41682 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41782 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41610 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41372 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41552 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41704 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41584 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41778 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41720 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41388 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41538 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41448 192.168.1.208:2379
ESTAB 0 0 192.168.1.11:41340 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41450 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41598 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41732 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41556 192.168.1.208:2379
ESTAB 0 0 192.168.1.11:41316 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41706 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41628 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41426 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41586 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41498 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41436 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41792 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41452 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41716 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41412 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41730 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41508 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41708 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41752 192.168.1.208:2379
ESTAB 0 0 192.168.1.11:41310 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41482 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41594 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41532 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41394 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41396 192.168.1.208:2379
ESTAB 0 0 192.168.1.11:41836 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41620 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41636 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41784 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41740 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41408 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41718 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41776 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41698 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41368 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41462 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41818 192.168.1.208:2379
ESTAB 0 0 192.168.1.11:41318 192.168.1.208:2379
ESTAB 0 0 192.168.1.11:41336 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41626 192.168.1.208:2379
ESTAB 0 0 192.168.1.11:41356 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41754 192.168.1.208:2379
ESTAB 0 0 192.168.1.11:41362 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41650 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41518 192.168.1.208:2379
ESTAB 0 0 192.168.1.11:41280 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41364 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41526 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41560 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41554 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41622 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41442 192.168.1.208:2379
ESTAB 0 0 192.168.1.11:41652 192.168.1.208:2379
ESTAB 0 0 192.168.1.11:41834 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41522 192.168.1.208:2379
ESTAB 0 0 192.168.1.11:41284 192.168.1.208:2379
ESTAB 0 0 192.168.1.11:41278 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41742 192.168.1.208:2379
ESTAB 0 0 192.168.1.11:41314 192.168.1.208:2379
ESTAB 0 0 192.168.1.11:41850 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41444 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41374 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41608 192.168.1.208:2379
ESTAB 0 0 192.168.1.11:41294 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41590 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41804 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41808 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41588 192.168.1.208:2379
ESTAB 0 0 192.168.1.11:41842 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41806 192.168.1.208:2379
ESTAB 0 0 192.168.1.11:41326 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41488 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41756 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41492 192.168.1.208:2379
ESTAB 0 0 192.168.1.11:41272 192.168.1.208:2379
ESTAB 0 0 192.168.1.11:41352 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41438 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41564 192.168.1.208:2379
ESTAB 0 0 192.168.1.11:41516 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41672 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41678 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41632 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41700 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41382 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41406 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41546 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41386 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41724 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41796 192.168.1.208:2379
ESTAB 0 0 192.168.1.11:41308 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41616 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41478 192.168.1.208:2379
ESTAB 0 0 192.168.1.11:41828 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41432 192.168.1.208:2379
ESTAB 0 0 192.168.1.11:41302 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41384 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41520 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41568 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41746 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41802 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41514 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41474 192.168.1.208:2379
ESTAB 0 0 192.168.1.11:41290 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41820 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41738 192.168.1.208:2379
ESTAB 0 0 192.168.1.11:41358 192.168.1.208:2379
ESTAB 0 0 192.168.1.11:41346 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41766 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41558 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41434 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41370 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41758 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41524 192.168.1.208:2379
ESTAB 0 0 192.168.1.11:41312 192.168.1.208:2379
ESTAB 0 0 192.168.1.11:41844 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41744 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41574 192.168.1.208:2379
ESTAB 0 0 192.168.1.11:41330 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41714 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41422 192.168.1.208:2379
ESTAB 0 0 192.168.1.11:41300 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41464 192.168.1.208:2379
ESTAB 0 0 192.168.1.11:41830 192.168.1.208:2379
ESTAB 0 0 192.168.1.11:41274 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41667 192.168.1.208:2379
ESTAB 0 0 192.168.1.11:41712 192.168.1.208:2379
ESTAB 0 0 192.168.1.11:41848 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41566 192.168.1.208:2379
ESTAB 0 0 192.168.1.11:41288 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41596 192.168.1.208:2379
ESTAB 0 0 192.168.1.11:41334 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41748 192.168.1.208:2379
ESTAB 0 0 192.168.1.11:41268 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41446 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41612 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41502 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41826 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41428 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41800 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41750 192.168.1.208:2379
ESTAB 0 0 192.168.1.11:41333 192.168.1.208:2379
ESTAB 0 0 192.168.1.11:41350 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41540 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41780 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41420 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41762 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41404 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41424 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41676 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41648 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41642 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41760 192.168.1.208:2379
ESTAB 0 0 192.168.1.11:41840 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41686 192.168.1.208:2379
ESTAB 0 0 192.168.1.11:41832 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41600 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41638 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41656 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41470 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41390 192.168.1.208:2379
ESTAB 0 0 192.168.1.11:41320 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41440 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41494 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41480 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41512 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41816 192.168.1.208:2379
ESTAB 0 0 192.168.1.11:41392 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41674 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41654 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41658 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41684 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41570 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41702 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41772 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41786 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41468 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41486 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41726 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41528 192.168.1.208:2379
ESTAB 0 0 192.168.1.11:41846 192.168.1.208:2379
ESTAB 0 0 192.168.1.11:41292 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41812 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41550 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41380 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41602 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41810 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41456 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41794 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41670 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41634 192.168.1.208:2379
ESTAB 0 0 192.168.1.11:41324 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41722 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41344 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41580 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41736 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41798 192.168.1.208:2379
ESTAB 0 0 192.168.1.11:41534 192.168.1.208:2379
ESTAB 0 0 192.168.1.11:41298 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41542 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41694 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41430 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41666 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41692 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41576 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41624 192.168.1.208:2379
ESTAB 0 0 192.168.1.11:41338 192.168.1.208:2379
ESTAB 0 0 192.168.1.11:41254 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41604 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41710 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41640 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41466 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41578 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41458 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41400 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41728 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41418 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41360 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41506 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41688 192.168.1.208:2379
ESTAB 0 0 192.168.1.11:41282 192.168.1.208:2379
ESTAB 0 0 192.168.1.11:41304 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41472 192.168.1.208:2379
ESTAB 0 0 192.168.1.11:41296 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41660 192.168.1.208:2379
ESTAB 0 0 192.168.1.11:41354 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41410 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41734 192.168.1.208:2379
FIN-WAIT-2 0 0 192.168.1.11:41838 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41572 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41614 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41366 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41788 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41222 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41814 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41646 192.168.1.208:2379
ESTAB 0 0 192.168.1.11:41276 192.168.1.208:2379
ESTAB 0 0 192.168.1.11:41328 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41376 192.168.1.208:2379
FIN-WAIT-1 0 1 192.168.1.11:41244 192.168.1.208:2379
ESTAB 0 0 192.168.1.11:41306 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41490 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41582 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41770 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41504 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41644 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41378 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41500 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41416 192.168.1.208:2379
ESTAB 0 0 192.168.1.11:41484 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41536 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41476 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41402 192.168.1.208:2379
TIME-WAIT 0 0 192.168.1.11:41690 192.168.1.208:2379
```
### Environment
- APISIX version (run `apisix version`):2.11.0
- Operating system (run `uname -a`):Linux 3.10.0-1160.45.1.el7.x86_64 #1 SMP Wed Oct 13 17:20:51 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
- OpenResty / Nginx version (run `openresty -V` or `nginx -V`):
nginx version: openresty/1.19.3.2
built by gcc 9.3.1 20200408 (Red Hat 9.3.1-2) (GCC)
built with OpenSSL 1.1.1l 24 Aug 2021 (running with OpenSSL 1.1.1k 25 Mar 2021)
TLS SNI support enabled
configure arguments: --prefix=/usr/local/openresty/nginx --with-cc-opt='-O2 -DAPISIX_BASE_VER=1.19.3.2.2 -DNGX_LUA_ABORT_AT_PANIC -I/usr/local/openresty/zlib/include -I/usr/local/openresty/pcre/include -I/usr/local/openresty/openssl111/include' --add-module=../ngx_devel_kit-0.3.1 --add-module=../echo-nginx-module-0.62 --add-module=../xss-nginx-module-0.06 --add-module=../ngx_coolkit-0.2 --add-module=../set-misc-nginx-module-0.32 --add-module=../form-input-nginx-module-0.12 --add-module=../encrypted-session-nginx-module-0.08 --add-module=../srcache-nginx-module-0.32 --add-module=../ngx_lua-0.10.19 --add-module=../ngx_lua_upstream-0.07 --add-module=../headers-more-nginx-module-0.33 --add-module=../array-var-nginx-module-0.05 --add-module=../memc-nginx-module-0.19 --add-module=../redis2-nginx-module-0.15 --add-module=../redis-nginx-module-0.3.7 --add-module=../ngx_stream_lua-0.0.9 --with-ld-opt='-Wl,-rpath,/usr/local/openresty/luajit/lib -Wl,-rpath,/usr/local/openresty/wasmtime-c-api
/lib -L/usr/local/openresty/zlib/lib -L/usr/local/openresty/pcre/lib -L/usr/local/openresty/openssl111/lib -Wl,-rpath,/usr/local/openresty/zlib/lib:/usr/local/openresty/pcre/lib:/usr/local/openresty/openssl111/lib' --add-module=/tmp/tmp.wRcqhSzQQC/openresty-1.19.3.2/../mod_dubbo --add-module=/tmp/tmp.wRcqhSzQQC/openresty-1.19.3.2/../ngx_multi_upstream_module --add-module=/tmp/tmp.wRcqhSzQQC/openresty-1.19.3.2/../apisix-nginx-module --add-module=/tmp/tmp.wRcqhSzQQC/openresty-1.19.3.2/../wasm-nginx-module --add-module=/tmp/tmp.wRcqhSzQQC/openresty-1.19.3.2/../lua-var-nginx-module --with-poll_module --with-pcre-jit --with-stream --with-stream_ssl_module --with-stream_ssl_preread_module --with-http_v2_module --without-mail_pop3_module --without-mail_imap_module --without-mail_smtp_module --with-http_stub_status_module --with-http_realip_module --with-http_addition_module --with-http_auth_request_module --with-http_secure_link_module --with-http_random_index_module --with-http_gzip_stati
c_module --with-http_sub_module --with-http_dav_module --with-http_flv_module --with-http_mp4_module --with-http_gunzip_module --with-threads --with-compat --with-stream --with-http_ssl_module
- etcd version, if relevant (run `curl http://127.0.0.1:9090/v1/server_info`):3.4.16
- APISIX Dashboard version, if relevant: `not relevant`
- Plugin runner version, for issues related to plugin runners: `not relevant`
- LuaRocks version, for installation issues (run `luarocks --version`): `not relevant`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] Mar1usSo commented on issue #6826: help request: Jmeter benchmark admin-api,but just 30 qps
Posted by GitBox <gi...@apache.org>.
Mar1usSo commented on issue #6826:
URL: https://github.com/apache/apisix/issues/6826#issuecomment-1096923061
> It's obvious the bottleneck is ETCD since it's a CP system with Raft.
>
> If you think there is something that the Admin API can be enhanced, you can benchmark the ETCD put operation directly, with the same data (upstream body).
Just done with benchmark the ETCD with same 'put' reqeust.
But the result jmeter result reach **707 qps with KeepAlive** and **241 qps with 'No KeepAlive'**
However,qps of Admin Api is **30**
- **etcd-KeepAlive**
- **etcd-No KeepAlive**
- Here is my test reqeust
```
curl -L http://192.168.1.208:2379/v3/kv/put -H 'Authorization:RPihXgWbiYLwCRtb.1169585' -X POST -d '{"key":"L2FwaXNpeC91cHN0cmVhbXMvOTkwMDAwMDAwMDAwMDAzNzI2NzI=","value":"e3RpbWVvdXQiOnsiY29ubmVjdCI6NiwicmVhZCI6Niwic2VuZCI6Nn0sImNyZWF0ZV90aW1lIjoxNjQ5NzczMjQ5LCJzY2hlbWUiOiJodHRwIiwiaWQiOiI5OTAwMDAwMDAwMDAwMDM3MjY3MiIsInBhc3NfaG9zdCI6InBhc3MiLCJkZXNjIjoiIiwidXBkYXRlX3RpbWUiOjE2NDk3NzMyNDksIm5hbWUiOiJzdWppZWZhbmd0b2tlbnRlc3QwMjE2LXRlc3QxLTIwMjIwNDEyMjIyMDQ5LW56eWEiLCJjaGVja3MiOnsiYWN0aXZlIjp7InRpbWVvdXQiOjUsInR5cGUiOiJodHRwIiwiaHR0cHNfdmVyaWZ5X2NlcnRpZmljYXRlIjp0cnVlLCJ1bmhlYWx0aHkiOnsiaHR0cF9mYWlsdXJlcyI6MiwiaW50ZXJ2YWwiOjMsInRpbWVvdXRzIjo1LCJodHRwX3N0YXR1c2VzIjpbNDA0LDUwMCw1MDEsNTAyLDUwMyw1MDQsNTA1XSwidGNwX2ZhaWx1cmVzIjoyfSwiaGVhbHRoeSI6eyJzdWNjZXNzZXMiOjUsImh0dHBfc3RhdHVzZXMiOlsyMDAsMzAxLDMwMl0sImludGVydmFsIjozfSwiY29uY3VycmVuY3kiOjEwLCJodHRwX3BhdGgiOiJcLyJ9fSwidHlwZSI6ImNoYXNoIiwiaGFzaF9vbiI6ImhlYWRlciIsInhhcGlLZXkiOiJGfkMwZVMzNFwvMkw3fDZZRS5tMWNuUzhcL2dYVUV+LV5qIiwiZW5k
cG9pbnQiOiJodHRwOlwvXC8xMC42OC4yMzkuMTA2OiA5MDgwIiwibm9kZXMiOnsiMTAuMjguODQuMjA5OiAyNTA5NSI6MX0sImtleSI6Imhvc3QifQ=="}'
```
- ** Admin Api **
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] Mar1usSo commented on issue #6826: help request: Jmeter benchmark admin-api,but just 30 qps
Posted by GitBox <gi...@apache.org>.
Mar1usSo commented on issue #6826:
URL: https://github.com/apache/apisix/issues/6826#issuecomment-1095906786
>
No error.log relevanted etcd ,just about load data from etcd as below
```
[root@test logs]# tail -f error.log
2022/04/12 11:09:27 [error] 21448#21448: *7487591 [lua] config_etcd.lua:569: failed to fetch data from etcd: grpc: received message larger than max (59785648 vs. 4194304), etcd key: /apisix/upstreams, context: ngx.timer
2022/04/12 11:09:38 [error] 21452#21452: *7477987 [lua] config_etcd.lua:569: failed to fetch data from etcd: grpc: received message larger than max (59785648 vs. 4194304), etcd key: /apisix/upstreams, context: ngx.timer
2022/04/12 11:09:40 [error] 21449#21449: *7487099 [lua] config_etcd.lua:569: failed to fetch data from etcd: grpc: received message larger than max (59785648 vs. 4194304), etcd key: /apisix/upstreams, context: ngx.timer
2022/04/12 11:09:47 [error] 21447#21447: *7482846 [lua] config_etcd.lua:569: failed to fetch data from etcd: grpc: received message larger than max (59785648 vs. 4194304), etcd key: /apisix/upstreams, context: ngx.timer
``
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] zhixiongdu027 commented on issue #6826: help request: Jmeter benchmark admin-api,but just 30 qps
Posted by GitBox <gi...@apache.org>.
zhixiongdu027 commented on issue #6826:
URL: https://github.com/apache/apisix/issues/6826#issuecomment-1098615211
> Actually,We are about to build a apisix management console like Dashboard
It's a great, I'm looking forward to it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] tzssangglass commented on issue #6826: help request: Jmeter benchmark admin-api,but just 30 qps
Posted by GitBox <gi...@apache.org>.
tzssangglass commented on issue #6826:
URL: https://github.com/apache/apisix/issues/6826#issuecomment-1098186779
> We encountered this problem during stress testing（we expect the qps should be large 50）
I got, great scenario (although this issue has nothing to do with APISIX). APISIX has plans to connect to ETCD via gRPC, but this is a long time coming.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] tzssangglass commented on issue #6826: help request: Jmeter benchmark admin-api,but just 30 qps
Posted by GitBox <gi...@apache.org>.
tzssangglass commented on issue #6826:
URL: https://github.com/apache/apisix/issues/6826#issuecomment-1095914312
It should be noted that each worker in APISIX maintains 13 connection to etcd, your configuration is 4 C, and by default there are 4 workers.
I haven't calibrated how etcd benchmark is tested, but apparently it's not appropriate to use etcd benchmark as a reference.
In fact, in the APISIX V3 roadmap, we will optimize the connectivity and performance between APISIX and etcd.ref: https://github.com/apache/apisix/issues/6473
We can continue to explore why the qps is only 30.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] github-actions[bot] commented on issue #6826: help request: Jmeter benchmark admin-api,but just 30 qps
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #6826:
URL: https://github.com/apache/apisix/issues/6826#issuecomment-1490034784
This issue has been marked as stale due to 350 days of inactivity. It will be closed in 2 weeks if no further activity occurs. If this issue is still relevant, please simply write any comment. Even if closed, you can still revive the issue at any time or discuss it on the dev@apisix.apache.org list. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] Mar1usSo commented on issue #6826: help request: Jmeter benchmark admin-api,but just 30 qps
Posted by GitBox <gi...@apache.org>.
Mar1usSo commented on issue #6826:
URL: https://github.com/apache/apisix/issues/6826#issuecomment-1098069610
> BTW, I'm not sure what the point of testing the performance of the admin API is.
Actually,We are about to build a apisix management console like Dashboard，in order to meet the needs of multi-tenancy & multi apisix management.
We encountered this problem during stress testing（we expect the qps should be large 50）
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] tzssangglass commented on issue #6826: help request: Jmeter benchmark admin-api,but just 30 qps
Posted by GitBox <gi...@apache.org>.
tzssangglass commented on issue #6826:
URL: https://github.com/apache/apisix/issues/6826#issuecomment-1095901148
any error logs in APISIX about etcd?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] github-actions[bot] commented on issue #6826: help request: Jmeter benchmark admin-api,but just 30 qps
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #6826:
URL: https://github.com/apache/apisix/issues/6826#issuecomment-1508268990
This issue has been closed due to lack of activity. If you think that is incorrect, or the issue requires additional review, you can revive the issue at any time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] tokers commented on issue #6826: help request: Jmeter benchmark admin-api,but just 30 qps
Posted by GitBox <gi...@apache.org>.
tokers commented on issue #6826:
URL: https://github.com/apache/apisix/issues/6826#issuecomment-1097453321
Note when Admin API does the upstream put operation, it'll try to fetch the key (before starting the put), so actually the upstream put operation is not equivalent to the single HTTP put.
See https://github.com/apache/apisix/blob/master/apisix/admin/utils.lua#L51.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] tzssangglass commented on issue #6826: help request: Jmeter benchmark admin-api,but just 30 qps
Posted by GitBox <gi...@apache.org>.
tzssangglass commented on issue #6826:
URL: https://github.com/apache/apisix/issues/6826#issuecomment-1097495578
BTW, I'm not sure what the point of testing the performance of the admin API is.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] tokers commented on issue #6826: help request: Jmeter benchmark admin-api,but just 30 qps
Posted by GitBox <gi...@apache.org>.
tokers commented on issue #6826:
URL: https://github.com/apache/apisix/issues/6826#issuecomment-1096448139
It's obvious the bottleneck is ETCD since it's a CP system with Raft.
If you think there is something that the Admin API can be enhanced, you can benchmark the ETCD put operation directly, with the same data (upstream body).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] github-actions[bot] closed issue #6826: help request: Jmeter benchmark admin-api,but just 30 qps
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #6826: help request: Jmeter benchmark admin-api,but just 30 qps
URL: https://github.com/apache/apisix/issues/6826
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org