You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@apisix.apache.org by GitBox <gi...@apache.org> on 2022/08/29 07:35:55 UTC
[GitHub] [apisix] akalittle opened a new issue, #7811: help request: wrk qps test result is weird
akalittle opened a new issue, #7811:
URL: https://github.com/apache/apisix/issues/7811
### Description
Compare to the [qps page](https://apisix.apache.org/docs/apisix/benchmark/)
the result of my test is extremely low.
Use wrk to test.
```
wrk -t4 -c3000 -d30s https://api.xxx.cc/hello/ping
Running 30s test @ https://api.xxx.cc/hello/ping
4 threads and 3000 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 869.55ms 428.65ms 2.00s 58.45%
Req/Sec 416.93 283.06 1.43k 63.96%
42106 requests in 30.09s, 7.92MB read
Socket errors: connect 15, read 0, write 0, timeout 3531
Requests/sec: 1399.17
Transfer/sec: 269.60KB
```
Config for helm chart.
```
#
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
global:
imagePullSecrets: []
apisix:
# Enable or disable Apache APISIX itself
# Set it to false and ingress-controller.enabled=true will deploy only ingress-controller
enabled: true
# Enable nginx IPv6 resolver
enableIPv6: true
# Use Pod metadata.uid as the APISIX id.
setIDFromPodUID: false
customLuaSharedDicts: []
# - name: foo
# size: 10k
# - name: bar
# size: 1m
luaModuleHook:
enabled: false
# extend lua_package_path to load third party code
luaPath: ""
# the hook module which will be used to inject third party code into APISIX
# use the lua require style like: "module.say_hello"
hookPoint: ""
# configmap that stores the codes
configMapRef:
name: ""
# mounts decides how to mount the codes to the container.
mounts:
- key: ""
path: ""
enableCustomizedConfig: false
customizedConfig: {}
image:
repository: apache/apisix
pullPolicy: IfNotPresent
# Overrides the image tag whose default is the chart appVersion.
tag: 2.15.0-alpine
# Use a `DaemonSet` or `Deployment`
kind: Deployment
# kind is DaemonSet,replicaCount not become effective
replicaCount: 1
podAnnotations: {}
podSecurityContext: {}
# fsGroup: 2000
securityContext: {}
# capabilities:
# drop:
# - ALL
# readOnlyRootFilesystem: true
# runAsNonRoot: true
# runAsUser: 1000
# See https://kubernetes.io/docs/tasks/run-application/configure-pdb/ for more details
podDisruptionBudget:
enabled: false
minAvailable: 90%
maxUnavailable: 1
resources: {}
# We usually recommend not to specify default resources and to leave this as a conscious
# choice for the user. This also increases chances charts run on environments with little
# resources, such as Minikube. If you do want to specify resources, uncomment the following
# lines, adjust them as necessary, and remove the curly braces after 'resources:'.
# limits:
# cpu: 100m
# memory: 128Mi
# requests:
# cpu: 100m
# memory: 128Mi
nodeSelector: {}
tolerations: []
affinity: {}
# If true, it will sets the anti-affinity of the Pod.
podAntiAffinity:
enabled: false
# timezone is the timezone where apisix uses.
# For example: "UTC" or "Asia/Shanghai"
# This value will be set on apisix container's environment variable TZ.
# You may need to set the timezone to be consistent with your local time zone,
# otherwise the apisix's logs may used to retrieve event maybe in wrong timezone.
timezone: ""
# extraEnvVars An array to add extra env vars
# e.g:
# extraEnvVars:
# - name: FOO
# value: "bar"
# - name: FOO2
# valueFrom:
# secretKeyRef:
# name: SECRET_NAME
# key: KEY
extraEnvVars: []
nameOverride: ""
fullnameOverride: ""
gateway:
type: NodePort
# If you want to keep the client source IP, you can set this to Local.
# ref: https://kubernetes.io/docs/tasks/access-application-cluster/create-external-load-balancer/#preserving-the-client-source-ip
externalTrafficPolicy: Cluster
# type: LoadBalancer
# annotations:
# service.beta.kubernetes.io/aws-load-balancer-type: nlb
externalIPs: []
http:
enabled: true
servicePort: 80
containerPort: 9080
tls:
enabled: false
servicePort: 443
containerPort: 9443
existingCASecret: ""
certCAFilename: ""
http2:
enabled: true
stream: # L4 proxy (TCP/UDP)
enabled: false
only: false
tcp: []
udp: []
ingress:
enabled: false
annotations: {}
# kubernetes.io/ingress.class: nginx
# kubernetes.io/tls-acme: "true"
hosts:
- host: apisix.local
paths: []
tls: []
# - secretName: apisix-tls
# hosts:
# - chart-example.local
admin:
# Enable Admin API
enabled: true
# admin service type
type: ClusterIP
# loadBalancerIP: a.b.c.d
# loadBalancerSourceRanges:
# - "143.231.0.0/16"
externalIPs: []
#
port: 9180
servicePort: 9180
# Admin API support CORS response headers
cors: true
# Admin API credentials
credentials:
admin: edd1c9f034335f136f87ad84b625c8f1
viewer: 4054f7cf07e344346cd3f287985e76a2
allow:
# The ip range for allowing access to Apache APISIX
ipList:
- 127.0.0.1/24
nginx:
workerRlimitNofile: "20480"
workerConnections: "10620"
workerProcesses: auto
enableCPUAffinity: true
# APISIX plugins to be enabled
plugins:
- api-breaker
- authz-keycloak
- basic-auth
- batch-requests
- consumer-restriction
- cors
- echo
- fault-injection
- grpc-transcode
- hmac-auth
- http-logger
- ip-restriction
- ua-restriction
- jwt-auth
- kafka-logger
- key-auth
- limit-conn
- limit-count
- limit-req
- node-status
- openid-connect
- authz-casbin
- prometheus
- proxy-cache
- proxy-mirror
- proxy-rewrite
- redirect
- referer-restriction
- request-id
- request-validation
- response-rewrite
- serverless-post-function
- serverless-pre-function
- sls-logger
- syslog
- tcp-logger
- udp-logger
- uri-blocker
- wolf-rbac
- zipkin
- traffic-split
- gzip
- real-ip
- ext-plugin-pre-req
- ext-plugin-post-req
- opentelemetry
stream_plugins:
- mqtt-proxy
- ip-restriction
- limit-conn
pluginAttrs:
opentelemetry:
trace_id_source: random
resource:
service.name: APISIX
# tenant.id: business_id
collector:
address: opentelemetry-deploy-opentelemetry-collector.apisix-istio.svc.cluster.local:4318
request_timeout: 3
batch_span_processor:
drop_on_queue_full: false
max_queue_size: 6
batch_timeout: 2
inactive_timeout: 1
max_export_batch_size: 2
extPlugin:
enabled: false
cmd: ["/path/to/apisix-plugin-runner/runner", "run"]
# customPlugins allows you to mount your own HTTP plugins.
customPlugins:
enabled: false
# the lua_path that tells APISIX where it can find plugins,
# note the last ';' is required.
luaPath: "/opts/custom_plugins/?.lua"
plugins:
# plugin name.
- name: ""
# plugin attrs
attrs: {}
# plugin codes can be saved inside configmap object.
configMap:
# name of configmap.
name: ""
# since keys in configmap is flat, mountPath allows to define the mount
# path, so that plugin codes can be mounted hierarchically.
mounts:
- key: ""
path: ""
- key: ""
path: ""
updateStrategy: {}
# type: RollingUpdate
extraVolumes: []
# - name: extras
# emptyDir: {}
extraVolumeMounts: []
# - name: extras
# mountPath: /usr/share/extras
# readOnly: true
discovery:
enabled: false
registry:
# Integration service discovery registry. E.g eureka\dns\nacos\consul_kv
# reference:
# https://apisix.apache.org/docs/apisix/discovery#configuration-for-eureka
# https://apisix.apache.org/docs/apisix/discovery/dns#service-discovery-via-dns
# https://apisix.apache.org/docs/apisix/discovery/consul_kv#configuration-for-consul-kv
# https://apisix.apache.org/docs/apisix/discovery/nacos#configuration-for-nacos
#
# an eureka example:
# eureka:
# host:
# - "http://${username}:${password}@${eureka_host1}:${eureka_port1}"
# - "http://${username}:${password}@${eureka_host2}:${eureka_port2}"
# prefix: "/eureka/"
# fetch_interval: 30
# weight: 100
# timeout:
# connect: 2000
# send: 2000
# read: 5000
# access log and error log configuration
logs:
enableAccessLog: true
accessLog: "/dev/stdout"
accessLogFormat: '$remote_addr - $remote_user [$time_local] $http_host \"$request\" $status $body_bytes_sent $request_time \"$http_referer\" \"$http_user_agent\" $upstream_addr $upstream_status $upstream_response_time \"$upstream_scheme://$upstream_host$upstream_uri\"'
accessLogFormatEscape: default
errorLog: "/dev/stderr"
errorLogLevel: "warn"
dns:
resolvers:
- 127.0.0.1
- 172.20.0.10
- 114.114.114.114
- 223.5.5.5
- 1.1.1.1
- 8.8.8.8
validity: 30
timeout: 5
initContainer:
image: busybox
tag: 1.28
autoscaling:
enabled: false
minReplicas: 1
maxReplicas: 100
targetCPUUtilizationPercentage: 80
targetMemoryUtilizationPercentage: 80
# Custom configuration snippet.
configurationSnippet:
main: |
httpStart: |
httpEnd: |
httpSrv: |
httpAdmin: |
stream: |
# Observability configuration.
# ref: https://apisix.apache.org/docs/apisix/plugins/prometheus/
serviceMonitor:
enabled: true
# namespace where the serviceMonitor is deployed, by default, it is the same as the namespace of the apisix
namespace: ""
# name of the serviceMonitor, by default, it is the same as the apisix fullname
name: ""
# interval at which metrics should be scraped
interval: 15s
# path of the metrics endpoint
path: /apisix/prometheus/metrics
# prefix of the metrics
metricPrefix: apisix_
# container port where the metrics are exposed
containerPort: 9091
# @param serviceMonitor.labels ServiceMonitor extra labels
labels: {}
# @param serviceMonitor.annotations ServiceMonitor annotations
annotations: {}
# etcd configuration
# use the FQDN address or the IP of the etcd
etcd:
# install etcd(v3) by default, set false if do not want to install etcd(v3) together
enabled: false
host:
- http://etcd-v3.etcd.svc.cluster.local:2379 # host or ip e.g. http://172.20.128.89:2379
prefix: "/apisix"
timeout: 30
# if etcd.enabled is true, set more values of bitnami/etcd helm chart
auth:
rbac:
# No authentication by default
create: false
user: ""
password: ""
tls:
enabled: false
existingSecret: ""
certFilename: ""
certKeyFilename: ""
verify: true
sni: ""
service:
port: 2379
replicaCount: 3
dashboard:
enabled: true
config:
conf:
listen:
host: 0.0.0.0
port: 9000
etcd:
# Supports defining multiple etcd host addresses for an etcd cluster
endpoints:
- etcd-v3.etcd.svc.cluster.local:2379
# apisix configurations prefix
prefix: "/apisix"
authentication:
secert: secert
expire_time: 36000
users:
- username: admin
password: admin
ingress-controller:
enabled: true
fullname: apisix-istio
config:
apisix:
name: apisix-istio
serviceNamespace: apisix-istio
serviceName: apisix-istio-admin
serviceMonitor:
enabled: true
namespace: "apisix-istio"
interval: 15s
# @param serviceMonitor.labels ServiceMonitor extra labels
labels: {}
# @param serviceMonitor.annotations ServiceMonitor annotations
annotations: {}
```
ApisixRoute：
```
apiVersion: apisix.apache.org/v2beta3
kind: ApisixRoute
metadata:
name: http-hello
namespace: launcher
spec:
http:
- name: http-hello
match:
hosts:
- api.xxx.cc
paths:
- /hello*
plugins:
- name: proxy-rewrite
enable: gtrue
config:
regex_uri:
- ^/hello/(.*)
- /$1
backends:
- serviceName: http-hello
servicePort: 8000
```
http-hello server:
```
func main() {
r := gin.Default()
r.GET("/ping", func(c *gin.Context) {
c.JSON(http.StatusOK, gin.H{
"message": "pong",
})
})
r.Run() // listen and serve on 0.0.0.0:8080 (for windows "localhost:8080")
}
```
### Environment
apache/apisix:2.15.0-alpine
apache/apisix-ingress-controller:1.4.1
docker.io/bitnami/etcd:3.5.4-debian-11-r14
Kubernetes cluster version:
Client Version: v1.24.0
Kustomize Version: v4.5.4
Server Version: v1.21.12-gke.2200
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] tzssangglass commented on issue #7811: help request: qps test result is wired
Posted by GitBox <gi...@apache.org>.
tzssangglass commented on issue #7811:
URL: https://github.com/apache/apisix/issues/7811#issuecomment-1231494694
what is the maximum open file limit?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] akalittle closed issue #7811: help request: qps test result is wired
Posted by GitBox <gi...@apache.org>.
akalittle closed issue #7811: help request: qps test result is wired
URL: https://github.com/apache/apisix/issues/7811
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] juzhiyuan commented on issue #7811: help request: qps test result is wired
Posted by GitBox <gi...@apache.org>.
juzhiyuan commented on issue #7811:
URL: https://github.com/apache/apisix/issues/7811#issuecomment-1230073434
Hi @bzp2010, could you please have a quick check and give some hints? I remember we have some related issues :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] akalittle commented on issue #7811: help request: qps test result is wired
Posted by GitBox <gi...@apache.org>.
akalittle commented on issue #7811:
URL: https://github.com/apache/apisix/issues/7811#issuecomment-1231089341
I've tried remove the `proxy-rewrite` plugin, and the result is the same.
The virtual resource are 3 nodes of n2-highcpu-8 (8 vCPUs) on Google Cloud
```
apiVersion: apisix.apache.org/v2beta3
kind: ApisixRoute
metadata:
name: http-hello
namespace: launcher
spec:
http:
- name: http-hello
match:
hosts:
- api.xxx.cc
paths:
- /ping
backends:
- serviceName: http-hello
servicePort: 8000
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] akalittle commented on issue #7811: help request: qps test result is wired
Posted by GitBox <gi...@apache.org>.
akalittle commented on issue #7811:
URL: https://github.com/apache/apisix/issues/7811#issuecomment-1230315412
Maybe you can check this later.
I'm considering this is because of the istio-ingress limits the qps.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] tzssangglass commented on issue #7811: help request: qps test result is wired
Posted by GitBox <gi...@apache.org>.
tzssangglass commented on issue #7811:
URL: https://github.com/apache/apisix/issues/7811#issuecomment-1230107069
How well does wrk test cozzz/http_hello directly?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] tokers commented on issue #7811: help request: qps test result is wired
Posted by GitBox <gi...@apache.org>.
tokers commented on issue #7811:
URL: https://github.com/apache/apisix/issues/7811#issuecomment-1232361608
I have some doubts:
1. Where did you launch the stress test with wrk? Out of Kubernetes cluster?
2. What about sending requests to the backend service directly?
3. The `workerProcesses` field is `auto` so it will launch bunch of worker processes according to the number of CPU cores in your K8s node (not the CPU request for the Pod), and when you only assign 1 vCPU, I think the CPU throttle and context switch are obvious. Try to set `workerProcesses` to `1` to check the QPS for single worker.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] soulbird commented on issue #7811: help request: qps test result is wired
Posted by GitBox <gi...@apache.org>.
soulbird commented on issue #7811:
URL: https://github.com/apache/apisix/issues/7811#issuecomment-1232355758
Try to make wrk's `-c` parameter smaller, like:
```shell
wrk -t16 -c16 -d200s https://api.xx.cc/hello/ping
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] tzssangglass commented on issue #7811: help request: qps test result is wired
Posted by GitBox <gi...@apache.org>.
tzssangglass commented on issue #7811:
URL: https://github.com/apache/apisix/issues/7811#issuecomment-1231060665
I don't understand how to troubleshoot issues like this in a virtualized environment. From the information you provided, I think again the possible performance bottlenecks are
1. proxy-rewrite plugin
2. environment information on virtual resources, such as ulimit, etc.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] akalittle commented on issue #7811: help request: qps test result is wired
Posted by GitBox <gi...@apache.org>.
akalittle commented on issue #7811:
URL: https://github.com/apache/apisix/issues/7811#issuecomment-1230086039
```yaml
apiVersion: core.kalm.dev/v1alpha1
kind: Component
metadata:
labels:
name: http-hello
namespace: launcher
spec:
dnsPolicy: ClusterFirst
image: cozzz/http_hello
ports:
- containerPort: 8080
protocol: http
servicePort: 8000
replicas: 3
restartStrategy: RollingUpdate
schedule: '* * * * *'
terminationGracePeriodSeconds: 30
workloadType: server
```
actually the http-server is above `cozzz/http_hello`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] akalittle commented on issue #7811: help request: qps test result is wired
Posted by GitBox <gi...@apache.org>.
akalittle commented on issue #7811:
URL: https://github.com/apache/apisix/issues/7811#issuecomment-1232467916
Sorry.It's not a problem of apisix.
I've tried a namespace without istio-injetced and the qps goes up rapidly.
Thanks for help.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org