You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@apisix.apache.org by "pliwang (via GitHub)" <gi...@apache.org> on 2023/08/21 02:37:28 UTC
[GitHub] [apisix] pliwang opened a new issue, #10062: bug: In the case of high concurrent requests, the default type of upstreams cannot achieve load balancing.
pliwang opened a new issue, #10062:
URL: https://github.com/apache/apisix/issues/10062
### Current Behavior
There are 30 routes corresponding to an upstream, including 5 nodes. When 3000 requests these 30 routes concurrently, the problem of 502 will occasionally occur.
### Expected Behavior
After careful investigation, it is found that the default load balancing policy type of upstream can not reach the polling choice of nodes,Thereby causing the problem of 502.How should I avoid such a problem?
### Error Logs

### Steps to Reproduce
1. 30 routes
2. 30 routes correspond to an upstream.
3. Concurrent access
4. Check whether the number of requests processed by upstream services is balanced.
### Environment
- APISIX version (run `apisix version`):
- Operating system (run `uname -a`):
- OpenResty / Nginx version (run `openresty -V` or `nginx -V`):
- etcd version, if relevant (run `curl http://127.0.0.1:9090/v1/server_info`):
- APISIX Dashboard version, if relevant:
- Plugin runner version, for issues related to plugin runners:
- LuaRocks version, for installation issues (run `luarocks --version`):
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] pliwang commented on issue #10062: bug: In the case of high concurrent requests, the default type of upstreams cannot achieve load balancing.
Posted by "pliwang (via GitHub)" <gi...@apache.org>.
pliwang commented on issue #10062:
URL: https://github.com/apache/apisix/issues/10062#issuecomment-1685717366
` {
"createdIndex": 62589,
"modifiedIndex": 62589,
"key": "/apisix/2-default-ns/routes/daadccdb",
"value": {
"name": "ingress_demo11_cloudspace_demo1_8001__/podip/h11",
"desc": "Created by apisix-ingress-controller, DO NOT modify it manually",
"labels": {
"managed-by": "apisix-ingress-controller",
"svcPort": "8001",
"projectId": "2",
"sourceType": "ingress",
"sourceName": "demo11",
"path": "/podip/h11",
"serviceName": "demo1",
"namespace": "cloudspace"
},
"id": "daadccdb",
"create_time": 1692237845,
"update_time": 1692237845,
"priority": 0,
"status": 1,
"uris": [
"/podip/h11",
"/podip/h11/*"
],
"upstream_id": "bb223adb"
}
},
{
"createdIndex": 62606,
"modifiedIndex": 62606,
"key": "/apisix/2-default-ns/routes/de7714e6",
"value": {
"name": "ingress_demo7_cloudspace_demo1_8001__/podip/h7",
"desc": "Created by apisix-ingress-controller, DO NOT modify it manually",
"labels": {
"managed-by": "apisix-ingress-controller",
"svcPort": "8001",
"projectId": "2",
"sourceType": "ingress",
"sourceName": "demo7",
"path": "/podip/h7",
"serviceName": "demo1",
"namespace": "cloudspace"
},
"id": "de7714e6",
"create_time": 1692237848,
"update_time": 1692237848,
"priority": 0,
"status": 1,
"uris": [
"/podip/h7",
"/podip/h7/*"
],
"upstream_id": "bb223adb"
}
}`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] pliwang commented on issue #10062: bug: In the case of high concurrent requests, the default type of upstreams cannot achieve load balancing.
Posted by "pliwang (via GitHub)" <gi...@apache.org>.
pliwang commented on issue #10062:
URL: https://github.com/apache/apisix/issues/10062#issuecomment-1686170173
> > 30 routes correspond to an upstream.
>
> Do you mean to say that each route corresponds to a certain upstream or each route corresponds to the same upstream?
Each route used corresponds to an upstream.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] Revolyssup commented on issue #10062: bug: In the case of high concurrent requests, the default type of upstreams cannot achieve load balancing.
Posted by "Revolyssup (via GitHub)" <gi...@apache.org>.
Revolyssup commented on issue #10062:
URL: https://github.com/apache/apisix/issues/10062#issuecomment-1685593895
Can you provide the upstream configuration? One of the route configuration as well maybe?
And share the error log of APISIX when 502 occurs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] pliwang commented on issue #10062: bug: In the case of high concurrent requests, the default type of upstreams cannot achieve load balancing.
Posted by "pliwang (via GitHub)" <gi...@apache.org>.
pliwang commented on issue #10062:
URL: https://github.com/apache/apisix/issues/10062#issuecomment-1685717922
` {
"nodes": [
{
"createdIndex": 62549,
"modifiedIndex": 62632,
"key": "/apisix/2-default-ns/upstreams/bb223adb",
"value": {
"name": "cloudspace_demo1_8001",
"desc": "Created by apisix-ingress-controller, DO NOT modify it manually",
"nodes": [
{
"port": 8000,
"host": "192.168.167.100",
"weight": 100,
"priority": 0
},
{
"port": 8000,
"host": "192.168.228.235",
"weight": 100,
"priority": 0
},
{
"port": 8000,
"host": "192.168.23.198",
"weight": 100,
"priority": 0
},
{
"port": 8000,
"host": "192.168.235.237",
"weight": 100,
"priority": 0
},
{
"port": 8000,
"host": "192.168.71.143",
"weight": 100,
"priority": 0
}
],
"hash_on": "vars",
"labels": {
"managed-by": "apisix-ingress-controller",
"svc": "demo1",
"port": "8001",
"namespace": "cloudspace"
},
"id": "bb223adb",
"create_time": 1692237843,
"type": "roundrobin",
"scheme": "http",
"pass_host": "pass",
"update_time": 1692599826
}
}
],
"dir": true,
"key": "/apisix/2-default-ns/upstreams"
}
`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] shreemaan-abhishek commented on issue #10062: bug: In the case of high concurrent requests, the default type of upstreams cannot achieve load balancing.
Posted by "shreemaan-abhishek (via GitHub)" <gi...@apache.org>.
shreemaan-abhishek commented on issue #10062:
URL: https://github.com/apache/apisix/issues/10062#issuecomment-1685840031
> 30 routes correspond to an upstream.
Do you mean to say that each route corresponds to a certain upstream or each route corresponds to the same upstream?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] pliwang commented on issue #10062: bug: In the case of high concurrent requests, the default type of upstreams cannot achieve load balancing.
Posted by "pliwang (via GitHub)" <gi...@apache.org>.
pliwang commented on issue #10062:
URL: https://github.com/apache/apisix/issues/10062#issuecomment-1686171133
> @pliwang Can you provide the `ApisixUpstream` config or `Ingress` config, whatever you used with ingress controller to configure the upstream?
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
annotations:
kubernetes.io/ingress.class: apisix
creationTimestamp: "2023-08-17T02:04:02Z"
generation: 1
labels:
cube.gateway.apisix.projectId: "2"
managedFields:
- apiVersion: networking.k8s.io/v1
fieldsType: FieldsV1
fieldsV1:
f:spec:
f:rules: {}
manager: OpenAPI-Generator
operation: Update
time: "2023-08-17T02:04:02Z"
name: demo10
namespace: cloudspace
resourceVersion: "55774113"
selfLink: /apis/extensions/v1beta1/namespaces/cloudspace/ingresses/demo10
uid: e2f79de3-8282-4ac9-b55a-c5aafc01e979
spec:
rules:
- http:
paths:
- backend:
serviceName: demo1
servicePort: 8001
path: /podip/h10
pathType: Prefix
status:
loadBalancer: {}
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] pliwang commented on issue #10062: bug: In the case of high concurrent requests, the default type of upstreams cannot achieve load balancing.
Posted by "pliwang (via GitHub)" <gi...@apache.org>.
pliwang commented on issue #10062:
URL: https://github.com/apache/apisix/issues/10062#issuecomment-1685564746
Error Logs

--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] pliwang commented on issue #10062: bug: In the case of high concurrent requests, the default type of upstreams cannot achieve load balancing.
Posted by "pliwang (via GitHub)" <gi...@apache.org>.
pliwang commented on issue #10062:
URL: https://github.com/apache/apisix/issues/10062#issuecomment-1685564140
Error Logs
![Uploading IMG_20230821_111423.jpg…]()
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] Revolyssup commented on issue #10062: bug: In the case of high concurrent requests, the default type of upstreams cannot achieve load balancing.
Posted by "Revolyssup (via GitHub)" <gi...@apache.org>.
Revolyssup commented on issue #10062:
URL: https://github.com/apache/apisix/issues/10062#issuecomment-1685974859
@pliwang Can you provide the `ApisixUpstream` config or `Ingress` config, whatever you used with ingress controller to configure the upstream?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] pliwang commented on issue #10062: bug: In the case of high concurrent requests, the default type of upstreams cannot achieve load balancing.
Posted by "pliwang (via GitHub)" <gi...@apache.org>.
pliwang commented on issue #10062:
URL: https://github.com/apache/apisix/issues/10062#issuecomment-1685716470
The reason of 502 is that my service can only support about 1000 concurrency. I have five such services. When 3000 concurrency, I expect each service to be able to load balance to 600 to 1000 requests, but in fact, I find that the load of a service may be as high as more than 2000 requests. The following are the related configurations of and.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org