You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@apisix.apache.org by "pliwang (via GitHub)" <gi...@apache.org> on 2023/08/21 02:37:28 UTC

[GitHub] [apisix] pliwang opened a new issue, #10062: bug: In the case of high concurrent requests, the default type of upstreams cannot achieve load balancing.

pliwang opened a new issue, #10062:
URL: https://github.com/apache/apisix/issues/10062

   ### Current Behavior
   
   There are 30 routes corresponding to an upstream, including 5 nodes. When 3000 requests these 30 routes concurrently, the problem of 502 will occasionally occur.
   
   ### Expected Behavior
   
   After careful investigation, it is found that the default load balancing policy type of upstream can not reach the polling choice of nodes,Thereby causing the problem of 502.How should I avoid such a problem?
   
   ### Error Logs
   
   ![Cache_76aaa31a04c064d9](https://github.com/apache/apisix/assets/142763240/2e516a2b-a1ee-44bb-bcf3-e10a33997db7)
   
   
   ### Steps to Reproduce
   
   1.   30 routes
   2.   30 routes correspond to an upstream.
   3.   Concurrent access
   4.    Check whether the number of requests processed by upstream services is balanced.
   
   ### Environment
   
   - APISIX version (run `apisix version`):
   - Operating system (run `uname -a`):
   - OpenResty / Nginx version (run `openresty -V` or `nginx -V`):
   - etcd version, if relevant (run `curl http://127.0.0.1:9090/v1/server_info`):
   - APISIX Dashboard version, if relevant:
   - Plugin runner version, for issues related to plugin runners:
   - LuaRocks version, for installation issues (run `luarocks --version`):
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [apisix] pliwang commented on issue #10062: bug: In the case of high concurrent requests, the default type of upstreams cannot achieve load balancing.

Posted by "pliwang (via GitHub)" <gi...@apache.org>.
pliwang commented on issue #10062:
URL: https://github.com/apache/apisix/issues/10062#issuecomment-1685717366

   `      {
           "createdIndex": 62589,
           "modifiedIndex": 62589,
           "key": "/apisix/2-default-ns/routes/daadccdb",
           "value": {
             "name": "ingress_demo11_cloudspace_demo1_8001__/podip/h11",
             "desc": "Created by apisix-ingress-controller, DO NOT modify it manually",
             "labels": {
               "managed-by": "apisix-ingress-controller",
               "svcPort": "8001",
               "projectId": "2",
               "sourceType": "ingress",
               "sourceName": "demo11",
               "path": "/podip/h11",
               "serviceName": "demo1",
               "namespace": "cloudspace"
             },
             "id": "daadccdb",
             "create_time": 1692237845,
             "update_time": 1692237845,
             "priority": 0,
             "status": 1,
             "uris": [
               "/podip/h11",
               "/podip/h11/*"
             ],
             "upstream_id": "bb223adb"
           }
         },
         {
           "createdIndex": 62606,
           "modifiedIndex": 62606,
           "key": "/apisix/2-default-ns/routes/de7714e6",
           "value": {
             "name": "ingress_demo7_cloudspace_demo1_8001__/podip/h7",
             "desc": "Created by apisix-ingress-controller, DO NOT modify it manually",
             "labels": {
               "managed-by": "apisix-ingress-controller",
               "svcPort": "8001",
               "projectId": "2",
               "sourceType": "ingress",
               "sourceName": "demo7",
               "path": "/podip/h7",
               "serviceName": "demo1",
               "namespace": "cloudspace"
             },
             "id": "de7714e6",
             "create_time": 1692237848,
             "update_time": 1692237848,
             "priority": 0,
             "status": 1,
             "uris": [
               "/podip/h7",
               "/podip/h7/*"
             ],
             "upstream_id": "bb223adb"
           }
         }`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [apisix] pliwang commented on issue #10062: bug: In the case of high concurrent requests, the default type of upstreams cannot achieve load balancing.

Posted by "pliwang (via GitHub)" <gi...@apache.org>.
pliwang commented on issue #10062:
URL: https://github.com/apache/apisix/issues/10062#issuecomment-1686170173

   > > 30 routes correspond to an upstream.
   > 
   > Do you mean to say that each route corresponds to a certain upstream or each route corresponds to the same upstream?
   
   Each route used corresponds to an upstream.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [apisix] Revolyssup commented on issue #10062: bug: In the case of high concurrent requests, the default type of upstreams cannot achieve load balancing.

Posted by "Revolyssup (via GitHub)" <gi...@apache.org>.
Revolyssup commented on issue #10062:
URL: https://github.com/apache/apisix/issues/10062#issuecomment-1685593895

   Can you provide the upstream configuration? One of the route configuration as well maybe?
   And share the error log of APISIX when 502 occurs.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [apisix] pliwang commented on issue #10062: bug: In the case of high concurrent requests, the default type of upstreams cannot achieve load balancing.

Posted by "pliwang (via GitHub)" <gi...@apache.org>.
pliwang commented on issue #10062:
URL: https://github.com/apache/apisix/issues/10062#issuecomment-1685717922

   ` {
       "nodes": [
         {
           "createdIndex": 62549,
           "modifiedIndex": 62632,
           "key": "/apisix/2-default-ns/upstreams/bb223adb",
           "value": {
             "name": "cloudspace_demo1_8001",
             "desc": "Created by apisix-ingress-controller, DO NOT modify it manually",
             "nodes": [
               {
                 "port": 8000,
                 "host": "192.168.167.100",
                 "weight": 100,
                 "priority": 0
               },
               {
                 "port": 8000,
                 "host": "192.168.228.235",
                 "weight": 100,
                 "priority": 0
               },
               {
                 "port": 8000,
                 "host": "192.168.23.198",
                 "weight": 100,
                 "priority": 0
               },
               {
                 "port": 8000,
                 "host": "192.168.235.237",
                 "weight": 100,
                 "priority": 0
               },
               {
                 "port": 8000,
                 "host": "192.168.71.143",
                 "weight": 100,
                 "priority": 0
               }
             ],
             "hash_on": "vars",
             "labels": {
               "managed-by": "apisix-ingress-controller",
               "svc": "demo1",
               "port": "8001",
               "namespace": "cloudspace"
             },
             "id": "bb223adb",
             "create_time": 1692237843,
             "type": "roundrobin",
             "scheme": "http",
             "pass_host": "pass",
             "update_time": 1692599826
           }
         }
       ],
       "dir": true,
       "key": "/apisix/2-default-ns/upstreams"
     }
   `


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [apisix] shreemaan-abhishek commented on issue #10062: bug: In the case of high concurrent requests, the default type of upstreams cannot achieve load balancing.

Posted by "shreemaan-abhishek (via GitHub)" <gi...@apache.org>.
shreemaan-abhishek commented on issue #10062:
URL: https://github.com/apache/apisix/issues/10062#issuecomment-1685840031

   > 30 routes correspond to an upstream.
   
   Do you mean to say that each route corresponds to a certain upstream or each route corresponds to the same upstream?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [apisix] pliwang commented on issue #10062: bug: In the case of high concurrent requests, the default type of upstreams cannot achieve load balancing.

Posted by "pliwang (via GitHub)" <gi...@apache.org>.
pliwang commented on issue #10062:
URL: https://github.com/apache/apisix/issues/10062#issuecomment-1686171133

   > @pliwang Can you provide the `ApisixUpstream` config or `Ingress` config, whatever you used with ingress controller to configure the upstream?
   
   apiVersion: extensions/v1beta1
   kind: Ingress
   metadata:
     annotations:
       kubernetes.io/ingress.class: apisix
     creationTimestamp: "2023-08-17T02:04:02Z"
     generation: 1
     labels:
       cube.gateway.apisix.projectId: "2"
     managedFields:
     - apiVersion: networking.k8s.io/v1
       fieldsType: FieldsV1
       fieldsV1:
         f:spec:
           f:rules: {}
       manager: OpenAPI-Generator
       operation: Update
       time: "2023-08-17T02:04:02Z"
     name: demo10
     namespace: cloudspace
     resourceVersion: "55774113"
     selfLink: /apis/extensions/v1beta1/namespaces/cloudspace/ingresses/demo10
     uid: e2f79de3-8282-4ac9-b55a-c5aafc01e979
   spec:
     rules:
     - http:
         paths:
         - backend:
             serviceName: demo1
             servicePort: 8001
           path: /podip/h10
           pathType: Prefix
   status:
     loadBalancer: {}
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [apisix] pliwang commented on issue #10062: bug: In the case of high concurrent requests, the default type of upstreams cannot achieve load balancing.

Posted by "pliwang (via GitHub)" <gi...@apache.org>.
pliwang commented on issue #10062:
URL: https://github.com/apache/apisix/issues/10062#issuecomment-1685564746

   Error Logs
   ![IMG_20230821_111423](https://github.com/apache/apisix/assets/142763240/e934d100-8532-40e7-a089-a3974e84524c)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [apisix] pliwang commented on issue #10062: bug: In the case of high concurrent requests, the default type of upstreams cannot achieve load balancing.

Posted by "pliwang (via GitHub)" <gi...@apache.org>.
pliwang commented on issue #10062:
URL: https://github.com/apache/apisix/issues/10062#issuecomment-1685564140

   Error Logs
   ![Uploading IMG_20230821_111423.jpg…]()
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [apisix] Revolyssup commented on issue #10062: bug: In the case of high concurrent requests, the default type of upstreams cannot achieve load balancing.

Posted by "Revolyssup (via GitHub)" <gi...@apache.org>.
Revolyssup commented on issue #10062:
URL: https://github.com/apache/apisix/issues/10062#issuecomment-1685974859

   @pliwang Can you provide the `ApisixUpstream` config or `Ingress` config, whatever you used with ingress controller to configure the upstream?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [apisix] pliwang commented on issue #10062: bug: In the case of high concurrent requests, the default type of upstreams cannot achieve load balancing.

Posted by "pliwang (via GitHub)" <gi...@apache.org>.
pliwang commented on issue #10062:
URL: https://github.com/apache/apisix/issues/10062#issuecomment-1685716470

   
   The reason of 502 is that my service can only support about 1000 concurrency. I have five such services. When 3000 concurrency, I expect each service to be able to load balance to 600 to 1000 requests, but in fact, I find that the load of a service may be as high as more than 2000 requests. The following are the related configurations of and.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org