You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@apisix.apache.org by "wjchan89 (via GitHub)" <gi...@apache.org> on 2025/06/17 09:34:06 UTC

[I] bug: multi-cluster mode Kubernetes service discovery is not working [apisix]

wjchan89 opened a new issue, #12341:
URL: https://github.com/apache/apisix/issues/12341

   ### Current Behavior
   
   **Single-Cluster Mode is OK**
   
   discovery:
     kubernetes:
     - id: aaa
       client:
         token: xxx
       service:
         schema: https
         port: '6443'
         host: 192.168.30.99
   
   
   curl http://127.0.0.1:9080/apisix/admin/routes/testaaa \
   -H 'X-API-KEY: xxx' -X PUT -d '
   {
       "methods": ["GET"],
       "uri": "/testaaa",
       "upstream": {
           "discovery_type": "kubernetes",
           "service_name": "aaa/namespace/appa:tcp26160"
       }
   }'
   
   curl http://localhost:9081/testaaa
   {"timestamp":"2025-06-17T09:06:13.478+00:00","status":404,"error":"Not Found","message":"","path":"/testaaa"}
   
   
   
   **Multi-Cluster Mode is not working**
   
   discovery:
     kubernetes:
     - id: aaa
       client:
         token: xxx
       service:
         schema: https
         port: '6443'
         host: 192.168.30.99
     - id: bbb
       client:
         token: xxx
       service:
         schema: https
         port: '6443'
         host: 192.168.30.89
   
   
   curl http://127.0.0.1:9080/apisix/admin/routes/testaaa \
   -H 'X-API-KEY: xxx' -X PUT -d '
   {
       "methods": ["GET"],
       "uri": "/testaaa",
       "upstream": {
           "discovery_type": "kubernetes",
           "service_name": "aaa/namespace/appa:tcp26160"
       }
   }'
   
   
   curl http://localhost:9081/testaaa -i
   HTTP/1.1 503 Service Temporarily Unavailable
   Date: Tue, 17 Jun 2025 09:10:30 GMT
   Content-Type: text/html; charset=utf-8
   Content-Length: 225
   Connection: keep-alive
   Server: APISIX/3.2.2
   
   <html>
   <head><title>503 Service Temporarily Unavailable</title></head>
   <body>
   <center><h1>503 Service Temporarily Unavailable</h1></center>
   <hr><center>openresty</center>
   <p><em>Powered by APISIX.</em></p></body>
   </html>
   
   
   error log:
   2025/06/17 17:10:30 [error] 103307#103307: *2653 [lua] init.lua:493: handle_upstream(): failed to set upstream: no valid upstream node: nil, client: ::1, server: _, request: "GET /testaaa HTTP/1.1", host: "localhost:9081"
   
   ### Expected Behavior
   
   how to config in Multi-Cluster Mode?
   
   ### Error Logs
   
   2025/06/17 17:10:30 [error] 103307#103307: *2653 [lua] init.lua:493: handle_upstream(): failed to set upstream: no valid upstream node: nil, client: ::1, server: _, request: "GET /testaaa HTTP/1.1", host: "localhost:9081"
   
   ### Steps to Reproduce
   
   1. config config.yaml
   2. restart apisix
   3. config route and sent request
   
   ### Environment
   
   - APISIX version (run `apisix version`):3.2
   - Operating system (run `uname -a`):
   - OpenResty / Nginx version (run `openresty -V` or `nginx -V`):
   - etcd version, if relevant (run `curl http://127.0.0.1:9090/v1/server_info`):
   - APISIX Dashboard version, if relevant:
   - Plugin runner version, for issues related to plugin runners:
   - LuaRocks version, for installation issues (run `luarocks --version`):
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] bug: multi-cluster mode Kubernetes service discovery is not working [apisix]

Posted by "Baoyuantop (via GitHub)" <gi...@apache.org>.
Baoyuantop commented on issue #12341:
URL: https://github.com/apache/apisix/issues/12341#issuecomment-2983562638

   > test in apisix 3.2 and 3.12
   
   Can you try using the control API dump data in 3.12?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] bug: multi-cluster mode Kubernetes service discovery is not working [apisix]

Posted by "Baoyuantop (via GitHub)" <gi...@apache.org>.
Baoyuantop commented on issue #12341:
URL: https://github.com/apache/apisix/issues/12341#issuecomment-2995042898

   Hi @wjchan89, any updates?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] bug: multi-cluster mode Kubernetes service discovery is not working [apisix]

Posted by "Baoyuantop (via GitHub)" <gi...@apache.org>.
Baoyuantop closed issue #12341: bug: multi-cluster mode Kubernetes service discovery is not working
URL: https://github.com/apache/apisix/issues/12341


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] bug: multi-cluster mode Kubernetes service discovery is not working [apisix]

Posted by "Baoyuantop (via GitHub)" <gi...@apache.org>.
Baoyuantop commented on issue #12341:
URL: https://github.com/apache/apisix/issues/12341#issuecomment-2982485135

   Are there any other error logs?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] bug: multi-cluster mode Kubernetes service discovery is not working [apisix]

Posted by "wjchan89 (via GitHub)" <gi...@apache.org>.
wjchan89 commented on issue #12341:
URL: https://github.com/apache/apisix/issues/12341#issuecomment-2983525626

   > Are there any other error logs?
   
   There are no more logs.
   
   But,if I delete the appa from one cluster,it work.
   Different Cluster ID in apisix , same namespace and same service in two cluster, not working. test in apisix 3.2 and 3.12


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] bug: multi-cluster mode Kubernetes service discovery is not working [apisix]

Posted by "Baoyuantop (via GitHub)" <gi...@apache.org>.
Baoyuantop commented on issue #12341:
URL: https://github.com/apache/apisix/issues/12341#issuecomment-3116930289

   If there is still a problem, please open it again.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] bug: multi-cluster mode Kubernetes service discovery is not working [apisix]

Posted by "wjchan89 (via GitHub)" <gi...@apache.org>.
wjchan89 commented on issue #12341:
URL: https://github.com/apache/apisix/issues/12341#issuecomment-2982455879

   > Hi [@wjchan89](https://github.com/wjchan89), are there any other error logs in apisix? We need to make sure that apisix can access the cluster normally. You can also check the service discovery data in apisix with the control api https://apisix.apache.org/docs/apisix/control-api/#get-v1discoveryservicedump
   
   apisix version 3.2,there is no such control api for discovery.
   
   when I set one cluster like this,
   ```bash
   discovery:
     kubernetes:
     - id: aaa
       client:
         token: xxx
       service:
         schema: https
         port: '6443'
         host: 192.168.30.99
   ```
   
   the routes:
   ```bash
   curl http: //127.0.0.1:9090/v1/routes
   [
       {
           "has_domain": false,
           "orig_modifiedIndex": 185,
           "createdIndex": 68,
           "modifiedIndex": 185,
           "value": {
               "create_time": 1750146490,
               "methods": [
                   "GET"
               ],
               "uri": "/flowRuleA",
               "update_time": 1750214026,
               "id": "testaaa",
               "upstream": {
                   "original_nodes": [
                       {
                           "port": 26160,
                           "weight": 50,
                           "host": "192.168.30.124",
                           "metadata": {
                               "pod-template-hash": "6f5c7b5c47",
                               "version": "v1",
                               "app": "appa"
                           }
                       }
                   ],
                   "hash_on": "vars",
                   "service_name": "aaa/namespace/appa:tcp26160",
                   "type": "roundrobin",
                   "pass_host": "pass",
                   "nodes": [
                       {
                           "metadata": {
                               "pod-template-hash": "6f5c7b5c47",
                               "version": "v1",
                               "app": "appa"
                           },
                           "port": 26160,
                           "upstream_host": "192.168.30.124:26160",
                           "priority": 0,
                           "host": "192.168.30.124",
                           "weight": 50
                       }
                   ],
                   "discovery_type": "kubernetes",
                   "scheme": "http",
                   "nodes_ref": [
                       {
                           "upstream_host": "192.168.30.124:26160",
                           "priority": 0,
                           "port": 26160,
                           "weight": 50,
                           "host": "192.168.30.124",
                           "metadata": {
                               "pod-template-hash": "6f5c7b5c47",
                               "version": "v1",
                               "app": "appa"
                           }
                       }
                   ]
               },
               "priority": 0,
               "status": 1
           },
           "clean_handlers": {},
           "key": "/apisix/routes/testaaa"
       },
       {
           "has_domain": false,
           "orig_modifiedIndex": 142,
           "createdIndex": 119,
           "modifiedIndex": 142,
           "value": {
               "update_time": 1750150991,
               "create_time": 1750148686,
               "priority": 0,
               "id": "testbbb",
               "uri": "/testbbb",
               "upstream": {
                   "pass_host": "pass",
                   "service_name": "bbb/namespace/appa:tcp8080",
                   "hash_on": "vars",
                   "discovery_type": "kubernetes",
                   "scheme": "http",
                   "type": "roundrobin"
               },
               "methods": [
                   "GET"
               ],
               "status": 1
           },
           "clean_handlers": {},
           "key": "/apisix/routes/testbbb"
       }
   ]
   ```
   
   when I set two clusters like this,
   ```bash
   discovery:
     kubernetes:
     - id: aaa
       client:
         token: xxx
       service:
         schema: https
         port: '6443'
         host: 192.168.30.99
     - id: bbb
       client:
         token: xxx
       service:
         schema: https
         port: '6443'
         host: 192.168.30.89
   ```
   
   the routes:
   ```bash
   curl http: //127.0.0.1:9090/v1/routes
   
   [
       {
           "createdIndex": 68,
           "clean_handlers": {},
           "modifiedIndex": 120,
           "key": "/apisix/routes/testaaa",
           "has_domain": false,
           "value": {
               "methods": [
                   "GET"
               ],
               "update_time": 1750148965,
               "create_time": 1750146490,
               "priority": 0,
               "uri": "/flowRuleA",
               "status": 1,
               "id": "testaaa",
               "upstream": {
                   "pass_host": "pass",
                   "service_name": "aaa/namespace/appa:tcp26160",
                   "scheme": "http",
                   "type": "roundrobin",
                   "hash_on": "vars",
                   "discovery_type": "kubernetes"
               }
           },
           "orig_modifiedIndex": 120
       },
       {
           "createdIndex": 119,
           "clean_handlers": {},
           "modifiedIndex": 142,
           "key": "/apisix/routes/testbbb",
           "has_domain": false,
           "value": {
               "upstream": {
                   "service_name": "bbb/namespace/appa:tcp8080",
                   "scheme": "http",
                   "pass_host": "pass",
                   "type": "roundrobin",
                   "hash_on": "vars",
                   "discovery_type": "kubernetes"
               },
               "update_time": 1750150991,
               "create_time": 1750148686,
               "methods": [
                   "GET"
               ],
               "uri": "/testbbb",
               "priority": 0,
               "id": "testbbb",
               "status": 1
           },
           "orig_modifiedIndex": 142
       }
   ]
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] bug: multi-cluster mode Kubernetes service discovery is not working [apisix]

Posted by "Baoyuantop (via GitHub)" <gi...@apache.org>.
Baoyuantop commented on issue #12341:
URL: https://github.com/apache/apisix/issues/12341#issuecomment-2982329623

   Hi @wjchan89, are there any other error logs in apisix? We need to make sure that apisix can access the cluster normally. You can also check the service discovery data in apisix with the control api https://apisix.apache.org/docs/apisix/control-api/#get-v1discoveryservicedump


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org